2024-09-11T02:36:37.698 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T02:36:37.719 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T02:36:37.818 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006 branch: wip-hemanth-testing-2024-09-10-0723 description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} email: hyelloji@redhat.com first_in_suite: false job_id: '7900006' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-hemanth-testing-2024-09-10-0723 ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-hemanth-testing-2024-09-10-0723 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 owner: scheduled_hyelloji@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9167 sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-hemanth-testing-2024-09-10-0723 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC29kLTUrIQi9ykqRkPmyOPDuuYW5727QeTKFV/QfTpi0fJ38TK6j33ZcuOenyXqj4oZURJmqaPoWi5ZE99z6Rw= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqA3w7um1PM9K9pP2EmmbqLJ8EKTOG5V/fvReCU9BrBibStFoii6XDaktx8tmak7dy8CWbyE0LBuJ/wVMRvlBs= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:16:14 tube: smithi user: hyelloji verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T02:36:37.818 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa; will attempt to use it 2024-09-11T02:36:37.819 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa/tasks 2024-09-11T02:36:37.819 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T02:36:37.820 INFO:teuthology.task.internal:Checking packages... 2024-09-11T02:36:37.844 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8f219c476cb4f26fc5719c661c0323719a45cb30' 2024-09-11T02:36:37.844 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T02:36:37.844 INFO:teuthology.packaging:ref: None 2024-09-11T02:36:37.844 INFO:teuthology.packaging:tag: None 2024-09-11T02:36:37.844 INFO:teuthology.packaging:branch: wip-hemanth-testing-2024-09-10-0723 2024-09-11T02:36:37.844 INFO:teuthology.packaging:sha1: 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T02:36:37.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-hemanth-testing-2024-09-10-0723 2024-09-11T02:36:38.026 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:36:38.028 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T02:36:38.035 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T02:36:38.035 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T02:36:38.041 INFO:teuthology.task.internal:Saving configuration 2024-09-11T02:36:38.059 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T02:36:38.065 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T02:36:38.095 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 02:30:53.819993', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEWHOgdufIoh9yrhQ4qVZ6pG9Vu3bpME2pecBACS7uNz'} 2024-09-11T02:36:38.127 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 02:30:53.822002', 'locked_by': 'scheduled_hyelloji@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIEIkxeJQFm+ZWylMDGFmJ1nn/CwmD/Maxfod+qNYBN+'} 2024-09-11T02:36:38.127 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T02:36:38.141 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-09-11T02:36:38.142 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-11T02:36:38.142 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T02:36:38.221 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f812ab27be0>, signals=[15]) 2024-09-11T02:36:38.221 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T02:36:38.227 INFO:teuthology.task.internal:Opening connections... 2024-09-11T02:36:38.228 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T02:36:38.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:36:38.323 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T02:36:38.324 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:36:38.419 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T02:36:38.430 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-09-11T02:36:38.436 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-09-11T02:36:38.436 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Ubuntu" 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="22.04" 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_CODENAME=jammy 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:ID=ubuntu 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE=debian 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T02:36:38.484 INFO:teuthology.orchestra.run.smithi098.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T02:36:38.485 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T02:36:38.485 INFO:teuthology.orchestra.run.smithi098.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T02:36:38.485 INFO:teuthology.orchestra.run.smithi098.stdout:UBUNTU_CODENAME=jammy 2024-09-11T02:36:38.485 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-09-11T02:36:38.527 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-09-11T02:36:38.532 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-09-11T02:36:38.533 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Ubuntu" 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="22.04" 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_CODENAME=jammy 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:ID=ubuntu 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE=debian 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T02:36:38.582 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T02:36:38.583 INFO:teuthology.orchestra.run.smithi190.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T02:36:38.583 INFO:teuthology.orchestra.run.smithi190.stdout:UBUNTU_CODENAME=jammy 2024-09-11T02:36:38.583 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-09-11T02:36:38.610 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T02:36:38.620 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T02:36:38.624 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T02:36:38.625 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-09-11T02:36:38.627 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-09-11T02:36:38.631 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T02:36:38.638 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T02:36:38.638 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-09-11T02:36:38.673 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-09-11T02:36:38.679 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T02:36:38.752 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T02:36:38.767 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T02:36:38.768 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T02:36:38.768 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T02:36:38.768 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T02:36:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:36:38.773 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-11T02:36:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:36:38.774 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-11T02:36:38.821 INFO:teuthology.orchestra.run.smithi098.stdout:5.15.0-56-generic 2024-09-11T02:36:38.821 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.15.0-56-generic 2024-09-11T02:36:38.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-09-11T02:36:38.822 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-56-generic 2024-09-11T02:36:38.823 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.15.0-56-generic 2024-09-11T02:36:38.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2024-09-11T02:36:38.949 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-09-11T02:36:38.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-09-11T02:36:39.113 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T02:36:39.122 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T02:36:39.195 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T02:36:39.227 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T02:36:39.277 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T02:36:39.309 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T02:36:39.317 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T02:36:39.344 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T02:36:39.413 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T02:36:39.423 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T02:36:39.426 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T02:36:39.428 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T02:36:39.429 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T02:36:39.449 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T02:36:39.449 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T02:36:39.450 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T02:36:39.452 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T02:36:39.452 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T02:36:39.453 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T02:36:39.453 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T02:36:39.456 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T02:36:39.462 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T02:36:39.463 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T02:36:39.463 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T02:36:39.463 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T02:36:39.471 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T02:36:39.471 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T02:36:39.474 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T02:36:39.474 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T02:36:39.478 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T02:36:39.480 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T02:36:39.483 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T02:36:39.484 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T02:36:39.486 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T02:36:39.486 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T02:36:39.507 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T02:36:39.519 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T02:36:39.519 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T02:36:39.631 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T02:36:39.664 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T02:36:39.770 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T02:36:39.802 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T02:36:40.084 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T02:36:40.117 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T02:36:40.119 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T02:36:40.134 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T02:36:40.134 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T02:36:40.153 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T02:36:40.167 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T02:36:40.168 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T02:36:40.201 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T02:36:40.204 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T02:36:40.217 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T02:36:40.217 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T02:36:40.234 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T02:36:40.237 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T02:36:40.253 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T02:36:40.253 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T02:36:40.254 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T02:36:40.273 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T02:36:40.278 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T02:36:40.279 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T02:36:40.280 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T02:36:40.286 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T02:36:40.305 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T02:36:40.310 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T02:36:40.311 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T02:36:40.312 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T02:36:40.329 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T02:36:40.330 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T02:36:40.330 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T02:36:40.331 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T02:36:40.362 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T02:36:40.363 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T02:36:40.363 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T02:36:40.364 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T02:36:40.364 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T02:36:40.364 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T02:36:40.364 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T02:36:40.365 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T02:36:40.365 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T02:36:40.395 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T02:36:40.395 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T02:36:40.396 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T02:36:40.397 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T02:36:40.397 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T02:36:40.410 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T02:36:40.443 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T02:36:45.631 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 15.8 MB in 2s (7,441 kB/s) 2024-09-11T02:36:45.839 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 15.8 MB in 2s (7,387 kB/s) 2024-09-11T02:36:47.118 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T02:36:47.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T02:36:47.203 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T02:36:47.332 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-11T02:36:47.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T02:36:47.395 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-09-11T02:36:47.395 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-09-11T02:36:47.415 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-11T02:36:47.563 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T02:36:47.604 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T02:36:47.605 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: python2.7-minimal 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2024-09-11T02:36:47.606 INFO:teuthology.orchestra.run.smithi190.stdout: thermald 2024-09-11T02:36:47.620 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2024-09-11T02:36:47.620 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T02:36:47.621 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T02:36:47.621 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T02:36:47.622 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages will be upgraded: 2024-09-11T02:36:47.623 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T02:36:47.624 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-09-11T02:36:47.625 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-09-11T02:36:47.803 INFO:teuthology.orchestra.run.smithi190.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T02:36:47.803 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 113 MB of archives. 2024-09-11T02:36:47.803 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T02:36:47.803 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T02:36:47.812 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T02:36:47.812 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T02:36:47.812 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T02:36:47.812 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T02:36:47.813 INFO:teuthology.orchestra.run.smithi098.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T02:36:47.813 INFO:teuthology.orchestra.run.smithi098.stdout: python2.7-minimal 2024-09-11T02:36:47.813 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T02:36:47.814 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-09-11T02:36:47.814 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T02:36:47.814 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T02:36:47.815 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T02:36:47.816 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-09-11T02:36:47.816 INFO:teuthology.orchestra.run.smithi098.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T02:36:47.817 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-09-11T02:36:47.817 INFO:teuthology.orchestra.run.smithi098.stdout: thermald 2024-09-11T02:36:47.863 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-09-11T02:36:47.864 INFO:teuthology.orchestra.run.smithi098.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T02:36:47.864 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T02:36:47.864 INFO:teuthology.orchestra.run.smithi098.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T02:36:47.865 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages will be upgraded: 2024-09-11T02:36:47.866 INFO:teuthology.orchestra.run.smithi098.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T02:36:47.918 INFO:teuthology.orchestra.run.smithi098.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T02:36:47.918 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 113 MB of archives. 2024-09-11T02:36:47.918 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T02:36:47.918 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T02:36:48.371 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T02:36:48.467 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T02:36:49.206 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T02:36:49.265 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T02:36:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T02:36:49.266 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T02:36:49.386 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T02:36:49.472 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T02:36:49.550 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T02:36:49.870 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 113 MB in 2s (70.9 MB/s) 2024-09-11T02:36:50.065 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T02:36:51.744 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T02:36:51.744 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T02:36:51.745 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T02:36:52.091 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T02:36:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T02:36:52.131 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:36:52.171 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T02:36:52.323 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:36:52.576 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 113 MB in 5s (25.0 MB/s) 2024-09-11T02:36:52.767 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T02:36:54.732 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T02:36:54.737 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:36:54.830 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:36:54.883 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T02:36:54.904 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:36:54.986 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:36:55.520 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T02:36:55.540 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:36:55.578 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:36:57.264 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T02:36:57.286 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:36:57.365 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:36:57.784 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T02:36:57.805 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:36:57.859 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:02.615 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T02:37:02.765 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T02:37:03.097 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T02:37:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T02:37:03.505 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T02:37:03.527 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T02:37:03.588 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T02:37:04.239 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T02:37:04.358 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T02:37:04.699 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T02:37:04.828 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T02:37:05.114 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T02:37:05.137 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T02:37:05.172 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T02:37:10.102 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T02:37:10.139 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:37:10.193 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:11.056 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T02:37:11.094 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T02:37:11.139 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:13.023 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T02:37:13.148 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T02:37:13.540 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T02:37:13.687 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:13.778 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T02:37:13.812 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T02:37:13.876 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T02:37:13.966 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:14.243 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T02:37:14.364 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:14.490 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T02:37:14.619 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:15.367 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T02:37:15.367 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T02:37:15.367 INFO:teuthology.orchestra.run.smithi098.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T02:37:15.367 INFO:teuthology.orchestra.run.smithi098.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:15.635 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:16.748 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T02:37:16.749 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T02:37:16.749 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T02:37:16.749 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:16.921 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:17.020 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:18.331 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:18.430 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T02:37:18.556 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T02:37:18.707 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:18.775 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T02:37:18.775 INFO:teuthology.orchestra.run.smithi098.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:19.708 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T02:37:19.833 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T02:37:19.960 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T02:37:20.030 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T02:37:20.030 INFO:teuthology.orchestra.run.smithi190.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:33.615 INFO:teuthology.orchestra.run.smithi098.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T02:37:33.616 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub' 2024-09-11T02:37:33.665 INFO:teuthology.orchestra.run.smithi098.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T02:37:33.686 INFO:teuthology.orchestra.run.smithi098.stdout:Generating grub configuration file ... 2024-09-11T02:37:34.044 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T02:37:34.066 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:34.137 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T02:37:34.137 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub' 2024-09-11T02:37:34.156 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T02:37:34.183 INFO:teuthology.orchestra.run.smithi190.stdout:Generating grub configuration file ... 2024-09-11T02:37:34.413 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T02:37:34.419 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T02:37:34.429 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T02:37:34.461 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:34.486 INFO:teuthology.orchestra.run.smithi098.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T02:37:34.492 INFO:teuthology.orchestra.run.smithi098.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T02:37:34.664 INFO:teuthology.orchestra.run.smithi098.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T02:37:34.664 INFO:teuthology.orchestra.run.smithi098.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T02:37:34.664 INFO:teuthology.orchestra.run.smithi098.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T02:37:34.696 INFO:teuthology.orchestra.run.smithi098.stdout:done 2024-09-11T02:37:34.776 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T02:37:34.782 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T02:37:34.859 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T02:37:34.865 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T02:37:35.030 INFO:teuthology.orchestra.run.smithi190.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T02:37:35.031 INFO:teuthology.orchestra.run.smithi190.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T02:37:35.031 INFO:teuthology.orchestra.run.smithi190.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T02:37:35.071 INFO:teuthology.orchestra.run.smithi190.stdout:done 2024-09-11T02:37:35.229 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 21 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-09-11T02:37:35.253 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.15.0.119.119 2024-09-11T02:37:35.254 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T02:37:35.254 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T02:37:35.254 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-09-11T02:37:35.254 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-09-11T02:37:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-09-11T02:37:35.254 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-09-11T02:37:35.254 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T02:37:35.254 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T02:37:35.254 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T02:37:35.254 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T02:37:35.255 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T02:37:35.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get clean 2024-09-11T02:37:35.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-09-11T02:37:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T02:37:35.531 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2024-09-11T02:37:35.555 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2024-09-11T02:37:35.555 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2024-09-11T02:37:35.555 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2024-09-11T02:37:35.555 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2024-09-11T02:37:35.555 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 21 2024-09-11T02:37:35.555 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T02:37:35.555 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.119.119 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2024-09-11T02:37:35.556 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2024-09-11T02:37:35.556 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T02:37:35.556 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T02:37:35.556 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T02:37:35.557 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2024-09-11T02:37:35.557 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T02:37:35.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2024-09-11T02:37:35.629 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T02:37:35.709 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T02:37:35.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-09-11T02:37:35.790 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T02:37:35.871 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T02:37:35.871 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T02:37:35.879 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T02:37:35.896 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T02:37:37.035 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-11T02:37:37.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T02:37:37.117 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-11T02:37:37.233 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T02:37:37.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T02:37:37.314 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T02:37:37.321 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-09-11T02:37:37.322 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-09-11T02:37:37.490 INFO:teuthology.orchestra.run.smithi098.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T02:37:37.490 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T02:37:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T02:37:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T02:37:37.490 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T02:37:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T02:37:37.491 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T02:37:37.491 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T02:37:37.522 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-09-11T02:37:37.523 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-09-11T02:37:37.528 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T02:37:37.530 DEBUG:teuthology.orchestra.run.smithi098:> dpkg -s linux-image-generic 2024-09-11T02:37:37.545 INFO:teuthology.orchestra.run.smithi098.stdout:Package: linux-image-generic 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Status: install ok installed 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Priority: optional 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Section: kernel 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Installed-Size: 21 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: amd64 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Source: linux-meta 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Version: 5.15.0.119.119 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Recommends: thermald 2024-09-11T02:37:37.546 INFO:teuthology.orchestra.run.smithi098.stdout:Description: Generic Linux kernel image 2024-09-11T02:37:37.547 INFO:teuthology.orchestra.run.smithi098.stdout: This package will always depend on the latest generic kernel image 2024-09-11T02:37:37.547 INFO:teuthology.orchestra.run.smithi098.stdout: available. 2024-09-11T02:37:37.547 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-11T02:37:37.595 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.Q7YqFCsh92 2024-09-11T02:37:37.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Q7YqFCsh92 2024-09-11T02:37:37.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.Q7YqFCsh92 2024-09-11T02:37:37.695 INFO:teuthology.orchestra.run.smithi190.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T02:37:37.695 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T02:37:37.695 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T02:37:37.695 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T02:37:37.695 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T02:37:37.695 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T02:37:37.696 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T02:37:37.696 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T02:37:37.734 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T02:37:37.736 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2024-09-11T02:37:37.752 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2024-09-11T02:37:37.752 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2024-09-11T02:37:37.752 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2024-09-11T02:37:37.752 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2024-09-11T02:37:37.752 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 21 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.119.119 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2024-09-11T02:37:37.753 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2024-09-11T02:37:37.754 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2024-09-11T02:37:37.754 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2024-09-11T02:37:37.754 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-11T02:37:37.772 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.Q7YqFCsh92 is 10KB 2024-09-11T02:37:37.784 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.Q7YqFCsh92 2024-09-11T02:37:37.791 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T02:37:37.796 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.idu6OxblA2 2024-09-11T02:37:37.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub/grub.cfg /tmp/tmp.idu6OxblA2 2024-09-11T02:37:37.846 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:37:37.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T02:37:37.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T02:37:37.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.idu6OxblA2 2024-09-11T02:37:37.934 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T02:37:37.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo update-grub 2024-09-11T02:37:37.967 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.idu6OxblA2 is 10KB 2024-09-11T02:37:37.979 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.idu6OxblA2 2024-09-11T02:37:38.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T02:37:38.066 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:37:38.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T02:37:38.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T02:37:38.138 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T02:37:38.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo update-grub 2024-09-11T02:37:39.483 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub' 2024-09-11T02:37:39.484 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T02:37:39.490 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub' 2024-09-11T02:37:39.492 INFO:teuthology.orchestra.run.smithi098.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T02:37:39.495 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-09-11T02:37:39.497 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-09-11T02:37:39.681 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T02:37:39.687 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:39.687 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T02:37:39.693 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T02:37:39.927 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T02:37:39.931 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T02:37:39.933 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T02:37:39.937 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T02:37:39.973 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T02:37:39.978 INFO:teuthology.orchestra.run.smithi098.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T02:37:39.979 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T02:37:39.983 INFO:teuthology.orchestra.run.smithi098.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T02:37:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T02:37:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T02:37:40.103 INFO:teuthology.orchestra.run.smithi098.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T02:37:40.113 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T02:37:40.113 INFO:teuthology.orchestra.run.smithi190.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T02:37:40.113 INFO:teuthology.orchestra.run.smithi190.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T02:37:40.121 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-09-11T02:37:40.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-09-11T02:37:40.124 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-09-11T02:37:40.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-09-11T02:38:10.169 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2024-09-11T02:38:10.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T02:38:10.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:38:10.181 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T02:38:10.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:38:10.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:38:28.669 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-11T02:38:28.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-11T02:38:37.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T02:38:37.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:38:37.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:38:37.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:38:40.769 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-09-11T02:38:40.769 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-11T02:38:52.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T02:38:52.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:38:52.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:38:52.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:39:11.165 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-11T02:39:24.535 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-11T02:39:25.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T02:39:25.126 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.15.0-119-generic"... 2024-09-11T02:39:25.126 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-09-11T02:39:25.171 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-119-generic 2024-09-11T02:39:25.172 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T02:39:25.172 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T02:39:25.172 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T02:39:26.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:39:26.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:39:26.177 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-09-11T02:39:26.178 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T02:39:26.179 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-11T02:39:26.257 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-09-11T02:39:26.277 DEBUG:teuthology.parallel:result is None 2024-09-11T02:39:44.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:39:44.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:39:44.431 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-11T02:39:44.994 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:39:44.994 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-11T02:39:44.994 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-11T02:39:45.043 INFO:teuthology.orchestra.run.smithi098.stdout:5.15.0-119-generic 2024-09-11T02:39:45.043 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T02:39:45.043 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T02:39:45.043 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T02:39:46.044 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T02:39:46.044 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T02:39:46.133 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-09-11T02:39:46.164 DEBUG:teuthology.parallel:result is None 2024-09-11T02:39:46.164 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T02:39:46.173 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T02:39:46.174 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T02:39:46.176 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T02:39:46.182 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T02:39:46.188 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T02:39:46.194 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T02:39:46.194 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T02:39:46.227 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T02:39:46.252 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T02:39:46.259 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T02:39:46.259 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T02:39:46.292 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T02:39:46.310 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T02:39:46.314 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T02:39:46.318 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T02:39:46.323 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T02:39:46.325 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T02:39:46.332 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T02:39:46.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T02:39:46.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T02:39:46.405 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T02:39:46.415 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T02:39:46.415 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T02:39:46.451 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T02:39:46.456 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T02:39:46.503 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T02:39:46.551 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:39:46.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T02:39:46.620 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T02:39:46.626 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T02:39:46.675 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:39:46.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T02:39:46.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-09-11T02:39:46.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-09-11T02:39:46.810 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T02:39:46.817 INFO:teuthology.task.internal:Starting timer... 2024-09-11T02:39:46.818 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T02:39:46.828 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T02:39:46.836 DEBUG:teuthology.task.selinux:Excluding smithi098: OS 'ubuntu' does not support SELinux 2024-09-11T02:39:46.836 DEBUG:teuthology.task.selinux:Excluding smithi190: OS 'ubuntu' does not support SELinux 2024-09-11T02:39:46.836 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T02:39:46.837 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-11T02:39:46.837 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T02:39:46.837 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T02:39:46.845 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T02:39:46.848 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T02:39:46.972 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T02:39:46.983 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T02:39:46.984 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T02:47:18.645 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-09-11T02:47:18.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:47:18.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:47:18.760 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-11T02:47:18.840 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-11T02:47:18.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T02:47:18.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T02:47:18.953 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-09-11T02:47:19.037 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-09-11T02:47:19.037 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T02:47:19.049 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T02:47:19.049 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T02:47:19.050 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T02:47:19.053 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T02:47:19.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T02:47:19.076 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T02:47:19.077 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Command line: ntpd -gq 2024-09-11T02:47:19.077 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: ---------------------------------------------------- 2024-09-11T02:47:19.077 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T02:47:19.077 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T02:47:19.077 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: corporation. Support and training for ntp-4 are 2024-09-11T02:47:19.077 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: available at https://www.nwtime.org/support 2024-09-11T02:47:19.077 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: ---------------------------------------------------- 2024-09-11T02:47:19.078 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: proto: precision = 0.061 usec (-24) 2024-09-11T02:47:19.078 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: basedate set to 2022-02-04 2024-09-11T02:47:19.078 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: gps base set to 2022-02-06 (week 2196) 2024-09-11T02:47:19.079 INFO:teuthology.orchestra.run.smithi098.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T02:47:19.079 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T02:47:19.079 INFO:teuthology.orchestra.run.smithi098.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T02:47:19.080 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T02:47:19.080 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T02:47:19.080 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T02:47:19.080 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T02:47:19.081 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Listen normally on 3 enp3s0f1 172.21.15.98:123 2024-09-11T02:47:19.081 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Listen normally on 4 lo [::1]:123 2024-09-11T02:47:19.081 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d26b%5]:123 2024-09-11T02:47:19.081 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:19 ntpd[18434]: Listening on routing socket on fd #22 for interface updates 2024-09-11T02:47:19.108 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T02:47:19.108 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Command line: ntpd -gq 2024-09-11T02:47:19.109 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: ---------------------------------------------------- 2024-09-11T02:47:19.109 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T02:47:19.109 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T02:47:19.109 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: corporation. Support and training for ntp-4 are 2024-09-11T02:47:19.109 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: available at https://www.nwtime.org/support 2024-09-11T02:47:19.109 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: ---------------------------------------------------- 2024-09-11T02:47:19.110 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: proto: precision = 0.056 usec (-24) 2024-09-11T02:47:19.110 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: basedate set to 2022-02-04 2024-09-11T02:47:19.110 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: gps base set to 2022-02-06 (week 2196) 2024-09-11T02:47:19.110 INFO:teuthology.orchestra.run.smithi190.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T02:47:19.111 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T02:47:19.111 INFO:teuthology.orchestra.run.smithi190.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T02:47:19.112 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T02:47:19.112 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T02:47:19.112 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T02:47:19.113 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T02:47:19.113 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Listen normally on 3 enp3s0f1 172.21.15.190:123 2024-09-11T02:47:19.113 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Listen normally on 4 lo [::1]:123 2024-09-11T02:47:19.113 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7763%5]:123 2024-09-11T02:47:19.113 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:19 ntpd[18423]: Listening on routing socket on fd #22 for interface updates 2024-09-11T02:47:20.077 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:20.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:20 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:21.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:21 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:21.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:21 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:21.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:21 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:21.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:21 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:21.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:21 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:21.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:21 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:22 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:23.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:23.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:23 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:24.080 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:24 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:24.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:24.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:24.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:24.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:24 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:25.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:25.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:25 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:26.078 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:26.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:26.079 INFO:teuthology.orchestra.run.smithi098.stdout:11 Sep 02:47:26 ntpd[18434]: ntpd: time slew +0.001602 s 2024-09-11T02:47:26.079 INFO:teuthology.orchestra.run.smithi098.stdout:ntpd: time slew +0.001602s 2024-09-11T02:47:26.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T02:47:26.079 INFO:teuthology.orchestra.run.smithi098.stderr:11 Sep 02:47:26 ntpd[18434]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T02:47:26.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:26 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:26.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:26 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:26.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:26 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:26.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:26 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:26.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:26 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:26.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:26 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T02:47:26.139 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-09-11T02:47:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:26.139 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:27.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:27.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:27.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:27.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:27.108 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T02:47:27.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T02:47:27.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T02:47:27.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T02:47:27.109 INFO:teuthology.orchestra.run.smithi190.stdout:11 Sep 02:47:27 ntpd[18423]: ntpd: time slew +0.000886 s 2024-09-11T02:47:27.109 INFO:teuthology.orchestra.run.smithi190.stdout:ntpd: time slew +0.000886s 2024-09-11T02:47:27.109 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T02:47:27.110 INFO:teuthology.orchestra.run.smithi190.stderr:11 Sep 02:47:27 ntpd[18423]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T02:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T02:47:27.166 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2024-09-11T02:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:27.167 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T02:47:27.168 INFO:teuthology.run_tasks:Running task install... 2024-09-11T02:47:27.176 DEBUG:teuthology.task.install:project ceph 2024-09-11T02:47:27.176 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}} 2024-09-11T02:47:27.176 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'} 2024-09-11T02:47:27.176 INFO:teuthology.task.install:Using flavor: default 2024-09-11T02:47:27.182 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T02:47:27.182 INFO:teuthology.task.install:extra packages: [] 2024-09-11T02:47:27.183 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-key list | grep Ceph 2024-09-11T02:47:27.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-key list | grep Ceph 2024-09-11T02:47:27.251 INFO:teuthology.orchestra.run.smithi098.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T02:47:27.263 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T02:47:27.280 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T02:47:27.281 INFO:teuthology.orchestra.run.smithi098.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T02:47:27.281 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-11T02:47:27.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T02:47:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T02:47:27.311 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T02:47:27.311 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-11T02:47:27.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T02:47:27.450 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default/ 2024-09-11T02:47:27.450 INFO:teuthology.task.install.deb:Package version is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:47:27.463 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default/ 2024-09-11T02:47:27.463 INFO:teuthology.task.install.deb:Package version is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:47:27.570 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:47:27.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T02:47:27.578 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:47:27.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T02:47:27.584 DEBUG:teuthology.orchestra.run.smithi098:> sudo apt-get update 2024-09-11T02:47:27.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-09-11T02:47:27.756 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T02:47:27.761 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T02:47:27.765 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T02:47:27.772 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T02:47:27.782 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T02:47:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T02:47:27.808 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T02:47:27.815 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T02:47:27.829 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-11T02:47:27.838 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-11T02:47:27.851 INFO:teuthology.orchestra.run.smithi098.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T02:47:27.860 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T02:47:27.872 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-11T02:47:27.881 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-09-11T02:47:27.895 INFO:teuthology.orchestra.run.smithi098.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T02:47:27.895 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T02:47:27.899 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-09-11T02:47:27.907 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-11T02:47:28.100 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 29.3 kB in 0s (89.3 kB/s) 2024-09-11T02:47:28.103 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 29.3 kB in 0s (86.8 kB/s) 2024-09-11T02:47:29.102 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T02:47:29.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4854-g8f219c47-1jammy cephadm=19.3.0-4854-g8f219c47-1jammy ceph-mds=19.3.0-4854-g8f219c47-1jammy ceph-mgr=19.3.0-4854-g8f219c47-1jammy ceph-common=19.3.0-4854-g8f219c47-1jammy ceph-fuse=19.3.0-4854-g8f219c47-1jammy ceph-test=19.3.0-4854-g8f219c47-1jammy ceph-volume=19.3.0-4854-g8f219c47-1jammy radosgw=19.3.0-4854-g8f219c47-1jammy python3-rados=19.3.0-4854-g8f219c47-1jammy python3-rgw=19.3.0-4854-g8f219c47-1jammy python3-cephfs=19.3.0-4854-g8f219c47-1jammy python3-rbd=19.3.0-4854-g8f219c47-1jammy libcephfs2=19.3.0-4854-g8f219c47-1jammy libcephfs-dev=19.3.0-4854-g8f219c47-1jammy librados2=19.3.0-4854-g8f219c47-1jammy librbd1=19.3.0-4854-g8f219c47-1jammy rbd-fuse=19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:47:29.137 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-11T02:47:29.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4854-g8f219c47-1jammy cephadm=19.3.0-4854-g8f219c47-1jammy ceph-mds=19.3.0-4854-g8f219c47-1jammy ceph-mgr=19.3.0-4854-g8f219c47-1jammy ceph-common=19.3.0-4854-g8f219c47-1jammy ceph-fuse=19.3.0-4854-g8f219c47-1jammy ceph-test=19.3.0-4854-g8f219c47-1jammy ceph-volume=19.3.0-4854-g8f219c47-1jammy radosgw=19.3.0-4854-g8f219c47-1jammy python3-rados=19.3.0-4854-g8f219c47-1jammy python3-rgw=19.3.0-4854-g8f219c47-1jammy python3-cephfs=19.3.0-4854-g8f219c47-1jammy python3-rbd=19.3.0-4854-g8f219c47-1jammy libcephfs2=19.3.0-4854-g8f219c47-1jammy libcephfs-dev=19.3.0-4854-g8f219c47-1jammy librados2=19.3.0-4854-g8f219c47-1jammy librbd1=19.3.0-4854-g8f219c47-1jammy rbd-fuse=19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:47:29.180 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-09-11T02:47:29.217 INFO:teuthology.orchestra.run.smithi098.stdout:Reading package lists... 2024-09-11T02:47:29.390 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-09-11T02:47:29.391 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-09-11T02:47:29.425 INFO:teuthology.orchestra.run.smithi098.stdout:Building dependency tree... 2024-09-11T02:47:29.426 INFO:teuthology.orchestra.run.smithi098.stdout:Reading state information... 2024-09-11T02:47:29.584 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T02:47:29.584 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T02:47:29.584 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T02:47:29.584 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T02:47:29.585 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T02:47:29.585 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T02:47:29.585 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T02:47:29.586 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2024-09-11T02:47:29.586 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T02:47:29.586 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T02:47:29.586 INFO:teuthology.orchestra.run.smithi190.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-11T02:47:29.586 INFO:teuthology.orchestra.run.smithi190.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-11T02:47:29.586 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-11T02:47:29.587 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet zip 2024-09-11T02:47:29.588 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2024-09-11T02:47:29.589 INFO:teuthology.orchestra.run.smithi190.stdout: btrfs-tools 2024-09-11T02:47:29.634 INFO:teuthology.orchestra.run.smithi098.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T02:47:29.634 INFO:teuthology.orchestra.run.smithi098.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T02:47:29.635 INFO:teuthology.orchestra.run.smithi098.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T02:47:29.635 INFO:teuthology.orchestra.run.smithi098.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T02:47:29.635 INFO:teuthology.orchestra.run.smithi098.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T02:47:29.635 INFO:teuthology.orchestra.run.smithi098.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T02:47:29.635 INFO:teuthology.orchestra.run.smithi098.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T02:47:29.637 INFO:teuthology.orchestra.run.smithi098.stdout:The following additional packages will be installed: 2024-09-11T02:47:29.637 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T02:47:29.637 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T02:47:29.637 INFO:teuthology.orchestra.run.smithi098.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-11T02:47:29.637 INFO:teuthology.orchestra.run.smithi098.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-11T02:47:29.637 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-11T02:47:29.638 INFO:teuthology.orchestra.run.smithi098.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-11T02:47:29.639 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T02:47:29.640 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet zip 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout:Suggested packages: 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout:Recommended packages: 2024-09-11T02:47:29.641 INFO:teuthology.orchestra.run.smithi098.stdout: btrfs-tools 2024-09-11T02:47:29.692 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2024-09-11T02:47:29.692 INFO:teuthology.orchestra.run.smithi190.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T02:47:29.692 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T02:47:29.692 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T02:47:29.692 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T02:47:29.692 INFO:teuthology.orchestra.run.smithi190.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-11T02:47:29.692 INFO:teuthology.orchestra.run.smithi190.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T02:47:29.693 INFO:teuthology.orchestra.run.smithi190.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T02:47:29.693 INFO:teuthology.orchestra.run.smithi190.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T02:47:29.693 INFO:teuthology.orchestra.run.smithi190.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T02:47:29.693 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T02:47:29.694 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T02:47:29.695 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T02:47:29.695 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T02:47:29.695 INFO:teuthology.orchestra.run.smithi190.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-11T02:47:29.742 INFO:teuthology.orchestra.run.smithi098.stdout:The following NEW packages will be installed: 2024-09-11T02:47:29.743 INFO:teuthology.orchestra.run.smithi098.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T02:47:29.743 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T02:47:29.743 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T02:47:29.743 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T02:47:29.743 INFO:teuthology.orchestra.run.smithi098.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-11T02:47:29.743 INFO:teuthology.orchestra.run.smithi098.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T02:47:29.744 INFO:teuthology.orchestra.run.smithi098.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T02:47:29.744 INFO:teuthology.orchestra.run.smithi098.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T02:47:29.744 INFO:teuthology.orchestra.run.smithi098.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T02:47:29.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T02:47:29.744 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T02:47:29.745 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T02:47:29.746 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T02:47:29.746 INFO:teuthology.orchestra.run.smithi098.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-11T02:47:29.753 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T02:47:29.753 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 251 MB of archives. 2024-09-11T02:47:29.753 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-11T02:47:29.753 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T02:47:29.836 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4854-g8f219c47-1jammy [3,633 kB] 2024-09-11T02:47:29.841 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T02:47:29.845 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T02:47:29.861 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T02:47:29.887 INFO:teuthology.orchestra.run.smithi098.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T02:47:29.887 INFO:teuthology.orchestra.run.smithi098.stdout:Need to get 251 MB of archives. 2024-09-11T02:47:29.888 INFO:teuthology.orchestra.run.smithi098.stdout:After this operation, 934 MB of additional disk space will be used. 2024-09-11T02:47:29.888 INFO:teuthology.orchestra.run.smithi098.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4854-g8f219c47-1jammy [3,633 kB] 2024-09-11T02:47:29.921 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T02:47:29.925 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T02:47:29.932 INFO:teuthology.orchestra.run.smithi098.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T02:47:29.938 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T02:47:29.942 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-11T02:47:29.943 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T02:47:29.944 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T02:47:29.945 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T02:47:29.953 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T02:47:29.955 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T02:47:29.957 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T02:47:29.959 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T02:47:29.959 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T02:47:29.976 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T02:47:29.979 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T02:47:29.980 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T02:47:29.981 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T02:47:29.981 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T02:47:29.982 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T02:47:29.983 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T02:47:29.983 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T02:47:29.993 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T02:47:29.994 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T02:47:30.010 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T02:47:30.011 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T02:47:30.013 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T02:47:30.014 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T02:47:30.014 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T02:47:30.015 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T02:47:30.016 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T02:47:30.016 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T02:47:30.024 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4854-g8f219c47-1jammy [3,330 kB] 2024-09-11T02:47:30.028 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T02:47:30.029 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T02:47:30.045 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T02:47:30.045 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T02:47:30.050 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T02:47:30.051 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T02:47:30.051 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T02:47:30.052 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T02:47:30.053 INFO:teuthology.orchestra.run.smithi190.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T02:47:30.055 INFO:teuthology.orchestra.run.smithi190.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T02:47:30.061 INFO:teuthology.orchestra.run.smithi190.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T02:47:30.063 INFO:teuthology.orchestra.run.smithi190.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T02:47:30.073 INFO:teuthology.orchestra.run.smithi190.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T02:47:30.075 INFO:teuthology.orchestra.run.smithi190.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T02:47:30.075 INFO:teuthology.orchestra.run.smithi190.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T02:47:30.076 INFO:teuthology.orchestra.run.smithi190.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T02:47:30.078 INFO:teuthology.orchestra.run.smithi190.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T02:47:30.089 INFO:teuthology.orchestra.run.smithi098.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4854-g8f219c47-1jammy [3,330 kB] 2024-09-11T02:47:30.117 INFO:teuthology.orchestra.run.smithi190.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4854-g8f219c47-1jammy [721 kB] 2024-09-11T02:47:30.129 INFO:teuthology.orchestra.run.smithi190.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T02:47:30.130 INFO:teuthology.orchestra.run.smithi190.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T02:47:30.131 INFO:teuthology.orchestra.run.smithi190.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T02:47:30.132 INFO:teuthology.orchestra.run.smithi190.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4854-g8f219c47-1jammy [356 kB] 2024-09-11T02:47:30.132 INFO:teuthology.orchestra.run.smithi190.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T02:47:30.133 INFO:teuthology.orchestra.run.smithi190.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T02:47:30.139 INFO:teuthology.orchestra.run.smithi190.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4854-g8f219c47-1jammy [32.8 kB] 2024-09-11T02:47:30.140 INFO:teuthology.orchestra.run.smithi190.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4854-g8f219c47-1jammy [180 kB] 2024-09-11T02:47:30.144 INFO:teuthology.orchestra.run.smithi190.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4854-g8f219c47-1jammy [71.2 kB] 2024-09-11T02:47:30.145 INFO:teuthology.orchestra.run.smithi190.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4854-g8f219c47-1jammy [338 kB] 2024-09-11T02:47:30.149 INFO:teuthology.orchestra.run.smithi190.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T02:47:30.151 INFO:teuthology.orchestra.run.smithi190.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T02:47:30.151 INFO:teuthology.orchestra.run.smithi190.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T02:47:30.165 INFO:teuthology.orchestra.run.smithi190.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T02:47:30.165 INFO:teuthology.orchestra.run.smithi190.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T02:47:30.167 INFO:teuthology.orchestra.run.smithi190.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T02:47:30.168 INFO:teuthology.orchestra.run.smithi190.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T02:47:30.168 INFO:teuthology.orchestra.run.smithi190.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T02:47:30.169 INFO:teuthology.orchestra.run.smithi190.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T02:47:30.171 INFO:teuthology.orchestra.run.smithi190.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4854-g8f219c47-1jammy [7,988 kB] 2024-09-11T02:47:30.185 INFO:teuthology.orchestra.run.smithi190.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T02:47:30.185 INFO:teuthology.orchestra.run.smithi190.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T02:47:30.188 INFO:teuthology.orchestra.run.smithi190.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T02:47:30.190 INFO:teuthology.orchestra.run.smithi190.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T02:47:30.191 INFO:teuthology.orchestra.run.smithi190.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T02:47:30.191 INFO:teuthology.orchestra.run.smithi190.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T02:47:30.200 INFO:teuthology.orchestra.run.smithi098.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4854-g8f219c47-1jammy [721 kB] 2024-09-11T02:47:30.202 INFO:teuthology.orchestra.run.smithi190.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T02:47:30.204 INFO:teuthology.orchestra.run.smithi190.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T02:47:30.205 INFO:teuthology.orchestra.run.smithi190.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T02:47:30.206 INFO:teuthology.orchestra.run.smithi190.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T02:47:30.214 INFO:teuthology.orchestra.run.smithi098.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4854-g8f219c47-1jammy [356 kB] 2024-09-11T02:47:30.219 INFO:teuthology.orchestra.run.smithi190.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T02:47:30.220 INFO:teuthology.orchestra.run.smithi190.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T02:47:30.223 INFO:teuthology.orchestra.run.smithi190.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T02:47:30.223 INFO:teuthology.orchestra.run.smithi190.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T02:47:30.223 INFO:teuthology.orchestra.run.smithi190.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T02:47:30.223 INFO:teuthology.orchestra.run.smithi190.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T02:47:30.236 INFO:teuthology.orchestra.run.smithi190.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T02:47:30.237 INFO:teuthology.orchestra.run.smithi190.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T02:47:30.242 INFO:teuthology.orchestra.run.smithi098.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4854-g8f219c47-1jammy [32.8 kB] 2024-09-11T02:47:30.243 INFO:teuthology.orchestra.run.smithi190.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T02:47:30.243 INFO:teuthology.orchestra.run.smithi190.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T02:47:30.244 INFO:teuthology.orchestra.run.smithi098.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4854-g8f219c47-1jammy [180 kB] 2024-09-11T02:47:30.248 INFO:teuthology.orchestra.run.smithi098.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4854-g8f219c47-1jammy [71.2 kB] 2024-09-11T02:47:30.249 INFO:teuthology.orchestra.run.smithi098.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4854-g8f219c47-1jammy [338 kB] 2024-09-11T02:47:30.253 INFO:teuthology.orchestra.run.smithi190.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T02:47:30.255 INFO:teuthology.orchestra.run.smithi190.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T02:47:30.255 INFO:teuthology.orchestra.run.smithi190.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T02:47:30.255 INFO:teuthology.orchestra.run.smithi098.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4854-g8f219c47-1jammy [7,988 kB] 2024-09-11T02:47:30.355 INFO:teuthology.orchestra.run.smithi098.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T02:47:30.367 INFO:teuthology.orchestra.run.smithi098.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T02:47:30.413 INFO:teuthology.orchestra.run.smithi190.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4854-g8f219c47-1jammy [112 kB] 2024-09-11T02:47:30.415 INFO:teuthology.orchestra.run.smithi190.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4854-g8f219c47-1jammy [473 kB] 2024-09-11T02:47:30.424 INFO:teuthology.orchestra.run.smithi190.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4854-g8f219c47-1jammy [28.3 MB] 2024-09-11T02:47:30.448 INFO:teuthology.orchestra.run.smithi098.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T02:47:30.519 INFO:teuthology.orchestra.run.smithi098.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4854-g8f219c47-1jammy [112 kB] 2024-09-11T02:47:30.521 INFO:teuthology.orchestra.run.smithi098.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4854-g8f219c47-1jammy [473 kB] 2024-09-11T02:47:30.559 INFO:teuthology.orchestra.run.smithi098.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4854-g8f219c47-1jammy [28.3 MB] 2024-09-11T02:47:30.678 INFO:teuthology.orchestra.run.smithi098.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T02:47:30.689 INFO:teuthology.orchestra.run.smithi098.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T02:47:30.718 INFO:teuthology.orchestra.run.smithi098.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T02:47:30.727 INFO:teuthology.orchestra.run.smithi098.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-11T02:47:30.729 INFO:teuthology.orchestra.run.smithi098.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T02:47:30.731 INFO:teuthology.orchestra.run.smithi098.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T02:47:30.732 INFO:teuthology.orchestra.run.smithi098.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T02:47:30.749 INFO:teuthology.orchestra.run.smithi098.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T02:47:30.753 INFO:teuthology.orchestra.run.smithi098.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T02:47:30.760 INFO:teuthology.orchestra.run.smithi098.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T02:47:30.839 INFO:teuthology.orchestra.run.smithi098.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T02:47:30.840 INFO:teuthology.orchestra.run.smithi098.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T02:47:30.844 INFO:teuthology.orchestra.run.smithi098.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T02:47:30.849 INFO:teuthology.orchestra.run.smithi098.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T02:47:30.852 INFO:teuthology.orchestra.run.smithi098.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T02:47:30.853 INFO:teuthology.orchestra.run.smithi098.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T02:47:30.854 INFO:teuthology.orchestra.run.smithi098.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T02:47:30.856 INFO:teuthology.orchestra.run.smithi098.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T02:47:30.856 INFO:teuthology.orchestra.run.smithi098.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T02:47:30.923 INFO:teuthology.orchestra.run.smithi098.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T02:47:30.939 INFO:teuthology.orchestra.run.smithi098.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T02:47:30.940 INFO:teuthology.orchestra.run.smithi098.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T02:47:30.940 INFO:teuthology.orchestra.run.smithi098.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T02:47:30.941 INFO:teuthology.orchestra.run.smithi098.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T02:47:30.946 INFO:teuthology.orchestra.run.smithi098.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T02:47:30.947 INFO:teuthology.orchestra.run.smithi098.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T02:47:31.005 INFO:teuthology.orchestra.run.smithi098.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T02:47:31.006 INFO:teuthology.orchestra.run.smithi098.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T02:47:31.007 INFO:teuthology.orchestra.run.smithi098.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T02:47:31.086 INFO:teuthology.orchestra.run.smithi098.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T02:47:31.086 INFO:teuthology.orchestra.run.smithi098.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T02:47:31.089 INFO:teuthology.orchestra.run.smithi098.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T02:47:31.090 INFO:teuthology.orchestra.run.smithi098.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T02:47:31.091 INFO:teuthology.orchestra.run.smithi098.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T02:47:31.102 INFO:teuthology.orchestra.run.smithi098.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T02:47:31.167 INFO:teuthology.orchestra.run.smithi098.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T02:47:31.168 INFO:teuthology.orchestra.run.smithi098.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T02:47:31.169 INFO:teuthology.orchestra.run.smithi098.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T02:47:31.172 INFO:teuthology.orchestra.run.smithi098.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T02:47:31.249 INFO:teuthology.orchestra.run.smithi098.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T02:47:31.250 INFO:teuthology.orchestra.run.smithi098.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T02:47:31.253 INFO:teuthology.orchestra.run.smithi098.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T02:47:31.254 INFO:teuthology.orchestra.run.smithi098.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T02:47:31.259 INFO:teuthology.orchestra.run.smithi098.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T02:47:31.260 INFO:teuthology.orchestra.run.smithi098.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T02:47:31.330 INFO:teuthology.orchestra.run.smithi098.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T02:47:31.335 INFO:teuthology.orchestra.run.smithi098.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T02:47:31.404 INFO:teuthology.orchestra.run.smithi190.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4854-g8f219c47-1jammy [5,206 kB] 2024-09-11T02:47:31.483 INFO:teuthology.orchestra.run.smithi190.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4854-g8f219c47-1jammy [251 kB] 2024-09-11T02:47:31.490 INFO:teuthology.orchestra.run.smithi190.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4854-g8f219c47-1jammy [124 kB] 2024-09-11T02:47:31.492 INFO:teuthology.orchestra.run.smithi190.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4854-g8f219c47-1jammy [1,562 kB] 2024-09-11T02:47:31.498 INFO:teuthology.orchestra.run.smithi098.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T02:47:31.500 INFO:teuthology.orchestra.run.smithi098.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T02:47:31.502 INFO:teuthology.orchestra.run.smithi098.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T02:47:31.502 INFO:teuthology.orchestra.run.smithi098.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T02:47:31.504 INFO:teuthology.orchestra.run.smithi098.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T02:47:31.538 INFO:teuthology.orchestra.run.smithi190.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4854-g8f219c47-1jammy [6,308 kB] 2024-09-11T02:47:31.602 INFO:teuthology.orchestra.run.smithi098.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T02:47:31.691 INFO:teuthology.orchestra.run.smithi098.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T02:47:31.691 INFO:teuthology.orchestra.run.smithi098.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T02:47:31.691 INFO:teuthology.orchestra.run.smithi098.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4854-g8f219c47-1jammy [5,206 kB] 2024-09-11T02:47:31.698 INFO:teuthology.orchestra.run.smithi190.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4854-g8f219c47-1jammy [23.1 MB] 2024-09-11T02:47:31.722 INFO:teuthology.orchestra.run.smithi098.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T02:47:31.723 INFO:teuthology.orchestra.run.smithi098.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T02:47:31.724 INFO:teuthology.orchestra.run.smithi098.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T02:47:31.725 INFO:teuthology.orchestra.run.smithi098.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T02:47:31.726 INFO:teuthology.orchestra.run.smithi098.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T02:47:31.727 INFO:teuthology.orchestra.run.smithi098.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T02:47:31.746 INFO:teuthology.orchestra.run.smithi098.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T02:47:31.784 INFO:teuthology.orchestra.run.smithi098.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4854-g8f219c47-1jammy [251 kB] 2024-09-11T02:47:31.787 INFO:teuthology.orchestra.run.smithi098.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4854-g8f219c47-1jammy [124 kB] 2024-09-11T02:47:31.790 INFO:teuthology.orchestra.run.smithi098.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4854-g8f219c47-1jammy [1,562 kB] 2024-09-11T02:47:31.809 INFO:teuthology.orchestra.run.smithi098.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T02:47:31.818 INFO:teuthology.orchestra.run.smithi098.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T02:47:31.822 INFO:teuthology.orchestra.run.smithi098.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T02:47:31.826 INFO:teuthology.orchestra.run.smithi098.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T02:47:31.827 INFO:teuthology.orchestra.run.smithi098.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T02:47:31.836 INFO:teuthology.orchestra.run.smithi098.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T02:47:31.842 INFO:teuthology.orchestra.run.smithi098.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4854-g8f219c47-1jammy [6,308 kB] 2024-09-11T02:47:31.843 INFO:teuthology.orchestra.run.smithi098.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T02:47:31.845 INFO:teuthology.orchestra.run.smithi098.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T02:47:31.846 INFO:teuthology.orchestra.run.smithi098.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T02:47:31.909 INFO:teuthology.orchestra.run.smithi098.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T02:47:31.910 INFO:teuthology.orchestra.run.smithi098.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T02:47:31.917 INFO:teuthology.orchestra.run.smithi098.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T02:47:31.990 INFO:teuthology.orchestra.run.smithi098.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T02:47:31.992 INFO:teuthology.orchestra.run.smithi098.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T02:47:31.993 INFO:teuthology.orchestra.run.smithi098.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T02:47:31.997 INFO:teuthology.orchestra.run.smithi098.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T02:47:31.998 INFO:teuthology.orchestra.run.smithi098.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T02:47:32.016 INFO:teuthology.orchestra.run.smithi098.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T02:47:32.017 INFO:teuthology.orchestra.run.smithi098.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T02:47:32.072 INFO:teuthology.orchestra.run.smithi098.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T02:47:32.077 INFO:teuthology.orchestra.run.smithi098.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T02:47:32.078 INFO:teuthology.orchestra.run.smithi098.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T02:47:32.079 INFO:teuthology.orchestra.run.smithi098.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4854-g8f219c47-1jammy [23.1 MB] 2024-09-11T02:47:32.468 INFO:teuthology.orchestra.run.smithi190.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4854-g8f219c47-1jammy [14.0 kB] 2024-09-11T02:47:32.468 INFO:teuthology.orchestra.run.smithi190.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4854-g8f219c47-1jammy [880 kB] 2024-09-11T02:47:32.484 INFO:teuthology.orchestra.run.smithi190.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4854-g8f219c47-1jammy [2,454 kB] 2024-09-11T02:47:32.524 INFO:teuthology.orchestra.run.smithi190.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4854-g8f219c47-1jammy [813 kB] 2024-09-11T02:47:32.534 INFO:teuthology.orchestra.run.smithi190.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4854-g8f219c47-1jammy [160 kB] 2024-09-11T02:47:32.536 INFO:teuthology.orchestra.run.smithi190.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4854-g8f219c47-1jammy [61.4 MB] 2024-09-11T02:47:32.949 INFO:teuthology.orchestra.run.smithi098.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4854-g8f219c47-1jammy [14.0 kB] 2024-09-11T02:47:33.075 INFO:teuthology.orchestra.run.smithi098.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4854-g8f219c47-1jammy [880 kB] 2024-09-11T02:47:33.076 INFO:teuthology.orchestra.run.smithi098.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4854-g8f219c47-1jammy [2,454 kB] 2024-09-11T02:47:33.076 INFO:teuthology.orchestra.run.smithi098.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4854-g8f219c47-1jammy [813 kB] 2024-09-11T02:47:33.076 INFO:teuthology.orchestra.run.smithi098.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4854-g8f219c47-1jammy [160 kB] 2024-09-11T02:47:33.076 INFO:teuthology.orchestra.run.smithi098.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4854-g8f219c47-1jammy [61.4 MB] 2024-09-11T02:47:34.686 INFO:teuthology.orchestra.run.smithi190.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4854-g8f219c47-1jammy [8,625 kB] 2024-09-11T02:47:34.871 INFO:teuthology.orchestra.run.smithi190.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4854-g8f219c47-1jammy [14.2 kB] 2024-09-11T02:47:34.872 INFO:teuthology.orchestra.run.smithi190.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4854-g8f219c47-1jammy [55.8 MB] 2024-09-11T02:47:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4854-g8f219c47-1jammy [8,625 kB] 2024-09-11T02:47:35.501 INFO:teuthology.orchestra.run.smithi098.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4854-g8f219c47-1jammy [14.2 kB] 2024-09-11T02:47:35.501 INFO:teuthology.orchestra.run.smithi098.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4854-g8f219c47-1jammy [55.8 MB] 2024-09-11T02:47:36.661 INFO:teuthology.orchestra.run.smithi190.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4854-g8f219c47-1jammy [130 kB] 2024-09-11T02:47:36.664 INFO:teuthology.orchestra.run.smithi190.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4854-g8f219c47-1jammy [39.0 kB] 2024-09-11T02:47:36.665 INFO:teuthology.orchestra.run.smithi190.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4854-g8f219c47-1jammy [15.9 MB] 2024-09-11T02:47:37.160 INFO:teuthology.orchestra.run.smithi190.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4854-g8f219c47-1jammy [91.2 kB] 2024-09-11T02:47:37.401 INFO:teuthology.orchestra.run.smithi098.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4854-g8f219c47-1jammy [130 kB] 2024-09-11T02:47:37.403 INFO:teuthology.orchestra.run.smithi098.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4854-g8f219c47-1jammy [39.0 kB] 2024-09-11T02:47:37.470 INFO:teuthology.orchestra.run.smithi098.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4854-g8f219c47-1jammy [15.9 MB] 2024-09-11T02:47:37.736 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 251 MB in 7s (33.7 MB/s) 2024-09-11T02:47:37.778 INFO:teuthology.orchestra.run.smithi098.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-hemanth-testing-2024-09-10-0723/8f219c476cb4f26fc5719c661c0323719a45cb30/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4854-g8f219c47-1jammy [91.2 kB] 2024-09-11T02:47:37.896 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T02:47:37.996 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T02:47:38.002 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T02:47:38.069 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T02:47:38.349 INFO:teuthology.orchestra.run.smithi098.stdout:Fetched 251 MB in 8s (31.3 MB/s) 2024-09-11T02:47:38.372 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T02:47:38.398 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T02:47:38.439 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T02:47:38.509 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T02:47:38.607 INFO:teuthology.orchestra.run.smithi098.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T02:47:38.612 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T02:47:38.700 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T02:47:38.767 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T02:47:38.793 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T02:47:38.833 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T02:47:39.036 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T02:47:39.063 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T02:47:39.103 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T02:47:39.295 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T02:47:39.321 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T02:47:39.370 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:47:39.398 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T02:47:39.425 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T02:47:39.456 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T02:47:39.706 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T02:47:39.733 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T02:47:39.773 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:47:39.885 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T02:47:39.911 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T02:47:39.959 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:47:40.118 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T02:47:40.144 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T02:47:40.184 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:47:40.355 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T02:47:40.381 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T02:47:40.421 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:47:40.546 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T02:47:40.572 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T02:47:40.612 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T02:47:40.898 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librados2. 2024-09-11T02:47:40.924 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../007-librados2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:40.950 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T02:47:40.956 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:40.976 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T02:47:41.016 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:47:41.328 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T02:47:41.356 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T02:47:41.394 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T02:47:41.426 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libnbd0. 2024-09-11T02:47:41.452 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-11T02:47:41.517 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-11T02:47:41.730 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librados2. 2024-09-11T02:47:41.757 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../007-librados2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:41.797 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:41.829 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librbd1. 2024-09-11T02:47:41.855 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../009-librbd1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:41.895 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:42.267 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libnbd0. 2024-09-11T02:47:42.293 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-11T02:47:42.333 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-11T02:47:42.551 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T02:47:42.579 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:42.636 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librbd1. 2024-09-11T02:47:42.663 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../009-librbd1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:42.703 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:42.739 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:42.992 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rados. 2024-09-11T02:47:43.018 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:43.059 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:43.122 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T02:47:43.138 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:43.173 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:43.395 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T02:47:43.400 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rados. 2024-09-11T02:47:43.416 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:43.421 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:47:43.450 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:43.461 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:43.714 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T02:47:43.729 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:43.761 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T02:47:43.772 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:43.788 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:47:43.828 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:44.041 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T02:47:44.067 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:47:44.080 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T02:47:44.099 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:44.107 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:44.147 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:44.386 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T02:47:44.412 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T02:47:44.425 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T02:47:44.451 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:47:44.452 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T02:47:44.492 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:44.761 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T02:47:44.777 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T02:47:44.780 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T02:47:44.806 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T02:47:44.811 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T02:47:44.847 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T02:47:45.106 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T02:47:45.115 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T02:47:45.130 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:45.132 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T02:47:45.172 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T02:47:45.174 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:45.434 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T02:47:45.460 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:45.485 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T02:47:45.500 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:45.511 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T02:47:45.552 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T02:47:45.820 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T02:47:45.829 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T02:47:45.847 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T02:47:45.856 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T02:47:45.878 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T02:47:45.895 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T02:47:46.157 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T02:47:46.180 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T02:47:46.215 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T02:47:46.264 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T02:47:46.291 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T02:47:46.339 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T02:47:46.543 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua5.1. 2024-09-11T02:47:46.570 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T02:47:46.610 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T02:47:46.617 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T02:47:46.632 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T02:47:46.676 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T02:47:46.929 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-any. 2024-09-11T02:47:46.956 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-11T02:47:46.971 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua5.1. 2024-09-11T02:47:46.996 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T02:47:46.997 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T02:47:47.037 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T02:47:47.207 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package zip. 2024-09-11T02:47:47.222 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-11T02:47:47.257 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T02:47:47.357 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-any. 2024-09-11T02:47:47.372 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-11T02:47:47.416 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T02:47:47.560 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package unzip. 2024-09-11T02:47:47.587 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T02:47:47.626 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T02:47:47.652 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package zip. 2024-09-11T02:47:47.667 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-11T02:47:47.710 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T02:47:47.988 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package unzip. 2024-09-11T02:47:48.014 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T02:47:48.022 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package luarocks. 2024-09-11T02:47:48.046 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T02:47:48.049 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T02:47:48.089 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T02:47:48.472 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librgw2. 2024-09-11T02:47:48.475 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package luarocks. 2024-09-11T02:47:48.498 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../026-librgw2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:48.502 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T02:47:48.538 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:48.542 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T02:47:48.942 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package librgw2. 2024-09-11T02:47:48.968 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../026-librgw2_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:49.008 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:49.106 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T02:47:49.122 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:49.156 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:49.459 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T02:47:49.485 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T02:47:49.526 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T02:47:49.619 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T02:47:49.634 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:49.669 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:49.821 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T02:47:49.847 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:49.887 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:49.939 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T02:47:49.965 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T02:47:50.005 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T02:47:50.317 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T02:47:50.323 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-common. 2024-09-11T02:47:50.344 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:50.350 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:50.384 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:50.390 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:50.746 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-common. 2024-09-11T02:47:50.772 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:50.812 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:51.962 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-base. 2024-09-11T02:47:51.988 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:52.038 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:52.309 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-base. 2024-09-11T02:47:52.336 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:47:52.386 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:47:52.816 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T02:47:52.842 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T02:47:52.882 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T02:47:52.930 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T02:47:52.957 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T02:47:52.997 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T02:47:53.252 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T02:47:53.258 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T02:47:53.274 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T02:47:53.278 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T02:47:53.308 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T02:47:53.326 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T02:47:53.603 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T02:47:53.629 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T02:47:53.661 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T02:47:53.681 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T02:47:53.708 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T02:47:53.748 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T02:47:53.939 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T02:47:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T02:47:54.006 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T02:47:54.051 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T02:47:54.066 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T02:47:54.109 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T02:47:54.309 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T02:47:54.334 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T02:47:54.376 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T02:47:54.420 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T02:47:54.446 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T02:47:54.495 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T02:47:54.653 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T02:47:54.680 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T02:47:54.720 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T02:47:54.865 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T02:47:54.891 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T02:47:54.939 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T02:47:54.973 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-portend. 2024-09-11T02:47:54.999 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-11T02:47:55.039 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T02:47:55.251 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-portend. 2024-09-11T02:47:55.277 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-11T02:47:55.317 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T02:47:55.351 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T02:47:55.378 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T02:47:55.418 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T02:47:55.595 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T02:47:55.621 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T02:47:55.661 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T02:47:55.721 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T02:47:55.748 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T02:47:55.787 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T02:47:56.006 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T02:47:56.033 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T02:47:56.081 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T02:47:56.166 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T02:47:56.193 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T02:47:56.233 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T02:47:56.384 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T02:47:56.400 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T02:47:56.434 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T02:47:56.544 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T02:47:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T02:47:56.611 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T02:47:56.762 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T02:47:56.789 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T02:47:56.829 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T02:47:57.157 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T02:47:57.180 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T02:47:57.183 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T02:47:57.208 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T02:47:57.223 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T02:47:57.255 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T02:47:57.526 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-mako. 2024-09-11T02:47:57.533 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-mako. 2024-09-11T02:47:57.552 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T02:47:57.560 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T02:47:57.592 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T02:47:57.600 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T02:47:57.887 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T02:47:57.896 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T02:47:57.902 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T02:47:57.923 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T02:47:57.936 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T02:47:57.954 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T02:47:58.173 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T02:47:58.200 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T02:47:58.239 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T02:47:58.240 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T02:47:58.267 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T02:47:58.306 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T02:47:58.551 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webob. 2024-09-11T02:47:58.578 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T02:47:58.617 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T02:47:58.618 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webob. 2024-09-11T02:47:58.644 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T02:47:58.684 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T02:47:58.962 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T02:47:58.989 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T02:47:59.029 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T02:47:59.031 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T02:47:59.055 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T02:47:59.099 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T02:47:59.340 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T02:47:59.367 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T02:47:59.407 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T02:47:59.415 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T02:47:59.442 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T02:47:59.481 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T02:47:59.635 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-paste. 2024-09-11T02:47:59.650 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T02:47:59.685 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T02:47:59.751 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-paste. 2024-09-11T02:47:59.778 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T02:47:59.809 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T02:48:00.005 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T02:48:00.020 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T02:48:00.063 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T02:48:00.129 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T02:48:00.144 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T02:48:00.212 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T02:48:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T02:48:00.393 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T02:48:00.449 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T02:48:00.473 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T02:48:00.489 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T02:48:00.523 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T02:48:00.736 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T02:48:00.763 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T02:48:00.802 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T02:48:00.818 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T02:48:00.844 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T02:48:00.884 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T02:48:01.072 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T02:48:01.087 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T02:48:01.122 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T02:48:01.170 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T02:48:01.197 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T02:48:01.237 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T02:48:01.427 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T02:48:01.443 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T02:48:01.477 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T02:48:01.600 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T02:48:01.626 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T02:48:01.666 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T02:48:01.780 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-idna. 2024-09-11T02:48:01.808 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T02:48:01.846 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T02:48:02.011 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-idna. 2024-09-11T02:48:02.037 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T02:48:02.077 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T02:48:02.372 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T02:48:02.399 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T02:48:02.430 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T02:48:02.708 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests. 2024-09-11T02:48:02.725 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T02:48:02.735 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T02:48:02.752 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T02:48:02.933 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T02:48:03.042 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T02:48:03.304 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests. 2024-09-11T02:48:03.330 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T02:48:03.362 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T02:48:03.690 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T02:48:03.716 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T02:48:03.756 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T02:48:04.053 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T02:48:04.070 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T02:48:04.080 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:04.085 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T02:48:04.119 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T02:48:04.120 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:04.431 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T02:48:04.458 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:04.498 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:04.500 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T02:48:04.526 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:04.566 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:04.878 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T02:48:04.885 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T02:48:04.904 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T02:48:04.912 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:04.944 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T02:48:04.952 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:05.214 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T02:48:05.241 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T02:48:05.255 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T02:48:05.272 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T02:48:05.284 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T02:48:05.330 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T02:48:05.583 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T02:48:05.608 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T02:48:05.609 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T02:48:05.635 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:05.649 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T02:48:05.675 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:05.994 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T02:48:06.021 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:06.060 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:06.070 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T02:48:06.097 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:06.136 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:06.446 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T02:48:06.474 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:06.513 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:06.740 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T02:48:06.767 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:06.806 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:07.067 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T02:48:07.094 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:07.124 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:07.885 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph. 2024-09-11T02:48:07.900 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../067-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:07.935 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:08.194 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph. 2024-09-11T02:48:08.221 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../067-ceph_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:08.261 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:08.263 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T02:48:08.290 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:08.329 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:08.613 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T02:48:08.641 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:08.688 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:08.699 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T02:48:08.726 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:08.765 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:09.058 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T02:48:09.085 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:09.124 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:09.152 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package cephadm. 2024-09-11T02:48:09.179 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../070-cephadm_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:09.218 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:09.485 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package cephadm. 2024-09-11T02:48:09.501 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../070-cephadm_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:09.530 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T02:48:09.543 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:09.556 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T02:48:09.588 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T02:48:09.838 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T02:48:09.865 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T02:48:09.890 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T02:48:09.904 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T02:48:09.917 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T02:48:09.957 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:10.215 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T02:48:10.231 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T02:48:10.274 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:10.971 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-babel. 2024-09-11T02:48:10.998 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T02:48:11.037 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:11.353 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-babel. 2024-09-11T02:48:11.381 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T02:48:11.407 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T02:48:11.420 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:11.434 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T02:48:11.473 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T02:48:11.731 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T02:48:11.747 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T02:48:11.781 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T02:48:11.810 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T02:48:11.837 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:11.876 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:12.109 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T02:48:12.136 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:12.175 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:12.213 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T02:48:12.240 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T02:48:12.281 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T02:48:12.545 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T02:48:12.573 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T02:48:12.583 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-routes. 2024-09-11T02:48:12.610 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T02:48:12.611 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T02:48:12.649 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T02:48:12.890 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-routes. 2024-09-11T02:48:12.916 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T02:48:12.948 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T02:48:12.977 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T02:48:13.004 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:13.044 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:13.259 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T02:48:13.272 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:13.317 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:15.783 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T02:48:15.800 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T02:48:15.841 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T02:48:16.005 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T02:48:16.035 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T02:48:16.113 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T02:48:16.336 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T02:48:16.364 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T02:48:16.403 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T02:48:16.592 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T02:48:16.620 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T02:48:16.658 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T02:48:16.740 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T02:48:16.767 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T02:48:16.806 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T02:48:16.978 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T02:48:17.005 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T02:48:17.044 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T02:48:17.084 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T02:48:17.111 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T02:48:17.236 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T02:48:17.364 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T02:48:17.392 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T02:48:17.524 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T02:48:18.188 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T02:48:18.217 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:18.242 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T02:48:18.262 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:18.270 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:18.308 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:19.291 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T02:48:19.318 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T02:48:19.357 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T02:48:19.469 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T02:48:19.497 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T02:48:19.527 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T02:48:19.644 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T02:48:19.671 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T02:48:19.710 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T02:48:19.805 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T02:48:19.833 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T02:48:19.880 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T02:48:20.055 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T02:48:20.083 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T02:48:20.122 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T02:48:20.217 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T02:48:20.245 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T02:48:20.283 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T02:48:20.417 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T02:48:20.444 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-11T02:48:20.483 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T02:48:20.595 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T02:48:20.623 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-11T02:48:20.661 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T02:48:20.779 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T02:48:20.806 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T02:48:20.845 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T02:48:20.965 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T02:48:20.993 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T02:48:21.031 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T02:48:21.165 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T02:48:21.193 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T02:48:21.231 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T02:48:21.317 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T02:48:21.334 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T02:48:21.367 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T02:48:21.543 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T02:48:21.571 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T02:48:21.610 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T02:48:21.612 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T02:48:21.637 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T02:48:21.678 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T02:48:21.913 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T02:48:21.941 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T02:48:21.979 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T02:48:21.999 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T02:48:22.026 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T02:48:22.065 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T02:48:22.644 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T02:48:22.672 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:22.710 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:22.720 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T02:48:22.736 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:22.778 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:23.072 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T02:48:23.098 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T02:48:23.099 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T02:48:23.126 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T02:48:23.138 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T02:48:23.164 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T02:48:23.459 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T02:48:23.487 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T02:48:23.525 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T02:48:23.552 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T02:48:23.591 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T02:48:23.617 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T02:48:23.869 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package jq. 2024-09-11T02:48:23.897 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T02:48:23.935 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T02:48:23.936 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package jq. 2024-09-11T02:48:23.965 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T02:48:24.003 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T02:48:24.163 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package socat. 2024-09-11T02:48:24.179 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T02:48:24.213 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T02:48:24.255 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package socat. 2024-09-11T02:48:24.283 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T02:48:24.332 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T02:48:24.499 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T02:48:24.516 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T02:48:24.557 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T02:48:24.642 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T02:48:24.670 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T02:48:24.708 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T02:48:24.912 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-test. 2024-09-11T02:48:24.929 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:24.962 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:25.046 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-test. 2024-09-11T02:48:25.074 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:25.104 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:27.466 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T02:48:27.483 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:27.516 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:27.625 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T02:48:27.653 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4854-g8f219c47-1jammy_all.deb ... 2024-09-11T02:48:27.691 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:27.821 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T02:48:27.849 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:27.879 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:28.003 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T02:48:28.032 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:28.070 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:28.157 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T02:48:28.185 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T02:48:28.224 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T02:48:28.373 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T02:48:28.401 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T02:48:28.432 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T02:48:28.561 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T02:48:28.589 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T02:48:28.627 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T02:48:28.735 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T02:48:28.763 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T02:48:28.802 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T02:48:28.990 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T02:48:29.018 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T02:48:29.056 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T02:48:29.172 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T02:48:29.200 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T02:48:29.239 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T02:48:29.478 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package pkg-config. 2024-09-11T02:48:29.506 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T02:48:29.553 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T02:48:29.660 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package pkg-config. 2024-09-11T02:48:29.688 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T02:48:29.727 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T02:48:29.781 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T02:48:29.798 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T02:48:29.831 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T02:48:29.980 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T02:48:29.996 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T02:48:30.029 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T02:48:30.201 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T02:48:30.217 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T02:48:30.251 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T02:48:30.441 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T02:48:30.469 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T02:48:30.507 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T02:48:30.579 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T02:48:30.607 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T02:48:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T02:48:30.744 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T02:48:30.771 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T02:48:30.801 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T02:48:30.966 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T02:48:30.994 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T02:48:31.032 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T02:48:31.097 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T02:48:31.124 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T02:48:31.163 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T02:48:31.344 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T02:48:31.371 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T02:48:31.410 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T02:48:31.499 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T02:48:31.527 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T02:48:31.566 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T02:48:31.755 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-py. 2024-09-11T02:48:31.783 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-11T02:48:31.830 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T02:48:31.894 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-py. 2024-09-11T02:48:31.922 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-11T02:48:31.960 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T02:48:32.150 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T02:48:32.178 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T02:48:32.216 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T02:48:32.288 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T02:48:32.317 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T02:48:32.355 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T02:48:32.695 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T02:48:32.723 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T02:48:32.762 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T02:48:32.834 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T02:48:32.863 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T02:48:32.900 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T02:48:33.023 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-toml. 2024-09-11T02:48:33.051 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-11T02:48:33.089 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T02:48:33.211 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-toml. 2024-09-11T02:48:33.240 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-11T02:48:33.278 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T02:48:33.376 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T02:48:33.404 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T02:48:33.442 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T02:48:33.581 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T02:48:33.609 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T02:48:33.647 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T02:48:34.019 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T02:48:34.029 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T02:48:34.048 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T02:48:34.057 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T02:48:34.085 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T02:48:34.104 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T02:48:34.363 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T02:48:34.392 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T02:48:34.398 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T02:48:34.423 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T02:48:34.429 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T02:48:34.465 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T02:48:35.194 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package radosgw. 2024-09-11T02:48:35.222 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../117-radosgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:35.260 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:35.442 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package radosgw. 2024-09-11T02:48:35.471 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../117-radosgw_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:35.508 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:36.197 INFO:teuthology.orchestra.run.smithi098.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T02:48:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:36.264 INFO:teuthology.orchestra.run.smithi098.stdout:Unpacking rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:36.386 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T02:48:36.403 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4854-g8f219c47-1jammy_amd64.deb ... 2024-09-11T02:48:36.444 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:36.560 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T02:48:36.740 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T02:48:36.810 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T02:48:36.937 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T02:48:36.990 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T02:48:37.117 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T02:48:37.202 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T02:48:37.383 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T02:48:37.588 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T02:48:37.699 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T02:48:38.008 INFO:teuthology.orchestra.run.smithi098.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T02:48:38.072 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:38.135 INFO:teuthology.orchestra.run.smithi190.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T02:48:38.201 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:38.514 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user cephadm....done 2024-09-11T02:48:38.670 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T02:48:38.718 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user cephadm....done 2024-09-11T02:48:38.874 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T02:48:39.098 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T02:48:39.167 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T02:48:39.418 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T02:48:39.423 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T02:48:39.541 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T02:48:39.544 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T02:48:39.783 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T02:48:39.803 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T02:48:40.035 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T02:48:40.071 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T02:48:40.161 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T02:48:40.214 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T02:48:40.488 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T02:48:40.532 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T02:48:40.848 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T02:48:40.893 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T02:48:41.107 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T02:48:41.127 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T02:48:41.384 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T02:48:41.387 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T02:48:41.627 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T02:48:41.638 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T02:48:41.898 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T02:48:41.911 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T02:48:42.148 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T02:48:42.179 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T02:48:42.400 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:42.439 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-argparse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:42.698 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T02:48:42.809 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T02:48:42.824 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-11T02:48:42.918 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-11T02:48:42.950 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T02:48:43.044 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T02:48:43.059 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T02:48:43.145 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T02:48:43.169 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T02:48:43.271 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T02:48:43.411 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T02:48:43.522 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T02:48:43.790 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T02:48:43.835 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T02:48:43.889 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T02:48:43.890 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T02:48:43.920 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T02:48:43.984 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T02:48:44.016 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T02:48:44.110 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T02:48:44.375 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:44.461 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:44.501 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T02:48:44.561 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T02:48:44.786 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T02:48:44.838 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T02:48:45.054 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T02:48:45.080 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T02:48:45.372 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T02:48:45.381 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T02:48:45.498 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T02:48:45.507 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T02:48:46.141 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T02:48:46.201 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T02:48:46.393 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T02:48:46.444 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T02:48:46.502 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T02:48:46.570 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T02:48:46.611 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T02:48:46.695 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T02:48:46.915 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T02:48:47.013 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T02:48:47.307 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T02:48:47.381 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T02:48:47.542 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T02:48:47.632 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T02:48:47.935 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T02:48:48.009 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T02:48:48.219 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T02:48:48.285 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T02:48:48.554 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T02:48:48.636 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T02:48:48.797 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T02:48:48.887 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T02:48:48.932 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:49.005 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:48:49.233 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T02:48:49.306 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T02:48:49.525 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T02:48:49.624 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T02:48:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T02:48:50.846 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T02:48:50.850 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:48:50.976 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T02:48:51.064 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:48:51.165 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T02:48:51.235 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T02:48:51.364 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T02:48:51.450 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T02:48:51.487 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T02:48:51.567 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T02:48:51.668 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T02:48:51.781 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T02:48:51.953 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T02:48:52.040 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T02:48:52.212 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T02:48:52.283 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:48:52.384 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T02:48:52.472 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:48:52.573 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T02:48:52.685 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T02:48:52.874 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T02:48:52.944 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T02:48:53.133 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T02:48:53.204 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T02:48:53.418 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T02:48:53.455 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T02:48:53.677 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T02:48:53.715 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T02:48:53.928 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T02:48:53.982 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T02:48:54.092 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T02:48:54.196 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T02:48:54.313 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T02:48:54.351 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T02:48:54.477 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T02:48:54.581 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T02:48:54.707 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T02:48:54.770 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T02:48:55.000 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T02:48:55.071 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T02:48:55.293 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T02:48:55.322 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T02:48:55.457 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T02:48:55.535 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T02:48:55.583 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:55.661 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T02:48:55.671 INFO:teuthology.orchestra.run.smithi098.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T02:48:55.787 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T02:48:55.876 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T02:48:55.901 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T02:48:56.027 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T02:48:56.106 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T02:48:56.206 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T02:48:56.420 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T02:48:56.575 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T02:48:56.688 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T02:48:56.814 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T02:48:56.834 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T02:48:56.960 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T02:48:57.074 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:48:57.191 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T02:48:57.211 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T02:48:57.329 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T02:48:57.504 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T02:48:57.655 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T02:48:57.785 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T02:48:57.939 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T02:48:58.053 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T02:48:58.179 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T02:48:58.198 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T02:48:58.366 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T02:48:58.455 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T02:48:58.634 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T02:48:58.817 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T02:48:58.978 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T02:48:59.060 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T02:48:59.238 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T02:48:59.470 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T02:48:59.598 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T02:48:59.771 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T02:48:59.891 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T02:49:00.099 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T02:49:00.208 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:00.227 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T02:49:00.309 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:00.345 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librados2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:00.435 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T02:49:00.471 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:00.597 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T02:49:00.772 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:00.898 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:00.908 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.032 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T02:49:01.034 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libradosstriper1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.160 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T02:49:01.300 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.410 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.445 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librbd1 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.536 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.571 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-modules-core (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.697 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:01.752 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T02:49:01.752 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T02:49:01.893 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T02:49:01.893 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T02:49:02.212 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:02.338 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:02.381 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs-dev (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:02.464 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:02.491 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rados (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:02.599 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T02:49:02.617 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librgw2 (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:02.743 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T02:49:03.827 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:03.953 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:03.980 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rbd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.079 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.106 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up rbd-fuse (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.205 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.231 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.331 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.358 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cephfs (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.484 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-common (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:04.841 INFO:teuthology.orchestra.run.smithi098.stdout:Adding group ceph....done 2024-09-11T02:49:04.968 INFO:teuthology.orchestra.run.smithi190.stdout:Adding group ceph....done 2024-09-11T02:49:05.250 INFO:teuthology.orchestra.run.smithi098.stdout:Adding system user ceph....done 2024-09-11T02:49:05.344 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user ceph....done 2024-09-11T02:49:05.420 INFO:teuthology.orchestra.run.smithi098.stdout:Setting system user ceph properties....done 2024-09-11T02:49:05.439 INFO:teuthology.orchestra.run.smithi098.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T02:49:05.504 INFO:teuthology.orchestra.run.smithi190.stdout:Setting system user ceph properties....done 2024-09-11T02:49:05.524 INFO:teuthology.orchestra.run.smithi190.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T02:49:05.576 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T02:49:05.665 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T02:49:05.922 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T02:49:05.980 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T02:49:06.383 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:06.468 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-test (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:06.501 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:06.610 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up radosgw (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:06.943 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T02:49:06.943 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T02:49:07.068 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T02:49:07.068 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T02:49:07.403 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:07.496 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-base (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:07.758 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T02:49:07.845 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T02:49:08.213 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:08.281 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mds (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:08.453 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T02:49:08.453 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T02:49:08.508 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T02:49:08.508 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T02:49:08.939 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:08.975 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:09.164 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T02:49:09.164 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T02:49:09.192 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T02:49:09.193 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T02:49:09.633 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:09.667 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-osd (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:09.906 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T02:49:09.906 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T02:49:09.948 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T02:49:09.949 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T02:49:10.408 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:10.427 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:10.511 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:10.553 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:10.662 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:10.704 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mon (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:10.876 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T02:49:10.876 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T02:49:10.918 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T02:49:10.918 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T02:49:11.363 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:11.462 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-cephadm (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:11.514 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:11.593 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:11.640 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:11.702 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-dashboard (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:11.783 INFO:teuthology.orchestra.run.smithi098.stdout:Setting up ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:11.836 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-volume (19.3.0-4854-g8f219c47-1jammy) ... 2024-09-11T02:49:12.128 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T02:49:12.197 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T02:49:12.228 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T02:49:12.298 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T02:49:12.381 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T02:49:12.434 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T02:49:13.457 INFO:teuthology.orchestra.run.smithi098.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T02:49:13.601 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T02:49:14.081 INFO:teuthology.orchestra.run.smithi098.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T02:49:14.087 DEBUG:teuthology.parallel:result is None 2024-09-11T02:49:14.286 INFO:teuthology.orchestra.run.smithi190.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T02:49:14.291 DEBUG:teuthology.parallel:result is None 2024-09-11T02:49:14.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T02:49:14.449 DEBUG:teuthology.orchestra.run.smithi098:> dpkg-query -W -f '${Version}' ceph 2024-09-11T02:49:14.468 INFO:teuthology.orchestra.run.smithi098.stdout:19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:49:14.468 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:49:14.468 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854-g8f219c47-1jammy is installed. 2024-09-11T02:49:14.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T02:49:14.615 DEBUG:teuthology.orchestra.run.smithi190:> dpkg-query -W -f '${Version}' ceph 2024-09-11T02:49:14.635 INFO:teuthology.orchestra.run.smithi190.stdout:19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:49:14.635 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4854-g8f219c47-1jammy 2024-09-11T02:49:14.635 INFO:teuthology.task.install:The correct ceph version 19.3.0-4854-g8f219c47-1jammy is installed. 2024-09-11T02:49:14.637 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T02:49:14.637 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:14.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T02:49:14.651 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:14.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T02:49:14.695 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T02:49:14.695 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:14.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T02:49:14.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T02:49:14.765 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:14.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T02:49:14.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T02:49:14.833 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T02:49:14.834 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:14.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T02:49:14.846 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T02:49:14.901 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:14.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T02:49:14.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T02:49:14.969 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T02:49:14.969 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:14.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T02:49:14.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T02:49:15.037 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:15.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T02:49:15.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T02:49:15.105 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T02:49:15.187 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T02:49:15.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-09-11T02:49:15.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-09-11T02:49:15.203 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T02:49:15.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T02:49:15.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T02:49:15.259 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T02:49:15.259 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T02:49:15.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T02:49:15.325 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T02:49:15.325 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T02:49:15.325 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006', 'branch': 'wip-hemanth-testing-2024-09-10-0723', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'hyelloji@redhat.com', 'first_in_suite': False, 'job_id': '7900006', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-hemanth-testing-2024-09-10-0723'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-hemanth-testing-2024-09-10-0723', 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30'}}, 'owner': 'scheduled_hyelloji@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9167, 'sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-hemanth-testing-2024-09-10-0723', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8f219c476cb4f26fc5719c661c0323719a45cb30/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8f219c476cb4f26fc5719c661c0323719a45cb30', 'targets': {'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC29kLTUrIQi9ykqRkPmyOPDuuYW5727QeTKFV/QfTpi0fJ38TK6j33ZcuOenyXqj4oZURJmqaPoWi5ZE99z6Rw=', 'smithi190.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEqA3w7um1PM9K9pP2EmmbqLJ8EKTOG5V/fvReCU9BrBibStFoii6XDaktx8tmak7dy8CWbyE0LBuJ/wVMRvlBs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_17:16:14', 'tube': 'smithi', 'user': 'hyelloji', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T02:49:15.326 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T02:49:15.359 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T02:49:15.364 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T02:49:15.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T02:49:15.420 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:15.420 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T02:49:15.466 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T02:49:15.466 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-09-11T02:49:15.514 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T02:49:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:15.515 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 877 Links: 1 2024-09-11T02:49:15.515 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:15.515 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-11 02:43:42.944634869 +0000 2024-09-11T02:49:15.515 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-11 02:43:42.692637120 +0000 2024-09-11T02:49:15.515 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-11 02:43:42.692637120 +0000 2024-09-11T02:49:15.515 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-11T02:49:15.515 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T02:49:15.568 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-11T02:49:15.569 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-11T02:49:15.569 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000378697 s, 1.4 MB/s 2024-09-11T02:49:15.570 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T02:49:15.619 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-09-11T02:49:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T02:49:15.666 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:15.666 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 891 Links: 1 2024-09-11T02:49:15.666 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:15.666 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-11 02:43:43.420630617 +0000 2024-09-11T02:49:15.666 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-11 02:43:43.176632797 +0000 2024-09-11T02:49:15.666 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-11 02:43:43.176632797 +0000 2024-09-11T02:49:15.667 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-11T02:49:15.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T02:49:15.720 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-11T02:49:15.721 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-11T02:49:15.721 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000441297 s, 1.2 MB/s 2024-09-11T02:49:15.722 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T02:49:15.775 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-11 02:43:43.896626366 +0000 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-11 02:43:43.652628545 +0000 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-11 02:43:43.652628545 +0000 2024-09-11T02:49:15.822 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-11T02:49:15.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T02:49:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-11T02:49:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-11T02:49:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00041573 s, 1.2 MB/s 2024-09-11T02:49:15.877 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T02:49:15.927 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-09-11T02:49:15.973 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T02:49:15.974 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:15.974 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-09-11T02:49:15.974 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:15.974 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-11 02:43:44.456621364 +0000 2024-09-11T02:49:15.974 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-11 02:43:44.176623864 +0000 2024-09-11T02:49:15.974 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-11 02:43:44.176623864 +0000 2024-09-11T02:49:15.974 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-09-11T02:49:15.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T02:49:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-11T02:49:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-11T02:49:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000280508 s, 1.8 MB/s 2024-09-11T02:49:16.029 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T02:49:16.079 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:16.079 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:16.080 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T02:49:16.085 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T02:49:16.086 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 02:43:50.669788632 +0000 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 02:43:50.441790669 +0000 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 02:43:50.441790669 +0000 2024-09-11T02:49:16.135 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T02:49:16.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T02:49:16.188 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T02:49:16.189 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T02:49:16.189 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000381574 s, 1.3 MB/s 2024-09-11T02:49:16.190 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T02:49:16.245 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-09-11T02:49:16.294 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T02:49:16.295 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:16.295 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-09-11T02:49:16.295 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:16.295 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 02:43:51.145784381 +0000 2024-09-11T02:49:16.295 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 02:43:50.917786417 +0000 2024-09-11T02:49:16.296 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 02:43:50.917786417 +0000 2024-09-11T02:49:16.296 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T02:49:16.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T02:49:16.348 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T02:49:16.349 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T02:49:16.349 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000413335 s, 1.2 MB/s 2024-09-11T02:49:16.350 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T02:49:16.403 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 02:43:51.641779951 +0000 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 02:43:51.369782380 +0000 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 02:43:51.369782380 +0000 2024-09-11T02:49:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T02:49:16.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T02:49:16.508 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T02:49:16.508 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T02:49:16.508 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000434116 s, 1.2 MB/s 2024-09-11T02:49:16.509 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T02:49:16.560 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-09-11T02:49:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T02:49:16.606 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T02:49:16.606 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-09-11T02:49:16.606 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T02:49:16.607 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-09-11 02:43:52.113775735 +0000 2024-09-11T02:49:16.607 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-09-11 02:43:51.869777914 +0000 2024-09-11T02:49:16.607 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-09-11 02:43:51.869777914 +0000 2024-09-11T02:49:16.607 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-09-11T02:49:16.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T02:49:16.666 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-09-11T02:49:16.666 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-09-11T02:49:16.666 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000373286 s, 1.4 MB/s 2024-09-11T02:49:16.667 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T02:49:16.716 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:16.716 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T02:49:16.716 INFO:tasks.ceph:Generating config... 2024-09-11T02:49:16.717 INFO:tasks.ceph:[client] debug monc = 20 2024-09-11T02:49:16.717 INFO:tasks.ceph:[client] debug ms = 1 2024-09-11T02:49:16.717 INFO:tasks.ceph:[global] debug ms = 1 2024-09-11T02:49:16.717 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-11T02:49:16.717 INFO:tasks.ceph:[global] debug rados = 20 2024-09-11T02:49:16.718 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T02:49:16.718 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-11T02:49:16.718 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T02:49:16.718 INFO:tasks.ceph:[global] ms type = async 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-11T02:49:16.718 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-11T02:49:16.719 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-11T02:49:16.719 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-11T02:49:16.719 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-11T02:49:16.719 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] bluestore write v2 = True 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-11T02:49:16.719 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-11T02:49:16.720 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T02:49:16.721 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-11T02:49:16.721 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-11T02:49:16.721 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T02:49:16.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:16.755 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T02:49:16.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T02:49:16.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T02:49:16.925 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.98'), ('mon.b', '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]'), ('mon.c', '[v2:172.21.15.98:3302,v1:172.21.15.98:6791]'), ('mon.d', '[v2:172.21.15.98:3303,v1:172.21.15.98:6792]'), ('mon.e', '[v2:172.21.15.98:3304,v1:172.21.15.98:6793]'), ('mon.f', '172.21.15.190'), ('mon.g', '[v2:172.21.15.190:3301,v1:172.21.15.190:6790]'), ('mon.h', '[v2:172.21.15.190:3302,v1:172.21.15.190:6791]'), ('mon.i', '[v2:172.21.15.190:3303,v1:172.21.15.190:6792]')] 2024-09-11T02:49:16.926 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],[v2:172.21.15.98:3302,v1:172.21.15.98:6791],[v2:172.21.15.98:3303,v1:172.21.15.98:6792],[v2:172.21.15.98:3304,v1:172.21.15.98:6793],172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790],[v2:172.21.15.190:3302,v1:172.21.15.190:6791],[v2:172.21.15.190:3303,v1:172.21.15.190:6792]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-09-11T02:49:16.927 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:16.927 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T02:49:16.974 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --addv b '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]' --addv c '[v2:172.21.15.98:3302,v1:172.21.15.98:6791]' --addv d '[v2:172.21.15.98:3303,v1:172.21.15.98:6792]' --addv e '[v2:172.21.15.98:3304,v1:172.21.15.98:6793]' --add f 172.21.15.190 --addv g '[v2:172.21.15.190:3301,v1:172.21.15.190:6790]' --addv h '[v2:172.21.15.190:3302,v1:172.21.15.190:6791]' --addv i '[v2:172.21.15.190:3303,v1:172.21.15.190:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:17.081 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:17.081 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:17.081 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-09-11T02:49:17.081 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2024-09-11T02:49:17.081 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:17.081 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:17.081 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 17 (quincy) 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 3 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:3: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:4: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:5: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:6: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:7: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:8: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:17.082 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-09-11T02:49:17.083 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T02:49:17.088 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 797ac5bf-8bfb-4032-bff5-2e861cee7479... 2024-09-11T02:49:17.089 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T02:49:17.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.159 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.160 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T02:49:17.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],[v2:172.21.15.98:3302,v1:172.21.15.98:6791],[v2:172.21.15.98:3303,v1:172.21.15.98:6792],[v2:172.21.15.98:3304,v1:172.21.15.98:6793],172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790],[v2:172.21.15.190:3302,v1:172.21.15.190:6791],[v2:172.21.15.190:3303,v1:172.21.15.190:6792]" 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: debug objecter = 20 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: debug rados = 20 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: mon election default strategy = 3 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-09-11T02:49:17.162 INFO:teuthology.orchestra.run.smithi098.stdout: ms type = async 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-09-11T02:49:17.163 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-09-11T02:49:17.164 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: bluefs allocator = hybrid 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore allocator = hybrid 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore write v2 = True 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore zero block detection = True 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 20 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 20 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-11T02:49:17.165 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd class default list = * 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd class load list = * 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify cached snaps = True 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify missing on start = True 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock override recovery settings = True 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue = debug_random 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue cut off = debug_random 2024-09-11T02:49:17.166 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-09-11T02:49:17.167 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T02:49:17.168 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon min osdmap epochs = 25 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune interval = 2 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune min = 15 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune txsize = 2 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub inject crc mismatch = 0.01 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub inject missing keys = 0.05 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub interval = 300 2024-09-11T02:49:17.169 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: paxos service trim min = 5 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-09-11T02:49:17.170 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.d] 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.e] 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.f] 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.g] 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.h] 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.i] 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi190.stdout:[global] 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi190.stdout: chdir = "" 2024-09-11T02:49:17.171 INFO:teuthology.orchestra.run.smithi190.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: auth supported = cephx 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: filestore xattr use omap = true 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: mon clock drift allowed = 1.000 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: osd crush chooseleaf type = 0 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: auth debug = true 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on old message = true 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: ms die on bug = true 2024-09-11T02:49:17.172 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon pg warn max object skew = 0 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default size = 2 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow primary affinity = true 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd allow pg remap = true 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on legacy crush tunables = false 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on crush straw calc version zero = false 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on no sortbitwise = false 2024-09-11T02:49:17.173 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on osd down out interval zero = false 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on too few osds = false 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: mon_allow_pool_size_one = true 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: osd default data pool replay window = 5 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: mon allow pool delete = true 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: mon cluster log file level = debug 2024-09-11T02:49:17.174 INFO:teuthology.orchestra.run.smithi190.stdout: debug asserts on shutdown = true 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: mon health detail to clog = false 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: mon host = "172.21.15.98,[v2:172.21.15.98:3301,v1:172.21.15.98:6790],[v2:172.21.15.98:3302,v1:172.21.15.98:6791],[v2:172.21.15.98:3303,v1:172.21.15.98:6792],[v2:172.21.15.98:3304,v1:172.21.15.98:6793],172.21.15.190,[v2:172.21.15.190:3301,v1:172.21.15.190:6790],[v2:172.21.15.190:3302,v1:172.21.15.190:6791],[v2:172.21.15.190:3303,v1:172.21.15.190:6792]" 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: debug objecter = 20 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: debug rados = 20 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: mon client directed command retry = 5 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: mon election default strategy = 3 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: ms inject socket failures = 5000 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: ms type = async 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: fsid = 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout:[osd] 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: osd journal size = 100 2024-09-11T02:49:17.175 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub load threshold = 5.0 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd scrub max interval = 600 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock profile = high_recovery_ops 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd recover clone overlap = true 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd recovery max chunk = 1048576 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug shutdown = true 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug op order = true 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify stray on activate = true 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug trim objects = true 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd open classes on start = true 2024-09-11T02:49:17.176 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug pg log writeout = true 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: osd deep scrub update digest min age = 30 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: osd map max advance = 10 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: journal zero on create = true 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: filestore ondisk finisher threads = 3 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: filestore apply finisher threads = 3 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: bdev debug aio = true 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug misdirected ops = true 2024-09-11T02:49:17.177 INFO:teuthology.orchestra.run.smithi190.stdout: bluefs allocator = hybrid 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore allocator = hybrid 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore block size = 96636764160 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore fsck on mount = True 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore write v2 = True 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: bluestore zero block detection = True 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluefs = 20 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: debug bluestore = 20 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: debug osd = 20 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: debug rocksdb = 10 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd full ratio = 0.9 2024-09-11T02:49:17.178 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd nearfull ratio = 0.8 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd class default list = * 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd class load list = * 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify cached snaps = True 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd debug verify missing on start = True 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd failsafe full ratio = 0.95 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd mclock override recovery settings = True 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd objectstore = bluestore 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue = debug_random 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: osd op queue cut off = debug_random 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr] 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: debug mgr = 20 2024-09-11T02:49:17.179 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: mgr/telemetry/nag = false 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout:[mon] 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: debug mon = 20 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: debug paxos = 20 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: debug auth = 20 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: mon data avail warn = 5 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: mon mgr mkfs grace = 240 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min pgs per osd = 4 2024-09-11T02:49:17.180 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd reporter subtree level = osd 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: mon osd prime pg temp = true 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: mon reweight min bytes per osd = 10 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: # 1m isn't quite enough 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: mon_down_mkfs_grace = 2m 2024-09-11T02:49:17.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon_warn_on_filestore_osds = false 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon min osdmap epochs = 25 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune interval = 2 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune min = 15 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon osdmap full prune txsize = 2 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub inject crc mismatch = 0.01 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub inject missing keys = 0.05 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon scrub interval = 300 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: mon warn on pool no app = False 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: paxos service trim min = 5 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout:[client] 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: rgw cache enabled = true 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable ops log = true 2024-09-11T02:49:17.182 INFO:teuthology.orchestra.run.smithi190.stdout: rgw enable usage log = true 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout: debug monc = 20 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout: debug ms = 1 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.a] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.b] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.c] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.d] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.e] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.f] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.g] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.h] 2024-09-11T02:49:17.183 INFO:teuthology.orchestra.run.smithi190.stdout:[mon.i] 2024-09-11T02:49:17.185 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T02:49:17.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T02:49:17.272 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T02:49:17.273 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:17.273 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T02:49:17.278 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:17.278 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T02:49:17.326 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T02:49:17.326 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:17.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T02:49:17.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T02:49:17.391 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:17.391 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:17.438 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T02:49:17.438 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:17.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T02:49:17.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T02:49:17.461 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:17.461 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:17.510 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T02:49:17.510 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T02:49:17.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T02:49:17.596 INFO:teuthology.orchestra.run.smithi190.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T02:49:17.609 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T02:49:17.609 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T02:49:17.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T02:49:17.683 INFO:teuthology.orchestra.run.smithi190.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T02:49:17.696 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T02:49:17.696 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi190.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T02:49:17.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T02:49:17.710 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:17.710 INFO:tasks.ceph:role: osd.0 2024-09-11T02:49:17.710 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T02:49:17.710 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T02:49:17.775 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T02:49:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T02:49:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T02:49:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T02:49:17.775 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T02:49:17.775 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-11T02:49:17.775 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T02:49:17.776 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T02:49:17.776 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T02:49:17.776 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T02:49:18.253 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-09-11T02:49:18.255 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-11T02:49:18.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T02:49:18.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T02:49:18.425 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T02:49:18.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:49:18.426 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T02:49:18.482 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:18.482 INFO:tasks.ceph:role: osd.1 2024-09-11T02:49:18.482 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T02:49:18.482 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T02:49:18.545 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T02:49:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T02:49:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T02:49:18.545 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T02:49:18.545 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T02:49:18.546 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-11T02:49:18.546 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T02:49:18.546 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T02:49:18.546 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T02:49:18.546 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T02:49:18.987 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-09-11T02:49:18.989 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-11T02:49:18.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T02:49:19.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T02:49:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:49:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T02:49:19.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T02:49:19.181 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:19.181 INFO:tasks.ceph:role: osd.2 2024-09-11T02:49:19.181 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T02:49:19.181 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T02:49:19.241 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T02:49:19.241 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T02:49:19.242 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T02:49:19.709 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-09-11T02:49:19.711 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-11T02:49:19.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T02:49:19.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T02:49:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:49:19.856 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T02:49:19.856 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:19.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:19.971+0000 7f279c8c07c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T02:49:19.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:19.975+0000 7f279c8c07c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T02:49:19.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:19.975+0000 7f279c8c07c0 -1 bdev(0x55d7ef253800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T02:49:19.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:19.975+0000 7f279c8c07c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T02:49:23.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T02:49:23.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:23.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:23.191+0000 7fdb8318b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T02:49:23.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:23.191+0000 7fdb8318b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T02:49:23.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:23.191+0000 7fdb8318b7c0 -1 bdev(0x562fe95cb800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T02:49:23.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:23.191+0000 7fdb8318b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T02:49:26.362 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T02:49:26.418 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:26.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:26.499+0000 7f9c9c9957c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T02:49:26.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:26.503+0000 7f9c9c9957c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T02:49:26.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:26.503+0000 7f9c9c9957c0 -1 bdev(0x55c22d6fb800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T02:49:26.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:26.503+0000 7f9c9c9957c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T02:49:29.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T02:49:29.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T02:49:29.730 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:29.730 INFO:tasks.ceph:role: osd.3 2024-09-11T02:49:29.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T02:49:29.730 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T02:49:29.797 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T02:49:29.798 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T02:49:30.256 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-11T02:49:30.257 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-11T02:49:30.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-11T02:49:30.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T02:49:30.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T02:49:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:49:30.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-11T02:49:30.424 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:30.424 INFO:tasks.ceph:role: osd.4 2024-09-11T02:49:30.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T02:49:30.425 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T02:49:30.488 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T02:49:30.958 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-11T02:49:30.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-11T02:49:30.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-11T02:49:31.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-11T02:49:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:49:31.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T02:49:31.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-11T02:49:31.178 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T02:49:31.178 INFO:tasks.ceph:role: osd.5 2024-09-11T02:49:31.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T02:49:31.178 DEBUG:teuthology.orchestra.run.smithi190:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T02:49:31.241 INFO:teuthology.orchestra.run.smithi190.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T02:49:31.241 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T02:49:31.241 INFO:teuthology.orchestra.run.smithi190.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T02:49:31.241 INFO:teuthology.orchestra.run.smithi190.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T02:49:31.241 INFO:teuthology.orchestra.run.smithi190.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T02:49:31.241 INFO:teuthology.orchestra.run.smithi190.stdout: = sunit=0 swidth=0 blks 2024-09-11T02:49:31.242 INFO:teuthology.orchestra.run.smithi190.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T02:49:31.242 INFO:teuthology.orchestra.run.smithi190.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T02:49:31.242 INFO:teuthology.orchestra.run.smithi190.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T02:49:31.242 INFO:teuthology.orchestra.run.smithi190.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T02:49:31.709 INFO:teuthology.orchestra.run.smithi190.stdout:Discarding blocks...Done. 2024-09-11T02:49:31.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi190.front.sepia.ceph.com -o noatime 2024-09-11T02:49:31.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-11T02:49:31.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-11T02:49:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:49:31.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T02:49:31.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:31.979 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:31.980+0000 7f684eccb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T02:49:31.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:31.980+0000 7f684eccb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T02:49:31.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:31.980+0000 7f684eccb7c0 -1 bdev(0x5576f32f1800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T02:49:31.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:31.980+0000 7f684eccb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T02:49:35.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T02:49:35.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:35.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:35.184+0000 7f2ae4dbf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-11T02:49:35.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:35.184+0000 7f2ae4dbf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-11T02:49:35.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:35.184+0000 7f2ae4dbf7c0 -1 bdev(0x556caa827800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-11T02:49:35.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:35.184+0000 7f2ae4dbf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-11T02:49:38.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-11T02:49:38.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:38.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:38.460+0000 7f1389c477c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-11T02:49:38.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:38.460+0000 7f1389c477c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-11T02:49:38.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:38.460+0000 7f1389c477c0 -1 bdev(0x55844c851800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-11T02:49:38.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:38.460+0000 7f1389c477c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-11T02:49:41.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T02:49:41.657 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T02:49:41.658 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:41.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T02:49:41.672 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:41.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T02:49:41.725 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:41.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T02:49:41.781 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:41.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T02:49:41.794 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:41.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T02:49:41.849 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:41.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-11T02:49:41.913 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:41.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-11T02:49:41.969 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:41.970 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T02:49:42.017 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T02:49:42.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T02:49:42.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAvBeFmOv4TOhAAZjG8z1KezP5PaGZbtp4eiw== 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAzBeFmkq+xCxAAgg3DUenS9DIzsLYqxyBUiA== 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQA2BeFm5AoWHhAAZNAkTil/7AqQu23nh+Z8LQ== 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-09-11T02:49:42.064 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAtBeFmmQzXIxAARHOFvAxsF6kW+VPn5LrLkw== 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQA7BeFmHgugOhAApcu5KzWrHmaD8hIyMHrVYA== 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQA/BeFmb5ZQCxAAsdaF5sQ+Z2RU6KVlDqGIGw== 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBCBeFmaaO4GxAAt6uFRLPIGmlb9KbkMHRcnA== 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-09-11T02:49:42.065 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAtBeFmx8QNKRAAVRx0b4/fmChUXOCS3ZfU0g== 2024-09-11T02:49:42.072 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.0] 2024-09-11T02:49:42.072 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAvBeFmOv4TOhAAZjG8z1KezP5PaGZbtp4eiw== 2024-09-11T02:49:42.072 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.1] 2024-09-11T02:49:42.072 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAzBeFmkq+xCxAAgg3DUenS9DIzsLYqxyBUiA== 2024-09-11T02:49:42.072 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.2] 2024-09-11T02:49:42.072 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQA2BeFm5AoWHhAAZNAkTil/7AqQu23nh+Z8LQ== 2024-09-11T02:49:42.072 INFO:teuthology.orchestra.run.smithi190.stdout:[mgr.x] 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAtBeFmmQzXIxAARHOFvAxsF6kW+VPn5LrLkw== 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.3] 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQA7BeFmHgugOhAApcu5KzWrHmaD8hIyMHrVYA== 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.4] 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQA/BeFmb5ZQCxAAsdaF5sQ+Z2RU6KVlDqGIGw== 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout:[osd.5] 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQBCBeFmaaO4GxAAt6uFRLPIGmlb9KbkMHRcnA== 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout:[client.0] 2024-09-11T02:49:42.073 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQAtBeFmx8QNKRAAVRx0b4/fmChUXOCS3ZfU0g== 2024-09-11T02:49:42.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.206 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T02:49:42.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T02:49:42.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.454 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T02:49:42.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T02:49:42.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T02:49:42.717 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T02:49:42.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T02:49:42.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:42.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T02:49:42.832 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-11T02:49:42.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:42.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-11T02:49:42.983 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-11T02:49:43.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:43.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-11T02:49:43.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-09-11T02:49:43.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:43.278 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-09-11T02:49:43.289 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-09-11T02:49:43.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:43.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-09-11T02:49:43.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-09-11T02:49:43.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:43.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-09-11T02:49:43.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-09-11T02:49:43.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:43.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-09-11T02:49:43.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-09-11T02:49:43.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:43.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-09-11T02:49:43.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-09-11T02:49:43.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T02:49:44.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-09-11T02:49:44.062 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:44.064 DEBUG:teuthology.orchestra.run.smithi190:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T02:49:44.109 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T02:49:44.110 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T02:49:44.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T02:49:44.113 INFO:tasks.ceph.mon.a:Started 2024-09-11T02:49:44.113 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T02:49:44.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T02:49:44.116 INFO:tasks.ceph.mon.b:Started 2024-09-11T02:49:44.117 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T02:49:44.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T02:49:44.120 INFO:tasks.ceph.mon.c:Started 2024-09-11T02:49:44.120 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-11T02:49:44.120 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-11T02:49:44.123 INFO:tasks.ceph.mon.d:Started 2024-09-11T02:49:44.123 INFO:tasks.ceph.mon.e:Restarting daemon 2024-09-11T02:49:44.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-09-11T02:49:44.128 INFO:tasks.ceph.mon.e:Started 2024-09-11T02:49:44.128 INFO:tasks.ceph.mon.f:Restarting daemon 2024-09-11T02:49:44.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-09-11T02:49:44.155 INFO:tasks.ceph.mon.f:Started 2024-09-11T02:49:44.156 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-11T02:49:44.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-11T02:49:44.158 INFO:tasks.ceph.mon.g:Started 2024-09-11T02:49:44.158 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-11T02:49:44.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-11T02:49:44.160 INFO:tasks.ceph.mon.h:Started 2024-09-11T02:49:44.160 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-11T02:49:44.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-11T02:49:44.162 INFO:tasks.ceph.mon.i:Started 2024-09-11T02:49:44.162 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T02:49:44.162 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T02:49:44.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T02:49:44.164 INFO:tasks.ceph.mgr.x:Started 2024-09-11T02:49:44.165 DEBUG:tasks.ceph:set 0 configs 2024-09-11T02:49:44.165 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T02:49:44.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient: get_monmap_and_config 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient: build_initial_monmap 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient: monmap: 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:44.298522+0000 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:44.298522+0000 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:44.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:44.296 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 1 Processor -- start 2024-09-11T02:49:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 1 -- start start 2024-09-11T02:49:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient: init 2024-09-11T02:49:44.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient: _add_conns ranks=[0,8,7,4,1,9,10,2,5,3,6] 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b80139b80 addr v1:172.21.15.98:6789/0 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient(hunting): picked mon.noname-e con 0x7f6b80139f60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient(hunting): picked mon.noname-i con 0x7f6b8013a380 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b80070320 con 0x7f6b80139b80 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f6b800713c0 con 0x7f6b8013a380 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6b8006fff0 con 0x7f6b80139f60 2024-09-11T02:49:44.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b86813640 10 monclient(hunting): authenticate will time out at 938.178442s 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6b80139b80 0x7f6b80072690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53228/0 (socket says 172.21.15.98:53228) 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7ffff640 1 -- 172.21.15.98:0/1384891436 learned_addr learned my addr 172.21.15.98:0/1384891436 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request con 0x7f6b80139f60 auth_method 0 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 msgr2=0x7f6b8013ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 msgr2=0x7f6b8013ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:49:44.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 10 monclient(hunting): get_auth_request con 0x7f6b8013a380 auth_method 0 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 msgr2=0x7f6b80140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 msgr2=0x7f6b80140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:49:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.295+0000 7f6b84db0640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request con 0x7f6b80139f60 auth_method 0 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 msgr2=0x7f6b8013ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 msgr2=0x7f6b8013ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:49:44.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 10 monclient(hunting): get_auth_request con 0x7f6b8013a380 auth_method 0 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 msgr2=0x7f6b80140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 msgr2=0x7f6b80140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:49:44.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.499+0000 7f6b84db0640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:49:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235149917 0 0) 0x7f6b80070320 con 0x7f6b80139b80 2024-09-11T02:49:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): _negotiate 2024-09-11T02:49:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): my global_id is 4104 2024-09-11T02:49:44.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b68003530 con 0x7f6b80139b80 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1143279230 0 0) 0x7f6b68003530 con 0x7f6b80139b80 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 msgr2=0x7f6b8013ea90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b8013ea90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 msgr2=0x7f6b80140fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6b8013a380 0x7f6b80140fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: _finish_hunting 0 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b800a9660 con 0x7f6b80139b80 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: _finish_auth 0 2024-09-11T02:49:44.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: _check_auth_tickets 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b86813640 5 monclient: authenticate success, global_id 4104 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b86813640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f6b70001970 con 0x7f6b80139b80 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: dump: 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:44.711 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:44.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:44.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:44.712 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:44.712 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:44.712 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:44.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 >> v1:172.21.15.98:6789/0 conn(0x7f6b80139b80 legacy=0x7f6b80072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient: _add_conns ranks=[8,4,1,3,2,7,0,5,6] 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b68004240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): picked mon.e con 0x7f6b80139f60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6b68004910 0x7f6b68004d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): picked mon.c con 0x7f6b68004910 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b680099c0 0x7f6b6800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): picked mon.f con 0x7f6b680099c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6b800713c0 con 0x7f6b680099c0 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6b8006fff0 con 0x7f6b68004910 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6b6800c450 con 0x7f6b80139f60 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7effd640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b86813640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6b68004910 0x7f6b68004d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b680099c0 0x7f6b6800bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request con 0x7f6b68004910 auth_method 0 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 10 monclient(hunting): get_auth_request con 0x7f6b680099c0 auth_method 0 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b68004240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 10 monclient(hunting): get_auth_request con 0x7f6b80139f60 auth_method 0 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b84db0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 10 monclient(hunting): handle_auth_done global_id 4104 payload 293 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.707+0000 7f6b7f7fe640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 msgr2=0x7f6b68004240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:44.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6b80139f60 0x7f6b68004240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b680099c0 msgr2=0x7f6b6800bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b680099c0 0x7f6b6800bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 1 monclient: found mon.c 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 1 -- 172.21.15.98:0/1384891436 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b68003530 con 0x7f6b68004910 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:49:44.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.711+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6b68004910 0x7f6b68004d90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400da40 tx=0x7f6b7401c940 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e907b990d034566f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b7400d640 con 0x7f6b68004910 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 10 monclient: dump: 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:44.725 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b740b0070 con 0x7f6b68004910 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:44.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 4 set_mon_vals no callback set 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 1 -- 172.21.15.98:0/1384891436 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b740abd10 con 0x7f6b68004910 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b7effd640 10 monclient: dump: 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:44.727 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 10 monclient: shutdown 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 1 -- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6b68004910 msgr2=0x7f6b68004d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 1 --2- 172.21.15.98:0/1384891436 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6b68004910 0x7f6b68004d90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6b7400da40 tx=0x7f6b7401c940 comp rx=0 tx=0).stop 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b84db0640 1 -- 172.21.15.98:0/1384891436 reap_dead start 2024-09-11T02:49:44.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 1 -- 172.21.15.98:0/1384891436 shutdown_connections 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 1 -- 172.21.15.98:0/1384891436 >> 172.21.15.98:0/1384891436 conn(0x7f6b800ac7f0 msgr2=0x7f6b800a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 1 -- 172.21.15.98:0/1384891436 shutdown_connections 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 1 -- 172.21.15.98:0/1384891436 wait complete. 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 10 monclient: build_initial_monmap 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.715+0000 7f6b86813640 10 monclient: monmap: 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:44.729 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 librados: starting msgr at 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 librados: starting objecter 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 Processor -- start 2024-09-11T02:49:44.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 -- start start 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 librados: setting wanted keys 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 librados: calling monclient init 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient: init 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient: _add_conns ranks=[6,5,0,7,3,1,2,4,8] 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6b80137940 0x7f6b80137d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): picked mon.noname-d con 0x7f6b80137940 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6b801d1490 0x7f6b800a3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): picked mon.noname-h con 0x7f6b801d1490 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b800a4500 0x7f6b800a4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): picked mon.noname-a con 0x7f6b800a4500 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80141f40 con 0x7f6b800a4500 2024-09-11T02:49:44.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f6b800517e0 con 0x7f6b801d1490 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6b801420c0 con 0x7f6b80137940 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient(hunting): authenticate will time out at 938.601982s 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b800a4500 0x7f6b800a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6b80137940 0x7f6b80137d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b800a4500 0x7f6b800a4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59674/0 (socket says 172.21.15.98:59674) 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 -- 172.21.15.98:0/3038012881 learned_addr learned my addr 172.21.15.98:0/3038012881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6b80137940 0x7f6b80137d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:38232/0 (socket says 172.21.15.98:38232) 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): get_auth_request con 0x7f6b800a4500 auth_method 0 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 10 monclient(hunting): get_auth_request con 0x7f6b80137940 auth_method 0 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6b801d1490 0x7f6b800a3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request con 0x7f6b801d1490 auth_method 0 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:44.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient(hunting): handle_auth_done global_id 4122 payload 824 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 -- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6b80137940 msgr2=0x7f6b80137d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6b80137940 0x7f6b80137d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 -- 172.21.15.98:0/3038012881 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6b801d1490 msgr2=0x7f6b800a3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6b801d1490 0x7f6b800a3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient: _finish_hunting 0 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 monclient: found mon.noname-a 2024-09-11T02:49:44.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 -- 172.21.15.98:0/3038012881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b800a5ab0 con 0x7f6b800a4500 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b84db0640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b800a4500 0x7f6b800a4920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6b8013a810 tx=0x7f6b78009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 client.?.objecter ms_handle_connect 0x7f6b800a4500 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 1 -- 172.21.15.98:0/3038012881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b7801a020 con 0x7f6b800a4500 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: dump: 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.735 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: _finish_auth 0 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: _check_auth_tickets 2024-09-11T02:49:44.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 5 monclient: authenticate success, global_id 4122 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 1 -- 172.21.15.98:0/3038012881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b78007c30 con 0x7f6b800a4500 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient: _renew_subs 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b800ab9e0 con 0x7f6b800a4500 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 1 -- 172.21.15.98:0/3038012881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6b78007de0 con 0x7f6b800a4500 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 client.4122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 monclient: dump: 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:44.737 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:44.738 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7e7fc640 4 set_mon_vals no callback set 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient: _renew_subs 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b80123b50 con 0x7f6b800a4500 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b86813640 1 librados: init done 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 1 -- 172.21.15.98:0/3038012881 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f6b780167f0 con 0x7f6b800a4500 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 1 -- 172.21.15.98:0/3038012881 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6b7801fcf0 con 0x7f6b800a4500 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 10 client.4122.objecter ms_dispatch 0x7f6b80000ba0 osd_map(1..1 src has 1..1) 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 3 client.4122.objecter handle_osd_map got epochs [1,1] > 0 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 3 client.4122.objecter handle_osd_map decoding full epoch 1 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.719+0000 7f6b7cff9640 20 client.4122.objecter dump_active .. 0 homeless 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.723+0000 7f6b86813640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:44.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.723+0000 7f6b86813640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.723+0000 7f6b86813640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.723+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b80069670 con 0x7f6b800a4500 2024-09-11T02:49:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.731+0000 7f6b7cff9640 1 -- 172.21.15.98:0/3038012881 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f6b78024030 con 0x7f6b800a4500 2024-09-11T02:49:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.731+0000 7f6b7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:44.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.731+0000 7f6b7cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.799+0000 7f6b86813640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T02:49:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.799+0000 7f6b86813640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T02:49:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.799+0000 7f6b86813640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:44.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.799+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6b80072690 con 0x7f6b800a4500 2024-09-11T02:49:44.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b7cff9640 1 -- 172.21.15.98:0/3038012881 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f6b78016cf0 con 0x7f6b800a4500 2024-09-11T02:49:44.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-11T02:49:44.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b7cff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-11T02:49:44.803 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-09-11T02:49:44.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 10 librados: watch_flush enter 2024-09-11T02:49:44.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 10 librados: watch_flush exit 2024-09-11T02:49:44.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 20 client.4122.objecter shutdown clearing up homeless session... 2024-09-11T02:49:44.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 10 client.4122.objecter successfully canceled tick 2024-09-11T02:49:44.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 10 monclient: shutdown 2024-09-11T02:49:44.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:44.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b800a4500 msgr2=0x7f6b800a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:44.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b800a4500 0x7f6b800a4920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6b8013a810 tx=0x7f6b78009f10 comp rx=0 tx=0).stop 2024-09-11T02:49:44.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 shutdown_connections 2024-09-11T02:49:44.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6b800a4500 0x7f6b800a4920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.803+0000 7f6b86813640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6b801d1490 0x7f6b800a3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.807+0000 7f6b86813640 1 --2- 172.21.15.98:0/3038012881 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6b80137940 0x7f6b80137d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:44.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.807+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 >> 172.21.15.98:0/3038012881 conn(0x7f6b800ac7f0 msgr2=0x7f6b80138a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:44.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.807+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 shutdown_connections 2024-09-11T02:49:44.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.807+0000 7f6b86813640 1 -- 172.21.15.98:0/3038012881 wait complete. 2024-09-11T02:49:44.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:44.807+0000 7f6b86813640 1 librados: shutdown 2024-09-11T02:49:44.822 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T02:49:44.823 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T02:49:44.823 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:44.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T02:49:44.835 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:44.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T02:49:44.889 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-11T02:49:44.889 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T02:49:44.945 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:44.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T02:49:44.962 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:44.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-11T02:49:45.017 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:49:45.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-11T02:49:45.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 84a3e39d-63f8-4735-a51d-6248655e4c80 0 2024-09-11T02:49:45.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient: get_monmap_and_config 2024-09-11T02:49:45.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient: monmap: 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:45.241220+0000 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:45.241220+0000 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.236 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.237 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.237 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.237 INFO:teuthology.orchestra.run.smithi190.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:45.237 INFO:teuthology.orchestra.run.smithi190.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:45.237 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.240 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 1 Processor -- start 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 1 -- start start 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient: init 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient: _add_conns ranks=[1,4,3,2,7,8,9,5,6,0,10] 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f57000a4710 addr v1:172.21.15.190:6789/0 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57000a5200 0x7f570013b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient(hunting): picked mon.noname-c con 0x7f57000a5200 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570013be50 0x7f570013e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient(hunting): picked mon.noname-g con 0x7f570013be50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57000a23b0 con 0x7f57000a4710 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5700070850 con 0x7f570013be50 2024-09-11T02:49:45.241 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f57000a0d50 con 0x7f57000a5200 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f57052e3640 10 monclient(hunting): authenticate will time out at 945.907842s 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f56feffd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f57000a4710 0x7f57000a4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:47312/0 (socket says 172.21.15.190:47312) 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f56feffd640 1 -- 172.21.15.190:0/2449724072 learned_addr learned my addr 172.21.15.190:0/2449724072 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.236+0000 7f56ff7fe640 1 --2- 172.21.15.190:0/2449724072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570013be50 0x7f570013e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): get_auth_request con 0x7f570013be50 auth_method 0 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.242 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fe7fc640 1 --2- 172.21.15.190:0/2449724072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57000a5200 0x7f570013b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 1 -- 172.21.15.190:0/2449724072 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651239085 0 0) 0x7f57000a23b0 con 0x7f57000a4710 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient(hunting): _negotiate 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient(hunting): my global_id is 4114 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 1 -- 172.21.15.190:0/2449724072 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e0003530 con 0x7f57000a4710 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fe7fc640 10 monclient(hunting): get_auth_request con 0x7f57000a5200 auth_method 0 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2024-09-11T02:49:45.243 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 1 -- 172.21.15.190:0/2449724072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57000a5200 msgr2=0x7f570013b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 1 --2- 172.21.15.190:0/2449724072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57000a5200 0x7f570013b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 1 -- 172.21.15.190:0/2449724072 >> v1:172.21.15.190:6789/0 conn(0x7f57000a4710 legacy=0x7f57000a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 1 monclient: found mon.noname-g 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 1 -- 172.21.15.190:0/2449724072 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57000a2690 con 0x7f570013be50 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56ff7fe640 1 --2- 172.21.15.190:0/2449724072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570013be50 0x7f570013e240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009dc0 tx=0x7f56f801c850 comp rx=0 tx=0).ready entity=mon.3 client_cookie=e351b23983d4ac6e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 1 -- 172.21.15.190:0/2449724072 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f56f80b2070 con 0x7f570013be50 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:49:45.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: dump: 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 1 -- 172.21.15.190:0/2449724072 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f80a5a50 con 0x7f570013be50 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 5 monclient: authenticate success, global_id 4107 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 4 set_mon_vals no callback set 2024-09-11T02:49:45.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 1 -- 172.21.15.190:0/2449724072 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f56f80a5c40 con 0x7f570013be50 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f56fdffb640 10 monclient: dump: 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 10 monclient: shutdown 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 -- 172.21.15.190:0/2449724072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570013be50 msgr2=0x7f570013e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 --2- 172.21.15.190:0/2449724072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570013be50 0x7f570013e240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009dc0 tx=0x7f56f801c850 comp rx=0 tx=0).stop 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 -- 172.21.15.190:0/2449724072 shutdown_connections 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 --2- 172.21.15.190:0/2449724072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f570013be50 0x7f570013e240 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 --2- 172.21.15.190:0/2449724072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57000a5200 0x7f570013b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 -- 172.21.15.190:0/2449724072 >> 172.21.15.190:0/2449724072 conn(0x7f57000a93e0 msgr2=0x7f57000a5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:45.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 -- 172.21.15.190:0/2449724072 shutdown_connections 2024-09-11T02:49:45.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.240+0000 7f57052e3640 1 -- 172.21.15.190:0/2449724072 wait complete. 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: monmap: 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 librados: starting msgr at 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 librados: starting objecter 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 Processor -- start 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 -- start start 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 librados: setting wanted keys 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 librados: calling monclient init 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: init 2024-09-11T02:49:45.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _add_conns ranks=[6,3,7,0,1,8,5,2,4] 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f57000a4710 0x7f570009d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): picked mon.noname-d con 0x7f57000a4710 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57000a5200 0x7f570013f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): picked mon.noname-g con 0x7f57000a5200 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f570013be50 0x7f570013fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): picked mon.noname-i con 0x7f570013be50 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f57000a0b30 con 0x7f57000a5200 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f57000a09b0 con 0x7f57000a4710 2024-09-11T02:49:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f570013ef90 con 0x7f570013be50 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient(hunting): authenticate will time out at 945.913356s 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57000a5200 0x7f570013f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57000a5200 0x7f570013f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:47276/0 (socket says 172.21.15.190:47276) 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 -- 172.21.15.190:0/957075084 learned_addr learned my addr 172.21.15.190:0/957075084 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): get_auth_request con 0x7f57000a5200 auth_method 0 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f570013be50 0x7f570013fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 10 monclient(hunting): get_auth_request con 0x7f570013be50 auth_method 0 2024-09-11T02:49:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56feffd640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f57000a4710 0x7f570009d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56feffd640 10 monclient(hunting): get_auth_request con 0x7f57000a4710 auth_method 0 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 -- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f570013be50 msgr2=0x7f570013fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f570013be50 0x7f570013fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 -- 172.21.15.190:0/957075084 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f57000a4710 msgr2=0x7f570009d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f57000a4710 0x7f570009d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 monclient: found mon.noname-g 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 -- 172.21.15.190:0/957075084 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570009f680 con 0x7f57000a5200 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56ff7fe640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f570013be50 0x7f570013fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fe7fc640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57000a5200 0x7f570013f790 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f56e800d830 tx=0x7f56e800dcf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 client.?.objecter ms_handle_connect 0x7f57000a5200 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 1 -- 172.21.15.190:0/957075084 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f56e800bb30 con 0x7f57000a5200 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: dump: 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 1 -- 172.21.15.190:0/957075084 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e8019070 con 0x7f57000a5200 2024-09-11T02:49:45.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 1 -- 172.21.15.190:0/957075084 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f56e800bcf0 con 0x7f57000a5200 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 5 monclient: authenticate success, global_id 4116 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _renew_subs 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570009df30 con 0x7f57000a5200 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 monclient: dump: 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.262 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56fd7fa640 4 set_mon_vals no callback set 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _renew_subs 2024-09-11T02:49:45.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5700140fd0 con 0x7f57000a5200 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 librados: init done 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 1 -- 172.21.15.190:0/957075084 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f56e8013070 con 0x7f57000a5200 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 1 -- 172.21.15.190:0/957075084 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f56e800f040 con 0x7f57000a5200 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 10 client.4116.objecter ms_dispatch 0x7f5700000ba0 osd_map(2..2 src has 1..2) 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f56f77fe640 20 client.4116.objecter dump_active .. 0 homeless 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.244+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f570013e460 con 0x7f57000a5200 2024-09-11T02:49:45.265 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.248+0000 7f56f77fe640 1 -- 172.21.15.190:0/957075084 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f570013e460 con 0x7f57000a5200 2024-09-11T02:49:45.265 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.248+0000 7f56f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.265 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.248+0000 7f56f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:45.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.320+0000 7f57052e3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "84a3e39d-63f8-4735-a51d-6248655e4c80", "id": 0}] 2024-09-11T02:49:45.318 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.320+0000 7f57052e3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "84a3e39d-63f8-4735-a51d-6248655e4c80", "id": 0}] 2024-09-11T02:49:45.318 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.320+0000 7f57052e3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.318 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.320+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "84a3e39d-63f8-4735-a51d-6248655e4c80", "id": 0} v 0) -- 0x7f570013f2f0 con 0x7f57000a5200 2024-09-11T02:49:45.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f56f77fe640 1 -- 172.21.15.190:0/957075084 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "84a3e39d-63f8-4735-a51d-6248655e4c80", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f56e8022020 con 0x7f57000a5200 2024-09-11T02:49:45.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f56f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "84a3e39d-63f8-4735-a51d-6248655e4c80", "id": 0}] 2024-09-11T02:49:45.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f56f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:45.326 INFO:teuthology.orchestra.run.smithi190.stdout:0 2024-09-11T02:49:45.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 10 librados: watch_flush enter 2024-09-11T02:49:45.327 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 10 librados: watch_flush exit 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 20 client.4116.objecter shutdown clearing up homeless session... 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 10 client.4116.objecter successfully canceled tick 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 10 monclient: shutdown 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57000a5200 msgr2=0x7f570013f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57000a5200 0x7f570013f790 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f56e800d830 tx=0x7f56e800dcf0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 shutdown_connections 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f570013be50 0x7f570013fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f57000a5200 0x7f570013f790 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 --2- 172.21.15.190:0/957075084 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f57000a4710 0x7f570009d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 >> 172.21.15.190:0/957075084 conn(0x7f57000a93e0 msgr2=0x7f57000a72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 shutdown_connections 2024-09-11T02:49:45.328 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 -- 172.21.15.190:0/957075084 wait complete. 2024-09-11T02:49:45.329 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.328+0000 7f57052e3640 1 librados: shutdown 2024-09-11T02:49:45.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 8753f916-e49b-493b-bfbc-790c2d4866a1 1 2024-09-11T02:49:45.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient: get_monmap_and_config 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient: monmap: 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:45.464495+0000 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:45.464495+0000 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.459 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:45.460 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 1 Processor -- start 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 1 -- start start 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient: init 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient: _add_conns ranks=[5,0,8,1,10,4,6,7,2,3,9] 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0fe80a4820 0x7f0fe80a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient(hunting): picked mon.noname-h con 0x7f0fe80a4820 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fe806d2e0 addr v1:172.21.15.98:6789/0 2024-09-11T02:49:45.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0fe80a5140 0x7f0fe813a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient(hunting): picked mon.noname-e con 0x7f0fe80a5140 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe80a0bb0 con 0x7f0fe806d2e0 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f0fe80a14e0 con 0x7f0fe80a4820 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0fe80a55f0 con 0x7f0fe80a5140 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fef709640 10 monclient(hunting): authenticate will time out at 946.130965s 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0fe80a4820 0x7f0fe80a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0fe80a4820 0x7f0fe80a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.190:60822/0 (socket says 172.21.15.190:60822) 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 -- 172.21.15.190:0/1961816741 learned_addr learned my addr 172.21.15.190:0/1961816741 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): get_auth_request con 0x7f0fe80a4820 auth_method 0 2024-09-11T02:49:45.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fedca6640 1 --2- 172.21.15.190:0/1961816741 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0fe80a5140 0x7f0fe813a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fedca6640 10 monclient(hunting): get_auth_request con 0x7f0fe80a5140 auth_method 0 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fedca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fedca6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fedca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 -- 172.21.15.190:0/1961816741 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0fe80a5140 msgr2=0x7f0fe813a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:45.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 --2- 172.21.15.190:0/1961816741 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0fe80a5140 0x7f0fe813a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 -- 172.21.15.190:0/1961816741 >> v1:172.21.15.98:6789/0 conn(0x7f0fe806d2e0 legacy=0x7f0fe81399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 monclient: found mon.noname-h 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.460+0000 7f0fed4a5640 1 -- 172.21.15.190:0/1961816741 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe80a6310 con 0x7f0fe80a4820 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fed4a5640 1 --2- 172.21.15.190:0/1961816741 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0fe80a4820 0x7f0fe80a4c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80098a0 tx=0x7f0fd801c440 comp rx=0 tx=0).ready entity=mon.5 client_cookie=8257f4f1afc6b38d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 1 -- 172.21.15.190:0/1961816741 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0fd80b1070 con 0x7f0fe80a4820 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: dump: 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.466 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.467 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 1 -- 172.21.15.190:0/1961816741 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fd80aa7b0 con 0x7f0fe80a4820 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 1 -- 172.21.15.190:0/1961816741 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0fd80aae10 con 0x7f0fe80a4820 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 5 monclient: authenticate success, global_id 4118 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 4 set_mon_vals no callback set 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fdffff640 10 monclient: dump: 2024-09-11T02:49:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.469 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.470 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.470 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 10 monclient: shutdown 2024-09-11T02:49:45.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:45.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 -- 172.21.15.190:0/1961816741 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0fe80a4820 msgr2=0x7f0fe80a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 --2- 172.21.15.190:0/1961816741 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0fe80a4820 0x7f0fe80a4c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80098a0 tx=0x7f0fd801c440 comp rx=0 tx=0).stop 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 -- 172.21.15.190:0/1961816741 shutdown_connections 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 --2- 172.21.15.190:0/1961816741 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0fe80a5140 0x7f0fe813a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 --2- 172.21.15.190:0/1961816741 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0fe80a4820 0x7f0fe80a4c00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 -- 172.21.15.190:0/1961816741 >> 172.21.15.190:0/1961816741 conn(0x7f0fe80a9410 msgr2=0x7f0fe80a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 -- 172.21.15.190:0/1961816741 shutdown_connections 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 -- 172.21.15.190:0/1961816741 wait complete. 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 10 monclient: monmap: 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:45.475 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.476 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 librados: starting msgr at 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 librados: starting objecter 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 Processor -- start 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 -- start start 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 librados: setting wanted keys 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 1 librados: calling monclient init 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.464+0000 7f0fef709640 10 monclient: init 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _add_conns ranks=[2,3,6,0,5,1,4,8,7] 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80a4820 0x7f0fe813f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe80a4820 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fe80a5140 0x7f0fe81d19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): picked mon.noname-g con 0x7f0fe80a5140 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0fe81d1ef0 0x7f0fe813d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): picked mon.noname-d con 0x7f0fe81d1ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe813a830 con 0x7f0fe80a4820 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe806ddd0 con 0x7f0fe80a5140 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0fe81444c0 con 0x7f0fe81d1ef0 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient(hunting): authenticate will time out at 946.136321s 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fe80a5140 0x7f0fe81d19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fe80a5140 0x7f0fe81d19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:47290/0 (socket says 172.21.15.190:47290) 2024-09-11T02:49:45.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 -- 172.21.15.190:0/3330909249 learned_addr learned my addr 172.21.15.190:0/3330909249 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): get_auth_request con 0x7f0fe80a5140 auth_method 0 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fed4a5640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80a4820 0x7f0fe813f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fedca6640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0fe81d1ef0 0x7f0fe813d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fed4a5640 10 monclient(hunting): get_auth_request con 0x7f0fe80a4820 auth_method 0 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fed4a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fed4a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fed4a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fedca6640 10 monclient(hunting): get_auth_request con 0x7f0fe81d1ef0 auth_method 0 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fedca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fedca6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fedca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient(hunting): handle_auth_done global_id 4125 payload 824 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 -- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0fe81d1ef0 msgr2=0x7f0fe813d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0fe81d1ef0 0x7f0fe813d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 -- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80a4820 msgr2=0x7f0fe813f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80a4820 0x7f0fe813f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 monclient: found mon.noname-g 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 -- 172.21.15.190:0/3330909249 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe813f530 con 0x7f0fe80a5140 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fecca4640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fe80a5140 0x7f0fe81d19b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4007fd0 tx=0x7f0fe400c890 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 client.?.objecter ms_handle_connect 0x7f0fe80a5140 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 1 -- 172.21.15.190:0/3330909249 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0fe4017070 con 0x7f0fe80a5140 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: dump: 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.482 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 5 monclient: authenticate success, global_id 4125 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _renew_subs 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 1 -- 172.21.15.190:0/3330909249 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe4003600 con 0x7f0fe80a5140 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe81239f0 con 0x7f0fe80a5140 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 1 -- 172.21.15.190:0/3330909249 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0fe4003ca0 con 0x7f0fe80a5140 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fdf7fe640 4 set_mon_vals no callback set 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 monclient: dump: 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.483 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.484 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _renew_subs 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe80a71b0 con 0x7f0fe80a5140 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 librados: init done 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 1 -- 172.21.15.190:0/3330909249 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f0fe40050d0 con 0x7f0fe80a5140 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 1 -- 172.21.15.190:0/3330909249 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f0fe4007160 con 0x7f0fe80a5140 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 10 client.4125.objecter ms_dispatch 0x7f0fe8000ba0 osd_map(3..3 src has 1..3) 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fddffb640 20 client.4125.objecter dump_active .. 0 homeless 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.486 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.486 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.468+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe80a4c00 con 0x7f0fe80a5140 2024-09-11T02:49:45.486 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.472+0000 7f0fddffb640 1 -- 172.21.15.190:0/3330909249 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f0fe40073f0 con 0x7f0fe80a5140 2024-09-11T02:49:45.486 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.472+0000 7f0fddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.486 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.472+0000 7f0fddffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:45.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.540+0000 7f0fef709640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8753f916-e49b-493b-bfbc-790c2d4866a1", "id": 1}] 2024-09-11T02:49:45.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.540+0000 7f0fef709640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8753f916-e49b-493b-bfbc-790c2d4866a1", "id": 1}] 2024-09-11T02:49:45.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.540+0000 7f0fef709640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.540+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8753f916-e49b-493b-bfbc-790c2d4866a1", "id": 1} v 0) -- 0x7f0fe8073360 con 0x7f0fe80a5140 2024-09-11T02:49:45.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.548+0000 7f0fddffb640 1 -- 172.21.15.190:0/3330909249 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8753f916-e49b-493b-bfbc-790c2d4866a1", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f0fe40075d0 con 0x7f0fe80a5140 2024-09-11T02:49:45.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.548+0000 7f0fddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8753f916-e49b-493b-bfbc-790c2d4866a1", "id": 1}] 2024-09-11T02:49:45.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.548+0000 7f0fddffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:45.549 INFO:teuthology.orchestra.run.smithi190.stdout:1 2024-09-11T02:49:45.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 10 librados: watch_flush enter 2024-09-11T02:49:45.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 10 librados: watch_flush exit 2024-09-11T02:49:45.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 20 client.4125.objecter shutdown clearing up homeless session... 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 10 client.4125.objecter successfully canceled tick 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 10 monclient: shutdown 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fe80a5140 msgr2=0x7f0fe81d19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fe80a5140 0x7f0fe81d19b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4007fd0 tx=0x7f0fe400c890 comp rx=0 tx=0).stop 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 shutdown_connections 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0fe81d1ef0 0x7f0fe813d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0fe80a5140 0x7f0fe81d19b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 --2- 172.21.15.190:0/3330909249 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0fe80a4820 0x7f0fe813f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 >> 172.21.15.190:0/3330909249 conn(0x7f0fe80a9410 msgr2=0x7f0fe8141490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 shutdown_connections 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 -- 172.21.15.190:0/3330909249 wait complete. 2024-09-11T02:49:45.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.552+0000 7f0fef709640 1 librados: shutdown 2024-09-11T02:49:45.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 0c1bb788-0d6d-49eb-aef4-a94a3b1569cb 2 2024-09-11T02:49:45.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient: get_monmap_and_config 2024-09-11T02:49:45.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient: monmap: 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:45.693353+0000 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:45.693353+0000 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.688 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:45.689 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 Processor -- start 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 -- start start 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient: init 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient: _add_conns ranks=[10,6,9,5,1,3,0,4,8,7,2] 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff4640a58c0 0x7ff4640a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): picked mon.noname-f con 0x7ff4640a58c0 addr v2:172.21.15.190:3300/0 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff464071d70 0x7ff4640a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): picked mon.noname-d con 0x7ff464071d70 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:45.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff4641376e0 0x7ff464137b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): picked mon.noname-a con 0x7ff4641376e0 addr v2:172.21.15.98:3300/0 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff4640a6720 con 0x7ff4641376e0 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff464073850 con 0x7ff4640a58c0 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff464073b30 con 0x7ff464071d70 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff469795640 10 monclient(hunting): authenticate will time out at 946.359777s 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.688+0000 7ff462ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff4640a58c0 0x7ff4640a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff4640a58c0 0x7ff4640a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:59920/0 (socket says 172.21.15.190:59920) 2024-09-11T02:49:45.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 -- 172.21.15.190:0/725881719 learned_addr learned my addr 172.21.15.190:0/725881719 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): get_auth_request con 0x7ff4640a58c0 auth_method 0 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4627fc640 1 --2- 172.21.15.190:0/725881719 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff464071d70 0x7ff4640a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4627fc640 10 monclient(hunting): get_auth_request con 0x7ff464071d70 auth_method 0 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4637fe640 1 --2- 172.21.15.190:0/725881719 >> v2:172.21.15.98:3300/0 conn(0x7ff4641376e0 0x7ff464137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 -- 172.21.15.190:0/725881719 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff464071d70 msgr2=0x7ff4640a61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 --2- 172.21.15.190:0/725881719 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff464071d70 0x7ff4640a61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 -- 172.21.15.190:0/725881719 >> v2:172.21.15.98:3300/0 conn(0x7ff4641376e0 msgr2=0x7ff464137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 --2- 172.21.15.190:0/725881719 >> v2:172.21.15.98:3300/0 conn(0x7ff4641376e0 0x7ff464137b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 monclient: found mon.noname-f 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 -- 172.21.15.190:0/725881719 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff464073fc0 con 0x7ff4640a58c0 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4637fe640 10 monclient: get_auth_request con 0x7ff4641376e0 auth_method 0 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff4637fe640 1 --2- 172.21.15.190:0/725881719 >> v2:172.21.15.98:3300/0 conn(0x7ff4641376e0 0x7ff464137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:49:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff462ffd640 1 --2- 172.21.15.190:0/725881719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4640a58c0 0x7ff4640a5ca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff46406d110 tx=0x7ff45c01c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9ef9417580ca7cd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 1 -- 172.21.15.190:0/725881719 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff45c0090d0 con 0x7ff4640a58c0 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: dump: 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.696 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 1 -- 172.21.15.190:0/725881719 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff45c0a5ab0 con 0x7ff4640a58c0 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 5 monclient: authenticate success, global_id 4123 2024-09-11T02:49:45.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 1 -- 172.21.15.190:0/725881719 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff45c0a5c60 con 0x7ff4640a58c0 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 4 set_mon_vals no callback set 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff461ffb640 10 monclient: dump: 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.698 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.699 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 10 monclient: shutdown 2024-09-11T02:49:45.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:45.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 -- 172.21.15.190:0/725881719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4640a58c0 msgr2=0x7ff4640a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 --2- 172.21.15.190:0/725881719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4640a58c0 0x7ff4640a5ca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff46406d110 tx=0x7ff45c01c8b0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 -- 172.21.15.190:0/725881719 shutdown_connections 2024-09-11T02:49:45.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 --2- 172.21.15.190:0/725881719 >> v2:172.21.15.98:3300/0 conn(0x7ff4641376e0 0x7ff464137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 --2- 172.21.15.190:0/725881719 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff464071d70 0x7ff4640a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 --2- 172.21.15.190:0/725881719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4640a58c0 0x7ff4640a5ca0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 -- 172.21.15.190:0/725881719 >> 172.21.15.190:0/725881719 conn(0x7ff4640aa4b0 msgr2=0x7ff4640a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 -- 172.21.15.190:0/725881719 shutdown_connections 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.692+0000 7ff469795640 1 -- 172.21.15.190:0/725881719 wait complete. 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: monmap: 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T02:49:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 librados: starting msgr at 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 librados: starting objecter 2024-09-11T02:49:45.706 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 Processor -- start 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 -- start start 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 librados: setting wanted keys 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 librados: calling monclient init 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: init 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _add_conns ranks=[8,3,2,6,1,0,4,7,5] 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4640a58c0 0x7ff464133550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): picked mon.noname-e con 0x7ff4640a58c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4641376e0 0x7ff464133a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): picked mon.noname-g con 0x7ff4641376e0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff46409d890 0x7ff46409dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): picked mon.noname-b con 0x7ff46409d890 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:49:45.707 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff464141340 con 0x7ff46409d890 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff4641411c0 con 0x7ff4641376e0 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff4641414c0 con 0x7ff4640a58c0 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient(hunting): authenticate will time out at 946.365260s 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4641376e0 0x7ff464133a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4641376e0 0x7ff464133a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:47292/0 (socket says 172.21.15.190:47292) 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 -- 172.21.15.190:0/869460841 learned_addr learned my addr 172.21.15.190:0/869460841 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): get_auth_request con 0x7ff4641376e0 auth_method 0 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.708 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4637fe640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff46409d890 0x7ff46409dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4637fe640 10 monclient(hunting): get_auth_request con 0x7ff46409d890 auth_method 0 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff462ffd640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4640a58c0 0x7ff464133550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4637fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff462ffd640 10 monclient(hunting): get_auth_request con 0x7ff4640a58c0 auth_method 0 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff462ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff462ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 -- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4640a58c0 msgr2=0x7ff464133550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4640a58c0 0x7ff464133550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 -- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff46409d890 msgr2=0x7ff46409dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff46409d890 0x7ff46409dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 monclient: found mon.noname-g 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 -- 172.21.15.190:0/869460841 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff464134410 con 0x7ff4641376e0 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4627fc640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4641376e0 0x7ff464133a90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00d830 tx=0x7ff44c00dcf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.710 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 client.?.objecter ms_handle_connect 0x7ff4641376e0 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 1 -- 172.21.15.190:0/869460841 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff44c00bb30 con 0x7ff4641376e0 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: dump: 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.711 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 1 -- 172.21.15.190:0/869460841 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c019070 con 0x7ff4641376e0 2024-09-11T02:49:45.712 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 5 monclient: authenticate success, global_id 4134 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _renew_subs 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4641c6860 con 0x7ff4641376e0 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 1 -- 172.21.15.190:0/869460841 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff44c00bcf0 con 0x7ff4641376e0 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 monclient: dump: 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.713 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.714 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff4617fa640 4 set_mon_vals no callback set 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _renew_subs 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff46409f340 con 0x7ff4641376e0 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 librados: init done 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 1 -- 172.21.15.190:0/869460841 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7ff44c013070 con 0x7ff4641376e0 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 1 -- 172.21.15.190:0/869460841 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7ff44c00f040 con 0x7ff4641376e0 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 10 client.4134.objecter ms_dispatch 0x7ff464000ba0 osd_map(4..4 src has 1..4) 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 3 client.4134.objecter handle_osd_map got epochs [4,4] > 0 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 3 client.4134.objecter handle_osd_map decoding full epoch 4 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff45b7fe640 20 client.4134.objecter dump_active .. 0 homeless 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.696+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff464134da0 con 0x7ff4641376e0 2024-09-11T02:49:45.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.700+0000 7ff45b7fe640 1 -- 172.21.15.190:0/869460841 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7ff44c022020 con 0x7ff4641376e0 2024-09-11T02:49:45.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.700+0000 7ff45b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.700+0000 7ff45b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:45.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.796+0000 7ff469795640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0c1bb788-0d6d-49eb-aef4-a94a3b1569cb", "id": 2}] 2024-09-11T02:49:45.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.796+0000 7ff469795640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0c1bb788-0d6d-49eb-aef4-a94a3b1569cb", "id": 2}] 2024-09-11T02:49:45.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.796+0000 7ff469795640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.794 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.796+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0c1bb788-0d6d-49eb-aef4-a94a3b1569cb", "id": 2} v 0) -- 0x7ff464137b50 con 0x7ff4641376e0 2024-09-11T02:49:45.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.804+0000 7ff45b7fe640 1 -- 172.21.15.190:0/869460841 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0c1bb788-0d6d-49eb-aef4-a94a3b1569cb", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7ff4641c6860 con 0x7ff4641376e0 2024-09-11T02:49:45.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.804+0000 7ff45b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0c1bb788-0d6d-49eb-aef4-a94a3b1569cb", "id": 2}] 2024-09-11T02:49:45.804 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.804+0000 7ff45b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:45.804 INFO:teuthology.orchestra.run.smithi190.stdout:2 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 10 librados: watch_flush enter 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 10 librados: watch_flush exit 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 20 client.4134.objecter shutdown clearing up homeless session... 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 10 client.4134.objecter successfully canceled tick 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 10 monclient: shutdown 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4641376e0 msgr2=0x7ff464133a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4641376e0 0x7ff464133a90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00d830 tx=0x7ff44c00dcf0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 shutdown_connections 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff46409d890 0x7ff46409dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.806 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff4641376e0 0x7ff464133a90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.807 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 --2- 172.21.15.190:0/869460841 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff4640a58c0 0x7ff464133550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.807 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 >> 172.21.15.190:0/869460841 conn(0x7ff4640aa4b0 msgr2=0x7ff46413d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:45.807 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 shutdown_connections 2024-09-11T02:49:45.807 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 -- 172.21.15.190:0/869460841 wait complete. 2024-09-11T02:49:45.807 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.808+0000 7ff469795640 1 librados: shutdown 2024-09-11T02:49:45.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new f33219e6-7712-40ac-8140-765749273dc8 3 2024-09-11T02:49:45.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient: get_monmap_and_config 2024-09-11T02:49:45.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient: monmap: 2024-09-11T02:49:45.933 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:45.939327+0000 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:45.939327+0000 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.934 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.935 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.935 INFO:teuthology.orchestra.run.smithi190.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:45.935 INFO:teuthology.orchestra.run.smithi190.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:45.935 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 Processor -- start 2024-09-11T02:49:45.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 -- start start 2024-09-11T02:49:45.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient: init 2024-09-11T02:49:45.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient: _add_conns ranks=[6,3,10,0,4,9,8,2,5,7,1] 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c0a4820 0x7f7a8c0a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): picked mon.noname-d con 0x7f7a8c0a4820 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a8c09fcf0 0x7f7a8c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): picked mon.noname-g con 0x7f7a8c09fcf0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7a8c1365d0 0x7f7a8c1369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): picked mon.noname-f con 0x7f7a8c1365d0 addr v2:172.21.15.190:3300/0 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7a8c0a0bf0 con 0x7f7a8c1365d0 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8c0a5680 con 0x7f7a8c09fcf0 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7a8c0a2ef0 con 0x7f7a8c0a4820 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a91b0b640 10 monclient(hunting): authenticate will time out at 946.605729s 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a8c09fcf0 0x7f7a8c0a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7a8c1365d0 0x7f7a8c1369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a8c09fcf0 0x7f7a8c0a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.190:47304/0 (socket says 172.21.15.190:47304) 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7a8c1365d0 0x7f7a8c1369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:59924/0 (socket says 172.21.15.190:59924) 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 -- 172.21.15.190:0/3741678297 learned_addr learned my addr 172.21.15.190:0/3741678297 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): get_auth_request con 0x7f7a8c09fcf0 auth_method 0 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request con 0x7f7a8c1365d0 auth_method 0 2024-09-11T02:49:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 -- 172.21.15.190:0/3741678297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c0a4820 msgr2=0x7f7a8c0a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 --2- 172.21.15.190:0/3741678297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c0a4820 0x7f7a8c0a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 -- 172.21.15.190:0/3741678297 >> v2:172.21.15.190:3300/0 conn(0x7f7a8c1365d0 msgr2=0x7f7a8c1369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 --2- 172.21.15.190:0/3741678297 >> v2:172.21.15.190:3300/0 conn(0x7f7a8c1365d0 0x7f7a8c1369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 monclient: found mon.noname-g 2024-09-11T02:49:45.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:49:45.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8affd640 1 -- 172.21.15.190:0/3741678297 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c0a2530 con 0x7f7a8c09fcf0 2024-09-11T02:49:45.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.936+0000 7f7a8bfff640 1 --2- 172.21.15.190:0/3741678297 >> v2:172.21.15.190:3300/0 conn(0x7f7a8c1365d0 0x7f7a8c1369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:49:45.945 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 1 --2- 172.21.15.190:0/3741678297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a8c09fcf0 0x7f7a8c0a5140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000bb10 tx=0x7f7a8001eaf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=5e506d3e6b27f6e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 1 -- 172.21.15.190:0/3741678297 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a800b4070 con 0x7f7a8c09fcf0 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: dump: 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.946 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 1 -- 172.21.15.190:0/3741678297 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a800a7ab0 con 0x7f7a8c09fcf0 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 1 -- 172.21.15.190:0/3741678297 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a800a7ce0 con 0x7f7a8c09fcf0 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 5 monclient: authenticate success, global_id 4143 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 4 set_mon_vals no callback set 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8a7fc640 10 monclient: dump: 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.948 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient: shutdown 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- 172.21.15.190:0/3741678297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a8c09fcf0 msgr2=0x7f7a8c0a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/3741678297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a8c09fcf0 0x7f7a8c0a5140 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000bb10 tx=0x7f7a8001eaf0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- 172.21.15.190:0/3741678297 shutdown_connections 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/3741678297 >> v2:172.21.15.190:3300/0 conn(0x7f7a8c1365d0 0x7f7a8c1369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/3741678297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7a8c09fcf0 0x7f7a8c0a5140 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/3741678297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c0a4820 0x7f7a8c0a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- 172.21.15.190:0/3741678297 >> 172.21.15.190:0/3741678297 conn(0x7f7a8c0a9410 msgr2=0x7f7a8c0a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- 172.21.15.190:0/3741678297 shutdown_connections 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- 172.21.15.190:0/3741678297 wait complete. 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient: build_initial_monmap 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient: monmap: 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T02:49:45.950 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 librados: starting msgr at 2024-09-11T02:49:45.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 librados: starting objecter 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 Processor -- start 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- start start 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 librados: setting wanted keys 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 librados: calling monclient init 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient: init 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient: _add_conns ranks=[0,4,6,8,3,5,1,2,7] 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c09fcf0 0x7f7a8c1cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8c09fcf0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a8c0a4820 0x7f7a8c1cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): picked mon.noname-c con 0x7f7a8c0a4820 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c1365d0 0x7f7a8c1ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:45.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): picked mon.noname-d con 0x7f7a8c1365d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c06d690 con 0x7f7a8c09fcf0 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7a8c0a62e0 con 0x7f7a8c0a4820 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7a8c06d810 con 0x7f7a8c1365d0 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a91b0b640 10 monclient(hunting): authenticate will time out at 946.611354s 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a8c0a4820 0x7f7a8c1cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a8c0a4820 0x7f7a8c1cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.190:35582/0 (socket says 172.21.15.190:35582) 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 1 -- 172.21.15.190:0/139936638 learned_addr learned my addr 172.21.15.190:0/139936638 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:45.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 10 monclient(hunting): get_auth_request con 0x7f7a8c0a4820 auth_method 0 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c1365d0 0x7f7a8c1ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8b7fe640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c09fcf0 0x7f7a8c1cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8b7fe640 10 monclient(hunting): get_auth_request con 0x7f7a8c09fcf0 auth_method 0 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request con 0x7f7a8c1365d0 auth_method 0 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:45.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.940+0000 7f7a8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 1 -- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c1365d0 msgr2=0x7f7a8c1ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c1365d0 0x7f7a8c1ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 1 -- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c09fcf0 msgr2=0x7f7a8c1cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c09fcf0 0x7f7a8c1cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 10 monclient: _finish_hunting 0 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 1 monclient: found mon.noname-c 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 1 -- 172.21.15.190:0/139936638 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c11f920 con 0x7f7a8c0a4820 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a8affd640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a8c0a4820 0x7f7a8c1cdb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000b2c0 tx=0x7f7a8001b890 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 client.?.objecter ms_handle_connect 0x7f7a8c0a4820 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 1 -- 172.21.15.190:0/139936638 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a800b4040 con 0x7f7a8c0a4820 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: dump: 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.957 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: _finish_auth 0 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: _check_auth_tickets 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 5 monclient: authenticate success, global_id 4117 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 1 -- 172.21.15.190:0/139936638 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a80004260 con 0x7f7a8c0a4820 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 monclient: _renew_subs 2024-09-11T02:49:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8c0a50a0 con 0x7f7a8c0a4820 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 monclient: _renew_subs 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8c0a4c00 con 0x7f7a8c0a4820 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 1 librados: init done 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 1 -- 172.21.15.190:0/139936638 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a80004450 con 0x7f7a8c0a4820 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a89ffb640 4 set_mon_vals no callback set 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 monclient: dump: 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:45.960 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 1 -- 172.21.15.190:0/139936638 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f7a800b7070 con 0x7f7a8c0a4820 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 1 -- 172.21.15.190:0/139936638 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7a800af440 con 0x7f7a8c0a4820 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 10 client.4117.objecter ms_dispatch 0x7f7a8c000ba0 osd_map(5..5 src has 1..5) 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 3 client.4117.objecter handle_osd_map got epochs [5,5] > 0 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 3 client.4117.objecter handle_osd_map decoding full epoch 5 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a73fff640 20 client.4117.objecter dump_active .. 0 homeless 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:45.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.944+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8c0732e0 con 0x7f7a8c0a4820 2024-09-11T02:49:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.952+0000 7f7a73fff640 1 -- 172.21.15.190:0/139936638 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f7a8c0a50a0 con 0x7f7a8c0a4820 2024-09-11T02:49:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.952+0000 7f7a73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:45.952+0000 7f7a73fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:46.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.020+0000 7f7a91b0b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f33219e6-7712-40ac-8140-765749273dc8", "id": 3}] 2024-09-11T02:49:46.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.020+0000 7f7a91b0b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f33219e6-7712-40ac-8140-765749273dc8", "id": 3}] 2024-09-11T02:49:46.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.020+0000 7f7a91b0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:46.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.020+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd new", "uuid": "f33219e6-7712-40ac-8140-765749273dc8", "id": 3} v 0) -- 0x7f7a8c06db10 con 0x7f7a8c0a4820 2024-09-11T02:49:46.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a73fff640 1 -- 172.21.15.190:0/139936638 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f33219e6-7712-40ac-8140-765749273dc8", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7a8c11f920 con 0x7f7a8c0a4820 2024-09-11T02:49:46.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f33219e6-7712-40ac-8140-765749273dc8", "id": 3}] 2024-09-11T02:49:46.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a73fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:46.033 INFO:teuthology.orchestra.run.smithi190.stdout:3 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 10 librados: watch_flush enter 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 10 librados: watch_flush exit 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 20 client.4117.objecter shutdown clearing up homeless session... 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 10 client.4117.objecter successfully canceled tick 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 10 monclient: shutdown 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a8c0a4820 msgr2=0x7f7a8c1cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a8c0a4820 0x7f7a8c1cdb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000b2c0 tx=0x7f7a8001b890 comp rx=0 tx=0).stop 2024-09-11T02:49:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 shutdown_connections 2024-09-11T02:49:46.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7a8c1365d0 0x7f7a8c1ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7a8c0a4820 0x7f7a8c1cdb80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 --2- 172.21.15.190:0/139936638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c09fcf0 0x7f7a8c1cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 >> 172.21.15.190:0/139936638 conn(0x7f7a8c0a9410 msgr2=0x7f7a8c0a1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:46.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 shutdown_connections 2024-09-11T02:49:46.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 -- 172.21.15.190:0/139936638 wait complete. 2024-09-11T02:49:46.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.036+0000 7f7a91b0b640 1 librados: shutdown 2024-09-11T02:49:46.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 655b63d7-a358-4460-85ff-bf7b2330f230 4 2024-09-11T02:49:46.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient: get_monmap_and_config 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient: build_initial_monmap 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient: monmap: 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:46.164722+0000 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:46.164722+0000 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:46.159 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:46.160 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 1 Processor -- start 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 1 -- start start 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient: init 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient: _add_conns ranks=[7,0,4,10,5,6,3,8,2,1,9] 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f079c0a4820 0x7f079c0a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient(hunting): picked mon.noname-i con 0x7f079c0a4820 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f079c09fcf0 addr v1:172.21.15.98:6789/0 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a5140 0x7f079c136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient(hunting): picked mon.noname-c con 0x7f079c0a5140 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f079c0a2ef0 con 0x7f079c09fcf0 2024-09-11T02:49:46.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f079c0a0bf0 con 0x7f079c0a5140 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f079c0a2530 con 0x7f079c0a4820 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a39a2640 10 monclient(hunting): authenticate will time out at 946.831283s 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f079c0a4820 0x7f079c0a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f079c0a4820 0x7f079c0a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.190:56114/0 (socket says 172.21.15.190:56114) 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 -- 172.21.15.190:0/1846579549 learned_addr learned my addr 172.21.15.190:0/1846579549 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): get_auth_request con 0x7f079c0a4820 auth_method 0 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a1f3f640 1 --2- 172.21.15.190:0/1846579549 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a5140 0x7f079c136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a1f3f640 10 monclient(hunting): get_auth_request con 0x7f079c0a5140 auth_method 0 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a1f3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a1f3f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a1f3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 -- 172.21.15.190:0/1846579549 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a5140 msgr2=0x7f079c136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 --2- 172.21.15.190:0/1846579549 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a5140 0x7f079c136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 -- 172.21.15.190:0/1846579549 >> v1:172.21.15.98:6789/0 conn(0x7f079c09fcf0 legacy=0x7f079c1365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient: _finish_hunting 0 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 monclient: found mon.noname-i 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.160+0000 7f07a173e640 1 -- 172.21.15.190:0/1846579549 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c0a0970 con 0x7f079c0a4820 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a173e640 1 --2- 172.21.15.190:0/1846579549 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f079c0a4820 0x7f079c0a4c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f079400bb40 tx=0x7f079401eb10 comp rx=0 tx=0).ready entity=mon.7 client_cookie=b0b427bdc46c795e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 1 -- 172.21.15.190:0/1846579549 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07940b4070 con 0x7f079c0a4820 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: dump: 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.168 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: _finish_auth 0 2024-09-11T02:49:46.169 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 1 -- 172.21.15.190:0/1846579549 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07940a7b10 con 0x7f079c0a4820 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 1 -- 172.21.15.190:0/1846579549 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07940a7d40 con 0x7f079c0a4820 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 5 monclient: authenticate success, global_id 4120 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 4 set_mon_vals no callback set 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f079bfff640 10 monclient: dump: 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.170 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.171 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 10 monclient: shutdown 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 -- 172.21.15.190:0/1846579549 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f079c0a4820 msgr2=0x7f079c0a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 --2- 172.21.15.190:0/1846579549 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f079c0a4820 0x7f079c0a4c00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f079400bb40 tx=0x7f079401eb10 comp rx=0 tx=0).stop 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 -- 172.21.15.190:0/1846579549 shutdown_connections 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 --2- 172.21.15.190:0/1846579549 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a5140 0x7f079c136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 --2- 172.21.15.190:0/1846579549 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f079c0a4820 0x7f079c0a4c00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 -- 172.21.15.190:0/1846579549 >> 172.21.15.190:0/1846579549 conn(0x7f079c0a9410 msgr2=0x7f079c0a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 -- 172.21.15.190:0/1846579549 shutdown_connections 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 -- 172.21.15.190:0/1846579549 wait complete. 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 10 monclient: build_initial_monmap 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 10 monclient: monmap: 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:46.172 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:46.173 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 librados: starting msgr at 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 librados: starting objecter 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 Processor -- start 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 -- start start 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 librados: setting wanted keys 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 1 librados: calling monclient init 2024-09-11T02:49:46.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.164+0000 7f07a39a2640 10 monclient: init 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _add_conns ranks=[8,4,5,3,2,6,1,7,0] 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f079c09fcf0 0x7f079c1d1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): picked mon.noname-e con 0x7f079c09fcf0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a4820 0x7f079c1d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): picked mon.noname-c con 0x7f079c0a4820 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f079c0a5140 0x7f079c13a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): picked mon.noname-h con 0x7f079c0a5140 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:49:46.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f079c1373a0 con 0x7f079c0a4820 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f079c137220 con 0x7f079c0a5140 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f079c06d690 con 0x7f079c09fcf0 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient(hunting): authenticate will time out at 946.836496s 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f079c0a5140 0x7f079c13a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f079c0a5140 0x7f079c13a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.190:60832/0 (socket says 172.21.15.190:60832) 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 -- 172.21.15.190:0/1078959429 learned_addr learned my addr 172.21.15.190:0/1078959429 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): get_auth_request con 0x7f079c0a5140 auth_method 0 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a0f3d640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a4820 0x7f079c1d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a173e640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f079c09fcf0 0x7f079c1d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a0f3d640 10 monclient(hunting): get_auth_request con 0x7f079c0a4820 auth_method 0 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a0f3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a0f3d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a0f3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a173e640 10 monclient(hunting): get_auth_request con 0x7f079c09fcf0 auth_method 0 2024-09-11T02:49:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a173e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a173e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a173e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient(hunting): handle_auth_done global_id 4127 payload 824 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 -- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f079c09fcf0 msgr2=0x7f079c1d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f079c09fcf0 0x7f079c1d1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 -- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a4820 msgr2=0x7f079c1d1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a4820 0x7f079c1d1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient: _finish_hunting 0 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 monclient: found mon.noname-h 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:49:46.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 -- 172.21.15.190:0/1078959429 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c124410 con 0x7f079c0a5140 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a1f3f640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f079c0a5140 0x7f079c13a7a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f078c00d560 tx=0x7f078c00da20 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 client.?.objecter ms_handle_connect 0x7f079c0a5140 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 1 -- 172.21.15.190:0/1078959429 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f078c012a60 con 0x7f079c0a5140 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: dump: 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.185 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: _finish_auth 0 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:49:46.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 5 monclient: authenticate success, global_id 4127 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 1 -- 172.21.15.190:0/1078959429 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078c019070 con 0x7f079c0a5140 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 1 -- 172.21.15.190:0/1078959429 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f078c012c60 con 0x7f079c0a5140 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f079b7fe640 4 set_mon_vals no callback set 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 monclient: dump: 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.187 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.188 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _renew_subs 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079c123870 con 0x7f079c0a5140 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _renew_subs 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c0a7f20 con 0x7f079c0a5140 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 librados: init done 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:49:46.189 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f079c13c1f0 con 0x7f079c0a5140 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 1 -- 172.21.15.190:0/1078959429 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f078c013070 con 0x7f079c0a5140 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 1 -- 172.21.15.190:0/1078959429 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f078c00f040 con 0x7f079c0a5140 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 10 client.4127.objecter ms_dispatch 0x7f079c000ba0 osd_map(6..6 src has 1..6) 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 3 client.4127.objecter handle_osd_map got epochs [6,6] > 0 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 3 client.4127.objecter handle_osd_map decoding full epoch 6 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.168+0000 7f0799ffb640 20 client.4127.objecter dump_active .. 0 homeless 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.172+0000 7f0799ffb640 1 -- 172.21.15.190:0/1078959429 <== mon.5 v2:172.21.15.190:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f078c0125f0 con 0x7f079c0a5140 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.172+0000 7f0799ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.172+0000 7f0799ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:46.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.240+0000 7f07a39a2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "655b63d7-a358-4460-85ff-bf7b2330f230", "id": 4}] 2024-09-11T02:49:46.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.240+0000 7f07a39a2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "655b63d7-a358-4460-85ff-bf7b2330f230", "id": 4}] 2024-09-11T02:49:46.239 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.240+0000 7f07a39a2640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:49:46.239 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.240+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "osd new", "uuid": "655b63d7-a358-4460-85ff-bf7b2330f230", "id": 4} v 0) -- 0x7f079c06e390 con 0x7f079c0a5140 2024-09-11T02:49:46.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f0799ffb640 1 -- 172.21.15.190:0/1078959429 <== mon.5 v2:172.21.15.190:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "655b63d7-a358-4460-85ff-bf7b2330f230", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f078c0127d0 con 0x7f079c0a5140 2024-09-11T02:49:46.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f0799ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "655b63d7-a358-4460-85ff-bf7b2330f230", "id": 4}] 2024-09-11T02:49:46.247 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f0799ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:46.248 INFO:teuthology.orchestra.run.smithi190.stdout:4 2024-09-11T02:49:46.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 10 librados: watch_flush enter 2024-09-11T02:49:46.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 10 librados: watch_flush exit 2024-09-11T02:49:46.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 20 client.4127.objecter shutdown clearing up homeless session... 2024-09-11T02:49:46.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 10 client.4127.objecter successfully canceled tick 2024-09-11T02:49:46.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 10 monclient: shutdown 2024-09-11T02:49:46.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:46.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f079c0a5140 msgr2=0x7f079c13a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f079c0a5140 0x7f079c13a7a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f078c00d560 tx=0x7f078c00da20 comp rx=0 tx=0).stop 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 shutdown_connections 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f079c0a5140 0x7f079c13a7a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f079c0a4820 0x7f079c1d1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 --2- 172.21.15.190:0/1078959429 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f079c09fcf0 0x7f079c1d1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 >> 172.21.15.190:0/1078959429 conn(0x7f079c0a9410 msgr2=0x7f079c0a8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 shutdown_connections 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 -- 172.21.15.190:0/1078959429 wait complete. 2024-09-11T02:49:46.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.248+0000 7f07a39a2640 1 librados: shutdown 2024-09-11T02:49:46.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo ceph --cluster ceph osd new 3d6351e6-ca58-4eb7-9b62-f52c423c19cb 5 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.376+0000 7fea45de6640 10 monclient: get_monmap_and_config 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.376+0000 7fea45de6640 10 monclient: build_initial_monmap 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.376+0000 7fea45de6640 10 monclient: monmap: 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:46.382791+0000 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:46.382791+0000 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:46.377 INFO:teuthology.orchestra.run.smithi190.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:46.378 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 1 Processor -- start 2024-09-11T02:49:46.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 1 -- start start 2024-09-11T02:49:46.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient: init 2024-09-11T02:49:46.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient: _add_conns ranks=[10,1,8,7,0,5,4,9,3,2,6] 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fea400a47b0 0x7fea400a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient(hunting): picked mon.noname-f con 0x7fea400a47b0 addr v2:172.21.15.190:3300/0 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fea400a50d0 addr v1:172.21.15.190:6789/0 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea40137c80 0x7fea4013f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient(hunting): picked mon.noname-e con 0x7fea40137c80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea400a2e00 con 0x7fea400a50d0 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fea4009cbf0 con 0x7fea400a47b0 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fea400a3010 con 0x7fea40137c80 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:46.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea45de6640 10 monclient(hunting): authenticate will time out at 947.049250s 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3effd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fea400a50d0 0x7fea40137570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.190:47324/0 (socket says 172.21.15.190:47324) 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3effd640 1 -- 172.21.15.190:0/635058566 learned_addr learned my addr 172.21.15.190:0/635058566 (peer_addr_for_me v1:172.21.15.190:0/0) 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 1 --2- 172.21.15.190:0/635058566 >> v2:172.21.15.190:3300/0 conn(0x7fea400a47b0 0x7fea400a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): get_auth_request con 0x7fea400a47b0 auth_method 0 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3ffff640 1 --2- 172.21.15.190:0/635058566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea40137c80 0x7fea4013f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3ffff640 10 monclient(hunting): get_auth_request con 0x7fea40137c80 auth_method 0 2024-09-11T02:49:46.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3e7fc640 1 -- 172.21.15.190:0/635058566 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2055311404 0 0) 0x7fea34001120 con 0x7fea400a50d0 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3e7fc640 10 monclient(hunting): _negotiate 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3e7fc640 10 monclient(hunting): my global_id is 4150 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3e7fc640 1 -- 172.21.15.190:0/635058566 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea28003530 con 0x7fea400a50d0 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_done global_id 4141 payload 274 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 1 -- 172.21.15.190:0/635058566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea40137c80 msgr2=0x7fea4013f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 1 --2- 172.21.15.190:0/635058566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea40137c80 0x7fea4013f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 1 -- 172.21.15.190:0/635058566 >> v1:172.21.15.190:6789/0 conn(0x7fea400a50d0 legacy=0x7fea40137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 1 monclient: found mon.noname-f 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.380+0000 7fea3f7fe640 1 -- 172.21.15.190:0/635058566 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4006d090 con 0x7fea400a47b0 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 --2- 172.21.15.190:0/635058566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea400a47b0 0x7fea400a4b90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fea300097f0 tx=0x7fea3001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8019c3f8b6bd1e46 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 1 -- 172.21.15.190:0/635058566 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea300b0070 con 0x7fea400a47b0 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: dump: 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.385 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 1 -- 172.21.15.190:0/635058566 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea300aa810 con 0x7fea400a47b0 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 5 monclient: authenticate success, global_id 4141 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 4 set_mon_vals no callback set 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 1 -- 172.21.15.190:0/635058566 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea300aae30 con 0x7fea400a47b0 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3e7fc640 10 monclient: dump: 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.387 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.388 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.389 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.389 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: shutdown 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- 172.21.15.190:0/635058566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea400a47b0 msgr2=0x7fea400a4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 --2- 172.21.15.190:0/635058566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea400a47b0 0x7fea400a4b90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fea300097f0 tx=0x7fea3001c360 comp rx=0 tx=0).stop 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- 172.21.15.190:0/635058566 shutdown_connections 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 --2- 172.21.15.190:0/635058566 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fea40137c80 0x7fea4013f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 --2- 172.21.15.190:0/635058566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea400a47b0 0x7fea400a4b90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- 172.21.15.190:0/635058566 >> 172.21.15.190:0/635058566 conn(0x7fea400a93a0 msgr2=0x7fea400a58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- 172.21.15.190:0/635058566 shutdown_connections 2024-09-11T02:49:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- 172.21.15.190:0/635058566 wait complete. 2024-09-11T02:49:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: build_initial_monmap 2024-09-11T02:49:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: monmap: 2024-09-11T02:49:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 0 2024-09-11T02:49:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 0.000000 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:created 0.000000 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 1 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:46.395 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 librados: starting msgr at 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 librados: starting objecter 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 Processor -- start 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- start start 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 librados: setting wanted keys 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 librados: calling monclient init 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: init 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: _add_conns ranks=[7,6,1,3,2,8,4,0,5] 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fea400a47b0 0x7fea401cf650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): picked mon.noname-i con 0x7fea400a47b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea400a50d0 0x7fea401cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): picked mon.noname-d con 0x7fea400a50d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea40137c80 0x7fea401d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): picked mon.noname-f con 0x7fea40137c80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea400a2b70 con 0x7fea40137c80 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fea400a29f0 con 0x7fea400a50d0 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fea40142740 con 0x7fea400a47b0 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient(hunting): authenticate will time out at 947.054415s 2024-09-11T02:49:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea40137c80 0x7fea401d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fea400a47b0 0x7fea401cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea40137c80 0x7fea401d00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:59956/0 (socket says 172.21.15.190:59956) 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 1 -- 172.21.15.190:0/3858322159 learned_addr learned my addr 172.21.15.190:0/3858322159 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 10 monclient(hunting): get_auth_request con 0x7fea40137c80 auth_method 0 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): get_auth_request con 0x7fea400a47b0 auth_method 0 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3effd640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea400a50d0 0x7fea401cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 -- 172.21.15.190:0/3858322159 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea400a50d0 msgr2=0x7fea401cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea400a50d0 0x7fea401cfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 -- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea40137c80 msgr2=0x7fea401d00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea40137c80 0x7fea401d00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 monclient: found mon.noname-i 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:49:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 -- 172.21.15.190:0/3858322159 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4013bfa0 con 0x7fea400a47b0 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3effd640 10 monclient: get_auth_request con 0x7fea400a50d0 auth_method 0 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3ffff640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea40137c80 0x7fea401d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3effd640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea400a50d0 0x7fea401cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3f7fe640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fea400a47b0 0x7fea401cf650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea400a9e20 tx=0x7fea30019100 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 client.?.objecter ms_handle_connect 0x7fea400a47b0 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 1 -- 172.21.15.190:0/3858322159 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea300b0040 con 0x7fea400a47b0 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: dump: 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.400 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.401 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: _finish_auth 0 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: _check_auth_tickets 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 1 -- 172.21.15.190:0/3858322159 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea300044b0 con 0x7fea400a47b0 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 5 monclient: authenticate success, global_id 4129 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: _renew_subs 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4013f7e0 con 0x7fea400a47b0 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 1 -- 172.21.15.190:0/3858322159 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea300aaca0 con 0x7fea400a47b0 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:49:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea3dffb640 4 set_mon_vals no callback set 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 monclient: dump: 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:epoch 1 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:election_strategy: 3 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: _renew_subs 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fea400a8300 con 0x7fea400a47b0 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea45de6640 1 librados: init done 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 1 -- 172.21.15.190:0/3858322159 <== mon.7 v2:172.21.15.190:3303/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fea300b3070 con 0x7fea400a47b0 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 1 -- 172.21.15.190:0/3858322159 <== mon.7 v2:172.21.15.190:3303/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fea300b9020 con 0x7fea400a47b0 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 10 client.4129.objecter ms_dispatch 0x7fea40000ba0 osd_map(7..7 src has 1..7) 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 3 client.4129.objecter handle_osd_map got epochs [7,7] > 0 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 3 client.4129.objecter handle_osd_map decoding full epoch 7 2024-09-11T02:49:46.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.384+0000 7fea27fff640 20 client.4129.objecter dump_active .. 0 homeless 2024-09-11T02:49:46.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.388+0000 7fea45de6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.388+0000 7fea45de6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.388+0000 7fea45de6640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:49:46.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.388+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea400a4b90 con 0x7fea400a47b0 2024-09-11T02:49:46.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.388+0000 7fea27fff640 1 -- 172.21.15.190:0/3858322159 <== mon.7 v2:172.21.15.190:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fea30016070 con 0x7fea400a47b0 2024-09-11T02:49:46.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.388+0000 7fea27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.388+0000 7fea27fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:46.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.460+0000 7fea45de6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3d6351e6-ca58-4eb7-9b62-f52c423c19cb", "id": 5}] 2024-09-11T02:49:46.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.460+0000 7fea45de6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3d6351e6-ca58-4eb7-9b62-f52c423c19cb", "id": 5}] 2024-09-11T02:49:46.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.460+0000 7fea45de6640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:49:46.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.460+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "osd new", "uuid": "3d6351e6-ca58-4eb7-9b62-f52c423c19cb", "id": 5} v 0) -- 0x7fea4006e390 con 0x7fea400a47b0 2024-09-11T02:49:46.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea27fff640 1 -- 172.21.15.190:0/3858322159 <== mon.7 v2:172.21.15.190:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3d6351e6-ca58-4eb7-9b62-f52c423c19cb", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fea30003cb0 con 0x7fea400a47b0 2024-09-11T02:49:46.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3d6351e6-ca58-4eb7-9b62-f52c423c19cb", "id": 5}] 2024-09-11T02:49:46.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea27fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:46.467 INFO:teuthology.orchestra.run.smithi190.stdout:5 2024-09-11T02:49:46.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 10 librados: watch_flush enter 2024-09-11T02:49:46.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 10 librados: watch_flush exit 2024-09-11T02:49:46.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 20 client.4129.objecter shutdown clearing up homeless session... 2024-09-11T02:49:46.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 10 client.4129.objecter successfully canceled tick 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 10 monclient: shutdown 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fea400a47b0 msgr2=0x7fea401cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fea400a47b0 0x7fea401cf650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea400a9e20 tx=0x7fea30019100 comp rx=0 tx=0).stop 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 shutdown_connections 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea40137c80 0x7fea401d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fea400a50d0 0x7fea401cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 --2- 172.21.15.190:0/3858322159 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fea400a47b0 0x7fea401cf650 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 >> 172.21.15.190:0/3858322159 conn(0x7fea400a93a0 msgr2=0x7fea400a7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 shutdown_connections 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 -- 172.21.15.190:0/3858322159 wait complete. 2024-09-11T02:49:46.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-09-11T02:49:46.468+0000 7fea45de6640 1 librados: shutdown 2024-09-11T02:49:46.483 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T02:49:46.483 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T02:49:46.486 INFO:tasks.ceph.osd.0:Started 2024-09-11T02:49:46.486 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T02:49:46.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T02:49:46.488 INFO:tasks.ceph.osd.1:Started 2024-09-11T02:49:46.488 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T02:49:46.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T02:49:46.491 INFO:tasks.ceph.osd.2:Started 2024-09-11T02:49:46.491 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T02:49:46.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T02:49:46.493 INFO:tasks.ceph.osd.3:Started 2024-09-11T02:49:46.493 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T02:49:46.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T02:49:46.495 INFO:tasks.ceph.osd.4:Started 2024-09-11T02:49:46.495 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-11T02:49:46.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-11T02:49:46.497 INFO:tasks.ceph.osd.5:Started 2024-09-11T02:49:46.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient: get_monmap_and_config 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient: build_initial_monmap 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient: monmap: 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:46.619746+0000 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:46.619746+0000 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:46.616 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:46.617 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 Processor -- start 2024-09-11T02:49:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 -- start start 2024-09-11T02:49:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient: init 2024-09-11T02:49:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient: _add_conns ranks=[3,2,4,9,6,1,10,0,5,8,7] 2024-09-11T02:49:46.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb640a4710 0x7fdb640a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): picked mon.noname-g con 0x7fdb640a4710 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb640a5030 0x7fdb641364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): picked mon.noname-b con 0x7fdb640a5030 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb64136a00 0x7fdb64136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): picked mon.noname-c con 0x7fdb64136a00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdb640a0cd0 con 0x7fdb640a5030 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdb640a2610 con 0x7fdb640a4710 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdb640a1d70 con 0x7fdb64136a00 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:46.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6bf71640 10 monclient(hunting): authenticate will time out at 940.499671s 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6950c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb640a5030 0x7fdb641364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6a50e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb64136a00 0x7fdb64136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6950c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb640a5030 0x7fdb641364c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54952/0 (socket says 172.21.15.98:54952) 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6950c640 1 -- 172.21.15.98:0/1485940948 learned_addr learned my addr 172.21.15.98:0/1485940948 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6950c640 10 monclient(hunting): get_auth_request con 0x7fdb640a5030 auth_method 0 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6950c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6950c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6950c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6a50e640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb64136a00 0x7fdb64136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42610/0 (socket says 172.21.15.98:42610) 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6a50e640 10 monclient(hunting): get_auth_request con 0x7fdb64136a00 auth_method 0 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6a50e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6a50e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb6a50e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.615+0000 7fdb69d0d640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb640a4710 0x7fdb640a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb69d0d640 10 monclient(hunting): get_auth_request con 0x7fdb640a4710 auth_method 0 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb69d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb69d0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb69d0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6950c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6950c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6950c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb69d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb69d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb69d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 10 monclient(hunting): handle_auth_done global_id 4153 payload 274 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 1 -- 172.21.15.98:0/1485940948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb640a4710 msgr2=0x7fdb640a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb640a4710 0x7fdb640a4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 1 -- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb640a5030 msgr2=0x7fdb641364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb640a5030 0x7fdb641364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 10 monclient: _finish_hunting 0 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 1 monclient: found mon.noname-c 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 1 -- 172.21.15.98:0/1485940948 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb640a2050 con 0x7fdb64136a00 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6a50e640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb64136a00 0x7fdb64136e60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0097f0 tx=0x7fdb5c01c430 comp rx=0 tx=0).ready entity=mon.4 client_cookie=fbca093c2b247566 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 1 -- 172.21.15.98:0/1485940948 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb5c0b0070 con 0x7fdb64136a00 2024-09-11T02:49:46.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: dump: 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.630 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: _finish_auth 0 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: _check_auth_tickets 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 1 -- 172.21.15.98:0/1485940948 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb5c0aa810 con 0x7fdb64136a00 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 5 monclient: authenticate success, global_id 4153 2024-09-11T02:49:46.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 1 -- 172.21.15.98:0/1485940948 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb5c0aaa40 con 0x7fdb64136a00 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 4 set_mon_vals no callback set 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb68d0b640 10 monclient: dump: 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.632 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.633 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 10 monclient: shutdown 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb64136a00 msgr2=0x7fdb64136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb64136a00 0x7fdb64136e60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c0097f0 tx=0x7fdb5c01c430 comp rx=0 tx=0).stop 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1485940948 shutdown_connections 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb64136a00 0x7fdb64136e60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdb640a5030 0x7fdb641364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1485940948 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb640a4710 0x7fdb640a4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1485940948 >> 172.21.15.98:0/1485940948 conn(0x7fdb6406ed00 msgr2=0x7fdb6406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1485940948 shutdown_connections 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1485940948 wait complete. 2024-09-11T02:49:46.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 10 monclient: build_initial_monmap 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 10 monclient: monmap: 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:46.635 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 librados: starting msgr at 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.619+0000 7fdb6bf71640 1 librados: starting objecter 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 Processor -- start 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 -- start start 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 librados: setting wanted keys 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 librados: calling monclient init 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: init 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _add_conns ranks=[8,4,3,7,1,6,2,5,0] 2024-09-11T02:49:46.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb640a4710 0x7fdb64072cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): picked mon.noname-e con 0x7fdb640a4710 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb640a5030 0x7fdb64132b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): picked mon.noname-c con 0x7fdb640a5030 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb64136a00 0x7fdb6413d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): picked mon.noname-g con 0x7fdb64136a00 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdb640a0410 con 0x7fdb64136a00 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fdb640a0290 con 0x7fdb640a5030 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdb640a2fe0 con 0x7fdb640a4710 2024-09-11T02:49:46.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient(hunting): authenticate will time out at 940.504699s 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb640a4710 0x7fdb64072cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb640a5030 0x7fdb64132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb640a4710 0x7fdb64072cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:46360/0 (socket says 172.21.15.98:46360) 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb640a5030 0x7fdb64132b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42626/0 (socket says 172.21.15.98:42626) 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 1 -- 172.21.15.98:0/1786859389 learned_addr learned my addr 172.21.15.98:0/1786859389 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 10 monclient(hunting): get_auth_request con 0x7fdb640a4710 auth_method 0 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): get_auth_request con 0x7fdb640a5030 auth_method 0 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:46.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb69d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6a50e640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb64136a00 0x7fdb6413d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient(hunting): handle_auth_done global_id 4162 payload 824 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 -- 172.21.15.98:0/1786859389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb640a4710 msgr2=0x7fdb64072cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb640a4710 0x7fdb64072cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 -- 172.21.15.98:0/1786859389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb64136a00 msgr2=0x7fdb6413d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb64136a00 0x7fdb6413d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient: _finish_hunting 0 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 monclient: found mon.noname-c 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 -- 172.21.15.98:0/1786859389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb6411f630 con 0x7fdb640a5030 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6a50e640 10 monclient: get_auth_request con 0x7fdb64136a00 auth_method 0 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6a50e640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb64136a00 0x7fdb6413d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6950c640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb640a5030 0x7fdb64132b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000efc0 tx=0x7fdb6000c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 client.?.objecter ms_handle_connect 0x7fdb640a5030 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 1 -- 172.21.15.98:0/1786859389 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb6000cba0 con 0x7fdb640a5030 2024-09-11T02:49:46.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: dump: 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 5 monclient: authenticate success, global_id 4162 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 1 -- 172.21.15.98:0/1786859389 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb60019070 con 0x7fdb640a5030 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:46.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _renew_subs 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 1 -- 172.21.15.98:0/1786859389 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb60014d60 con 0x7fdb640a5030 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb640a4af0 con 0x7fdb640a5030 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5bfff640 4 set_mon_vals no callback set 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 monclient: dump: 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:46.643 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _renew_subs 2024-09-11T02:49:46.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb6413e9c0 con 0x7fdb640a5030 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 librados: init done 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 1 -- 172.21.15.98:0/1786859389 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7fdb60014480 con 0x7fdb640a5030 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 1 -- 172.21.15.98:0/1786859389 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fdb6000cd40 con 0x7fdb640a5030 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 10 client.4162.objecter ms_dispatch 0x7fdb64000ba0 osd_map(8..8 src has 1..8) 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 3 client.4162.objecter handle_osd_map got epochs [8,8] > 0 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 3 client.4162.objecter handle_osd_map decoding full epoch 8 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb5a7fc640 20 client.4162.objecter dump_active .. 0 homeless 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:46.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.623+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb641364c0 con 0x7fdb640a5030 2024-09-11T02:49:46.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.627+0000 7fdb5a7fc640 1 -- 172.21.15.98:0/1786859389 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7fdb60022020 con 0x7fdb640a5030 2024-09-11T02:49:46.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.627+0000 7fdb5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:46.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.627+0000 7fdb5a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:46.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:46.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:46.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:49:46.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb6413c8d0 con 0x7fdb640a5030 2024-09-11T02:49:46.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb5a7fc640 1 -- 172.21.15.98:0/1786859389 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fdb60010040 con 0x7fdb640a5030 2024-09-11T02:49:46.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:46.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb5a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:46.687 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:49:46.687 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T02:49:46.462961+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T02:49:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 10 librados: watch_flush enter 2024-09-11T02:49:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 10 librados: watch_flush exit 2024-09-11T02:49:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 20 client.4162.objecter shutdown clearing up homeless session... 2024-09-11T02:49:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 10 client.4162.objecter successfully canceled tick 2024-09-11T02:49:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 10 monclient: shutdown 2024-09-11T02:49:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:46.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb640a5030 msgr2=0x7fdb64132b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb640a5030 0x7fdb64132b80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fdb6000efc0 tx=0x7fdb6000c490 comp rx=0 tx=0).stop 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 shutdown_connections 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb64136a00 0x7fdb6413d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fdb640a5030 0x7fdb64132b80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 --2- 172.21.15.98:0/1786859389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdb640a4710 0x7fdb64072cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 >> 172.21.15.98:0/1786859389 conn(0x7fdb6406ed00 msgr2=0x7fdb6409ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 shutdown_connections 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 -- 172.21.15.98:0/1786859389 wait complete. 2024-09-11T02:49:46.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:49:46.687+0000 7fdb6bf71640 1 librados: shutdown 2024-09-11T02:49:46.706 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T02:49:46.707 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T02:49:47.311 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T02:49:47.432 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.431+0000 7f95c03bf640 10 monclient: get_monmap_and_config 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.431+0000 7f95c03bf640 10 monclient: build_initial_monmap 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.431+0000 7f95c03bf640 10 monclient: monmap: 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:47.436285+0000 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:47.436285+0000 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:47.433 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:47.434 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:47.434 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:47.434 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:47.434 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:47.434 INFO:teuthology.misc.health.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:47.434 INFO:teuthology.misc.health.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:47.434 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:47.438 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.431+0000 7f95c03bf640 1 Processor -- start 2024-09-11T02:49:47.438 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.431+0000 7f95c03bf640 1 -- start start 2024-09-11T02:49:47.438 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.431+0000 7f95c03bf640 10 monclient: init 2024-09-11T02:49:47.438 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:47.438 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient: _add_conns ranks=[3,0,5,4,1,8,2,7,10,9,6] 2024-09-11T02:49:47.438 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 0x7f95b806cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.438 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient(hunting): picked mon.noname-g con 0x7f95b806c900 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95b8073820 addr v1:172.21.15.98:6789/0 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95b806d220 0x7f95b8137aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient(hunting): picked mon.noname-h con 0x7f95b806d220 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95b80a25e0 con 0x7f95b8073820 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f95b80a53c0 con 0x7f95b806c900 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f95b8074050 con 0x7f95b806d220 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 10 monclient(hunting): authenticate will time out at 941.316700s 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd95a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f95b8073820 0x7f95b8137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58960/0 (socket says 172.21.15.98:58960) 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd95a640 1 -- 172.21.15.98:0/2962749526 learned_addr learned my addr 172.21.15.98:0/2962749526 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 0x7f95b806cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:47.439 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): get_auth_request con 0x7f95b806c900 auth_method 0 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95b806d220 0x7f95b8137aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 10 monclient(hunting): get_auth_request con 0x7f95b806d220 auth_method 0 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:47.440 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606706950 0 0) 0x7f95b80a25e0 con 0x7f95b8073820 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): _negotiate 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): my global_id is 4185 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95a0003530 con 0x7f95b8073820 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be95c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3700748532 0 0) 0x7f95a0003530 con 0x7f95b8073820 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95b806d220 msgr2=0x7f95b8137aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:47.441 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95b806d220 0x7f95b8137aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 msgr2=0x7f95b806cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 0x7f95b806cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: _finish_hunting 0 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b80a5140 con 0x7f95b8073820 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: _finish_auth 0 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: _check_auth_tickets 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f95ac001970 con 0x7f95b8073820 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 0x7f95b806cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 5 monclient: authenticate success, global_id 4185 2024-09-11T02:49:47.442 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: dump: 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:47.443 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 >> v1:172.21.15.98:6789/0 conn(0x7f95b8073820 legacy=0x7f95b8137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient: _add_conns ranks=[2,7,3,8,4,6,0,1,5] 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a0004210 0x7f95a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): picked mon.b con 0x7f95a0004210 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95a0004da0 0x7f95a0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.444 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): picked mon.i con 0x7f95a0004da0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 0x7f95a000bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): picked mon.g con 0x7f95b806c900 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f95a000c1f0 con 0x7f95a0004210 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f95a000c370 con 0x7f95b806c900 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f95a000c520 con 0x7f95a0004da0 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd159640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95c03bf640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a0004210 0x7f95a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:47.445 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): get_auth_request con 0x7f95a0004210 auth_method 0 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd95a640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95a0004da0 0x7f95a0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd95a640 10 monclient(hunting): get_auth_request con 0x7f95a0004da0 auth_method 0 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd95a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd95a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95bd95a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient(hunting): handle_auth_done global_id 4185 payload 293 2024-09-11T02:49:47.446 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 -- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95a0004da0 msgr2=0x7f95a0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95a0004da0 0x7f95a0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 -- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 msgr2=0x7f95a000bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f95b806c900 0x7f95a000bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient: _finish_hunting 0 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 monclient: found mon.b 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 1 -- 172.21.15.98:0/2962749526 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a0003530 con 0x7f95a0004210 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient: _finish_auth 0 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 10 monclient: _check_auth_tickets 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.435+0000 7f95be15b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95be15b640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a0004210 0x7f95a0004810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e320 tx=0x7f95a801cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b49b00ac86293dcf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95a80b2070 con 0x7f95a0004210 2024-09-11T02:49:47.447 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 10 monclient: dump: 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:47.448 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a80a5e20 con 0x7f95a0004210 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 1 -- 172.21.15.98:0/2962749526 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95a8009de0 con 0x7f95a0004210 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd159640 10 monclient: dump: 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:47.449 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:47.450 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:47.451 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 4 set_mon_vals no callback set 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient: shutdown 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- 172.21.15.98:0/2962749526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a0004210 msgr2=0x7f95a0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 --2- 172.21.15.98:0/2962749526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f95a0004210 0x7f95a0004810 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e320 tx=0x7f95a801cdc0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95be95c640 1 -- 172.21.15.98:0/2962749526 reap_dead start 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- 172.21.15.98:0/2962749526 shutdown_connections 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- 172.21.15.98:0/2962749526 >> 172.21.15.98:0/2962749526 conn(0x7f95b80a95b0 msgr2=0x7f95b80a56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- 172.21.15.98:0/2962749526 shutdown_connections 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- 172.21.15.98:0/2962749526 wait complete. 2024-09-11T02:49:47.455 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient: build_initial_monmap 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient: monmap: 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:47.456 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 librados: starting msgr at 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 librados: starting objecter 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 Processor -- start 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- start start 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 librados: setting wanted keys 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 librados: calling monclient init 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient: init 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:47.457 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient: _add_conns ranks=[5,6,7,1,3,2,4,8,0] 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95a0004210 0x7f95b8132e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): picked mon.noname-h con 0x7f95a0004210 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95b81333d0 0x7f95b809ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): picked mon.noname-d con 0x7f95b81333d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95b81337b0 0x7f95b809f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): picked mon.noname-i con 0x7f95b81337b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f95b8137fe0 con 0x7f95a0004210 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f95b8050f90 con 0x7f95b81333d0 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:47.458 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f95b8138160 con 0x7f95b81337b0 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95c03bf640 10 monclient(hunting): authenticate will time out at 941.323836s 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.439+0000 7f95bd95a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95b81333d0 0x7f95b809ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95b81333d0 0x7f95b809ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:42396/0 (socket says 172.21.15.98:42396) 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 -- 172.21.15.98:0/1380359626 learned_addr learned my addr 172.21.15.98:0/1380359626 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): get_auth_request con 0x7f95b81333d0 auth_method 0 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95a0004210 0x7f95b8132e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 10 monclient(hunting): get_auth_request con 0x7f95a0004210 auth_method 0 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:47.459 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95be15b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient(hunting): handle_auth_done global_id 4173 payload 824 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 -- 172.21.15.98:0/1380359626 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95b81337b0 msgr2=0x7f95b809f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95b81337b0 0x7f95b809f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 -- 172.21.15.98:0/1380359626 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95a0004210 msgr2=0x7f95b8132e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95a0004210 0x7f95b8132e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient: _finish_hunting 0 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 monclient: found mon.noname-d 2024-09-11T02:49:47.460 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 -- 172.21.15.98:0/1380359626 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b811fbe0 con 0x7f95b81333d0 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bd95a640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95b81333d0 0x7f95b809ed50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00e7b0 tx=0x7f95ac00ec70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 client.?.objecter ms_handle_connect 0x7f95b81333d0 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 1 -- 172.21.15.98:0/1380359626 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95ac00ee60 con 0x7f95b81333d0 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: dump: 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.461 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: _finish_auth 0 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:47.462 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 1 -- 172.21.15.98:0/1380359626 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ac018070 con 0x7f95b81333d0 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 1 -- 172.21.15.98:0/1380359626 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95ac00ccc0 con 0x7f95b81333d0 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 monclient: dump: 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:47.463 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95bc958640 4 set_mon_vals no callback set 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 5 monclient: authenticate success, global_id 4173 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 monclient: _renew_subs 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b80a7ad0 con 0x7f95b81333d0 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:47.464 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 monclient: _renew_subs 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b8134040 con 0x7f95b81333d0 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 1 librados: init done 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 1 -- 172.21.15.98:0/1380359626 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 1) ==== 811+0+0 (secure 0 0 0) 0x7f95ac00c850 con 0x7f95b81333d0 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 1 -- 172.21.15.98:0/1380359626 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f95ac010040 con 0x7f95b81333d0 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 10 client.4173.objecter ms_dispatch 0x7f95b8000ba0 osd_map(8..8 src has 1..8) 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 3 client.4173.objecter handle_osd_map got epochs [8,8] > 0 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 3 client.4173.objecter handle_osd_map decoding full epoch 8 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95b6ffd640 20 client.4173.objecter dump_active .. 0 homeless 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.443+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b8137390 con 0x7f95b81333d0 2024-09-11T02:49:47.465 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.447+0000 7f95b6ffd640 1 -- 172.21.15.98:0/1380359626 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+76091 (secure 0 0 0) 0x7f95ac01d460 con 0x7f95b81333d0 2024-09-11T02:49:47.466 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.447+0000 7f95b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:47.466 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.447+0000 7f95b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:47.506 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:47.506 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:47.507 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:49:47.507 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95b811f960 con 0x7f95b81333d0 2024-09-11T02:49:47.507 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95b6ffd640 1 -- 172.21.15.98:0/1380359626 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f95ac01d640 con 0x7f95b81333d0 2024-09-11T02:49:47.507 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:47.507 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:47.507 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-11T02:49:47.508 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":8,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T02:49:46.462961+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 10 librados: watch_flush enter 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 10 librados: watch_flush exit 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 20 client.4173.objecter shutdown clearing up homeless session... 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 10 client.4173.objecter successfully canceled tick 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 10 monclient: shutdown 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95b81333d0 msgr2=0x7f95b809ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95b81333d0 0x7f95b809ed50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00e7b0 tx=0x7f95ac00ec70 comp rx=0 tx=0).stop 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 shutdown_connections 2024-09-11T02:49:47.509 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f95b81337b0 0x7f95b809f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.510 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f95b81333d0 0x7f95b809ed50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.510 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 --2- 172.21.15.98:0/1380359626 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f95a0004210 0x7f95b8132e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:47.510 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 >> 172.21.15.98:0/1380359626 conn(0x7f95b80a95b0 msgr2=0x7f95b80a72e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:47.510 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 shutdown_connections 2024-09-11T02:49:47.510 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 -- 172.21.15.98:0/1380359626 wait complete. 2024-09-11T02:49:47.510 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:47.507+0000 7f95c03bf640 1 librados: shutdown 2024-09-11T02:49:47.524 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-11T02:49:47.643 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-11T02:49:47.643+0000 7f2be20b87c0 -1 Falling back to public interface 2024-09-11T02:49:47.654 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-11T02:49:47.655+0000 7f4c1d16b7c0 -1 Falling back to public interface 2024-09-11T02:49:47.658 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-11T02:49:47.659+0000 7f1fcd14c7c0 -1 Falling back to public interface 2024-09-11T02:49:47.707 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T02:49:47.708+0000 7efdf6b147c0 -1 Falling back to public interface 2024-09-11T02:49:47.826 INFO:tasks.ceph.mgr.x.smithi190.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T02:49:47.826 INFO:tasks.ceph.mgr.x.smithi190.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T02:49:47.826 INFO:tasks.ceph.mgr.x.smithi190.stderr: from numpy import show_config as show_numpy_config 2024-09-11T02:49:47.875 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T02:49:47.876+0000 7f8b8c4c37c0 -1 Falling back to public interface 2024-09-11T02:49:48.162 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T02:49:48.164+0000 7fbf85e6e7c0 -1 Falling back to public interface 2024-09-11T02:49:50.743 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.744+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-09-11T02:49:50.744 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.744+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T02:49:50.746 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.748+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-11T02:49:50.747 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.748+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-09-11T02:49:50.747 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.748+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-09-11T02:49:50.747 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.748+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-09-11T02:49:50.747 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.748+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-09-11T02:49:50.747 INFO:tasks.ceph.mgr.x.smithi190.stderr:2024-09-11T02:49:50.748+0000 7f8c939b8640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-09-11T02:49:51.630 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-11T02:49:51.631+0000 7f2be20b87c0 -1 osd.0 0 log_to_monitors true 2024-09-11T02:49:51.723 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-11T02:49:51.723+0000 7f4c1d16b7c0 -1 osd.1 0 log_to_monitors true 2024-09-11T02:49:51.809 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-11T02:49:51.807+0000 7f1fcd14c7c0 -1 osd.2 0 log_to_monitors true 2024-09-11T02:49:51.919 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T02:49:51.920+0000 7f8b8c4c37c0 -1 osd.5 0 log_to_monitors true 2024-09-11T02:49:52.021 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T02:49:52.024+0000 7efdf6b147c0 -1 osd.3 0 log_to_monitors true 2024-09-11T02:49:52.345 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T02:49:52.348+0000 7fbf85e6e7c0 -1 osd.4 0 log_to_monitors true 2024-09-11T02:49:52.804 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-11T02:49:52.803+0000 7f2bde061640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T02:49:52.823 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-11T02:49:52.823+0000 7f2bd9e9b640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T02:49:53.601 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-11T02:49:53.599+0000 7f4c19926640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T02:49:53.626 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-11T02:49:53.627+0000 7f4c10f46640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T02:49:53.807 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T02:49:53.808+0000 7fbf81e17640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T02:49:53.807 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T02:49:53.808+0000 7f8b8846c640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T02:49:53.809 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T02:49:53.812+0000 7efdf2abd640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T02:49:53.829 INFO:tasks.ceph.osd.4.smithi190.stderr:2024-09-11T02:49:53.832+0000 7fbf7dc51640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T02:49:53.849 INFO:tasks.ceph.osd.5.smithi190.stderr:2024-09-11T02:49:53.852+0000 7f8b842a6640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T02:49:53.851 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T02:49:53.852+0000 7efdee8f7640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T02:49:54.129 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T02:49:54.252 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.251+0000 7f15ee6a1640 10 monclient: get_monmap_and_config 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.251+0000 7f15ee6a1640 10 monclient: build_initial_monmap 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.251+0000 7f15ee6a1640 10 monclient: monmap: 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:54.256447+0000 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:54.256447+0000 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:54.253 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:49:54.254 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:54.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 1 Processor -- start 2024-09-11T02:49:54.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 1 -- start start 2024-09-11T02:49:54.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient: init 2024-09-11T02:49:54.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:54.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient: _add_conns ranks=[8,3,0,10,2,7,9,6,1,5,4] 2024-09-11T02:49:54.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f15e80a45d0 0x7f15e80a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient(hunting): picked mon.noname-e con 0x7f15e80a45d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e8073310 0x7f15e80a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient(hunting): picked mon.noname-g con 0x7f15e8073310 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15e806cf60 addr v1:172.21.15.98:6789/0 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e80a5430 con 0x7f15e806cf60 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f15e80744c0 con 0x7f15e8073310 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f15e8073f00 con 0x7f15e80a45d0 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient(hunting): authenticate will time out at 948.136775s 2024-09-11T02:49:54.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ecc3e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f15e806cf60 0x7f15e8137390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:58966/0 (socket says 172.21.15.98:58966) 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f15e80a45d0 0x7f15e80a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ecc3e640 1 -- 172.21.15.98:0/1544232350 learned_addr learned my addr 172.21.15.98:0/1544232350 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): get_auth_request con 0x7f15e80a45d0 auth_method 0 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e77fe640 1 --2- 172.21.15.98:0/1544232350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e8073310 0x7f15e80a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e77fe640 10 monclient(hunting): get_auth_request con 0x7f15e8073310 auth_method 0 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:54.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 1 -- 172.21.15.98:0/1544232350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e8073310 msgr2=0x7f15e80a4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 1 --2- 172.21.15.98:0/1544232350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e8073310 0x7f15e80a4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 1 -- 172.21.15.98:0/1544232350 >> v1:172.21.15.98:6789/0 conn(0x7f15e806cf60 legacy=0x7f15e8137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient: _finish_hunting 0 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 1 monclient: found mon.noname-e 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:49:54.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 1 -- 172.21.15.98:0/1544232350 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e80a6500 con 0x7f15e80a45d0 2024-09-11T02:49:54.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e7fff640 1 --2- 172.21.15.98:0/1544232350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f15e80a45d0 0x7f15e80a49b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15d80097f0 tx=0x7f15d801c3b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5ad818ec57399da5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:54.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 1 -- 172.21.15.98:0/1544232350 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f15d80b0070 con 0x7f15e80a45d0 2024-09-11T02:49:54.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:54.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: dump: 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:54.266 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: _finish_auth 0 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 5 monclient: authenticate success, global_id 4193 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 1 -- 172.21.15.98:0/1544232350 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15d80aa810 con 0x7f15e80a45d0 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:54.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 10 monclient: get_monmap_and_config success 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15ee6a1640 4 set_mon_vals no callback set 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 1 -- 172.21.15.98:0/1544232350 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f15d80a9300 con 0x7f15e80a45d0 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.255+0000 7f15e6ffd640 10 monclient: dump: 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:54.268 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: shutdown 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- 172.21.15.98:0/1544232350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f15e80a45d0 msgr2=0x7f15e80a49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/1544232350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f15e80a45d0 0x7f15e80a49b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15d80097f0 tx=0x7f15d801c3b0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- 172.21.15.98:0/1544232350 shutdown_connections 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/1544232350 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f15e8073310 0x7f15e80a4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/1544232350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f15e80a45d0 0x7f15e80a49b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- 172.21.15.98:0/1544232350 >> 172.21.15.98:0/1544232350 conn(0x7f15e80a56c0 msgr2=0x7f15e80a5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- 172.21.15.98:0/1544232350 shutdown_connections 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- 172.21.15.98:0/1544232350 wait complete. 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: build_initial_monmap 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: monmap: 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:49:54.270 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 librados: starting msgr at 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 librados: starting objecter 2024-09-11T02:49:54.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 Processor -- start 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- start start 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 librados: setting wanted keys 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 librados: calling monclient init 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: init 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: _reopen_session rank -1 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: _add_conns ranks=[1,0,7,3,6,5,4,2,8] 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e806cf60 0x7f15e81d16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): picked mon.noname-f con 0x7f15e806cf60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e80a45d0 0x7f15e8136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): picked mon.noname-a con 0x7f15e80a45d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f15e8136c90 0x7f15e813c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): picked mon.noname-i con 0x7f15e8136c90 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:49:54.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15e81423b0 con 0x7f15e80a45d0 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f15e8142230 con 0x7f15e806cf60 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f15e8142530 con 0x7f15e8136c90 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): _renew_subs 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient(hunting): authenticate will time out at 948.142712s 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e80a45d0 0x7f15e8136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e80a45d0 0x7f15e8136750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47524/0 (socket says 172.21.15.98:47524) 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 -- 172.21.15.98:0/67996820 learned_addr learned my addr 172.21.15.98:0/67996820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): get_auth_request con 0x7f15e80a45d0 auth_method 0 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:54.273 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e7fff640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e806cf60 0x7f15e81d16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e7fff640 10 monclient(hunting): get_auth_request con 0x7f15e806cf60 auth_method 0 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ecc3e640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f15e8136c90 0x7f15e813c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-09-11T02:49:54.274 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 -- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f15e8136c90 msgr2=0x7f15e813c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f15e8136c90 0x7f15e813c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 -- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e806cf60 msgr2=0x7f15e81d16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e806cf60 0x7f15e81d16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient: _finish_hunting 0 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 monclient: found mon.noname-a 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 -- 172.21.15.98:0/67996820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e8072ad0 con 0x7f15e80a45d0 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ecc3e640 10 monclient: get_auth_request con 0x7f15e8136c90 auth_method 0 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ecc3e640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f15e8136c90 0x7f15e813c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e77fe640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e80a45d0 0x7f15e8136750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00d830 tx=0x7f15dc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 client.?.objecter ms_handle_connect 0x7f15e80a45d0 2024-09-11T02:49:54.275 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 1 -- 172.21.15.98:0/67996820 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f15dc00bb30 con 0x7f15e80a45d0 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: dump: 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:54.276 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: _finish_auth 0 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 5 monclient: authenticate success, global_id 4230 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 1 -- 172.21.15.98:0/67996820 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15dc019070 con 0x7f15e80a45d0 2024-09-11T02:49:54.277 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: _renew_subs 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e813ea20 con 0x7f15e80a45d0 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 1 -- 172.21.15.98:0/67996820 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f15dc00bcd0 con 0x7f15e80a45d0 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e67fc640 4 set_mon_vals no callback set 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 client.4230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15e4ff9640 10 monclient: dump: 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:49:54.278 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:49:54.279 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: _renew_subs 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e80a7ad0 con 0x7f15e80a45d0 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.259+0000 7f15ee6a1640 1 librados: init done 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15ee6a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15ee6a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15ee6a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e8137390 con 0x7f15e80a45d0 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 1 -- 172.21.15.98:0/67996820 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 48470+0+0 (secure 0 0 0) 0x7f15dc013070 con 0x7f15e80a45d0 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f15d403deb0 0x7f15d4040370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 1 -- 172.21.15.98:0/67996820 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2487+0+0 (secure 0 0 0) 0x7f15dc00f040 con 0x7f15e80a45d0 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 10 client.4230.objecter ms_dispatch 0x7f15e8000ba0 osd_map(11..11 src has 1..11) 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 3 client.4230.objecter handle_osd_map got epochs [11,11] > 0 2024-09-11T02:49:54.280 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 3 client.4230.objecter handle_osd_map decoding full epoch 11 2024-09-11T02:49:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 20 client.4230.objecter dump_active .. 0 homeless 2024-09-11T02:49:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e7fff640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f15d403deb0 0x7f15d4040370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:49:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e7fff640 10 monclient: get_auth_request con 0x7f15d403deb0 auth_method 0 2024-09-11T02:49:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e4ff9640 10 client.4230.objecter ms_handle_connect 0x7f15d403deb0 2024-09-11T02:49:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.263+0000 7f15e7fff640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f15d403deb0 0x7f15d4040370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f15d80ad000 tx=0x7f15d80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:49:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.267+0000 7f15e4ff9640 1 -- 172.21.15.98:0/67996820 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f15dc022020 con 0x7f15e80a45d0 2024-09-11T02:49:54.281 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.267+0000 7f15e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:49:54.282 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.267+0000 7f15e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:49:54.497 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.495+0000 7f15ee6a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:54.497 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.495+0000 7f15ee6a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:54.497 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.495+0000 7f15ee6a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:49:54.497 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.495+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15e806eff0 con 0x7f15e80a45d0 2024-09-11T02:49:54.497 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.495+0000 7f15e4ff9640 1 -- 172.21.15.98:0/67996820 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6577 (secure 0 0 0) 0x7f15dc00b8d0 con 0x7f15e80a45d0 2024-09-11T02:49:54.498 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.495+0000 7f15e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:49:54.498 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.495+0000 7f15e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:49:54.498 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-11T02:49:54.498 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T02:49:53.796086+0000","last_up_change":"2024-09-11T02:49:53.796086+0000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6801","nonce":1456395826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6803","nonce":1456395826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6807","nonce":1456395826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6805","nonce":1456395826}]},"public_addr":"172.21.15.98:6801/1456395826","cluster_addr":"172.21.15.98:6803/1456395826","heartbeat_back_addr":"172.21.15.98:6807/1456395826","heartbeat_front_addr":"172.21.15.98:6805/1456395826","state":["exists","up"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6809","nonce":2313933591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6811","nonce":2313933591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6815","nonce":2313933591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6813","nonce":2313933591}]},"public_addr":"172.21.15.98:6809/2313933591","cluster_addr":"172.21.15.98:6811/2313933591","heartbeat_back_addr":"172.21.15.98:6815/2313933591","heartbeat_front_addr":"172.21.15.98:6813/2313933591","state":["exists","up"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 10 librados: watch_flush enter 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 10 librados: watch_flush exit 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 20 client.4230.objecter shutdown clearing up homeless session... 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 10 client.4230.objecter successfully canceled tick 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f15d403deb0 msgr2=0x7f15d4040370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f15d403deb0 0x7f15d4040370 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f15d80ad000 tx=0x7f15d80a6000 comp rx=0 tx=0).stop 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 10 monclient: shutdown 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e80a45d0 msgr2=0x7f15e8136750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:49:54.501 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e80a45d0 0x7f15e8136750 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00d830 tx=0x7f15dc00dcf0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 shutdown_connections 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f15d403deb0 0x7f15d4040370 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f15e8136c90 0x7f15e813c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15e80a45d0 0x7f15e8136750 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 --2- 172.21.15.98:0/67996820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15e806cf60 0x7f15e81d16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 >> 172.21.15.98:0/67996820 conn(0x7f15e80a56c0 msgr2=0x7f15e809d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 shutdown_connections 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 -- 172.21.15.98:0/67996820 wait complete. 2024-09-11T02:49:54.502 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:49:54.499+0000 7f15ee6a1640 1 librados: shutdown 2024-09-11T02:49:54.516 DEBUG:teuthology.misc:2 of 6 OSDs are up 2024-09-11T02:49:54.659 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-11T02:49:54.659+0000 7f1fc90f5640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T02:49:54.694 INFO:tasks.ceph.osd.2.smithi098.stderr:2024-09-11T02:49:54.695+0000 7f1fc0f27640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T02:50:01.122 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.239+0000 7fb0f8ac3640 10 monclient: get_monmap_and_config 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.239+0000 7fb0f8ac3640 10 monclient: build_initial_monmap 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.239+0000 7fb0f8ac3640 10 monclient: monmap: 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:50:01.244794+0000 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:50:01.244794+0000 2024-09-11T02:50:01.241 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:01.242 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 Processor -- start 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 -- start start 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient: init 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient: _add_conns ranks=[6,10,7,9,2,8,5,3,0,1,4] 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0f40a6800 0x7fb0f40a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): picked mon.noname-d con 0x7fb0f40a6800 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:01.245 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb0f40a6be0 0x7fb0f40a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): picked mon.noname-f con 0x7fb0f40a6be0 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0f41395c0 0x7fb0f4139a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): picked mon.noname-i con 0x7fb0f41395c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb0f40a83d0 con 0x7fb0f40a6be0 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb0f40a8610 con 0x7fb0f40a6800 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb0f40a45a0 con 0x7fb0f41395c0 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f8ac3640 10 monclient(hunting): authenticate will time out at 955.125022s 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0f40a6800 0x7fb0f40a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.246 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0f40a6800 0x7fb0f40a56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57078/0 (socket says 172.21.15.98:57078) 2024-09-11T02:50:01.247 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 -- 172.21.15.98:0/2832527762 learned_addr learned my addr 172.21.15.98:0/2832527762 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:01.247 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): get_auth_request con 0x7fb0f40a6800 auth_method 0 2024-09-11T02:50:01.247 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.247 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.247 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.247 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:01.247 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f1d9b640 1 --2- 172.21.15.98:0/2832527762 >> v2:172.21.15.190:3300/0 conn(0x7fb0f40a6be0 0x7fb0f40a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f1d9b640 10 monclient(hunting): get_auth_request con 0x7fb0f40a6be0 auth_method 0 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f2d9d640 1 --2- 172.21.15.98:0/2832527762 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0f41395c0 0x7fb0f4139a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 -- 172.21.15.98:0/2832527762 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0f41395c0 msgr2=0x7fb0f4139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 --2- 172.21.15.98:0/2832527762 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0f41395c0 0x7fb0f4139a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.248 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 -- 172.21.15.98:0/2832527762 >> v2:172.21.15.190:3300/0 conn(0x7fb0f40a6be0 msgr2=0x7fb0f40a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 --2- 172.21.15.98:0/2832527762 >> v2:172.21.15.190:3300/0 conn(0x7fb0f40a6be0 0x7fb0f40a6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient: _finish_hunting 0 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 monclient: found mon.noname-d 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f259c640 1 -- 172.21.15.98:0/2832527762 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f40a4880 con 0x7fb0f40a6800 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f2d9d640 10 monclient: get_auth_request con 0x7fb0f41395c0 auth_method 0 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f1d9b640 1 --2- 172.21.15.98:0/2832527762 >> v2:172.21.15.190:3300/0 conn(0x7fb0f40a6be0 0x7fb0f40a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:50:01.249 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.243+0000 7fb0f2d9d640 1 --2- 172.21.15.98:0/2832527762 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0f41395c0 0x7fb0f4139a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 1 --2- 172.21.15.98:0/2832527762 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0f40a6800 0x7fb0f40a56b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e50 tx=0x7fb0e401c8f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8b6c39142e801656 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 1 -- 172.21.15.98:0/2832527762 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0e40090d0 con 0x7fb0f40a6800 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: dump: 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.256 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: _finish_auth 0 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:01.257 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 1 -- 172.21.15.98:0/2832527762 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0e40a5b60 con 0x7fb0f40a6800 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 1 -- 172.21.15.98:0/2832527762 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0e40a5d50 con 0x7fb0f40a6800 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f159a640 10 monclient: dump: 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.258 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 5 monclient: authenticate success, global_id 4236 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 4 set_mon_vals no callback set 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient: shutdown 2024-09-11T02:50:01.259 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/2832527762 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0f40a6800 msgr2=0x7fb0f40a56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/2832527762 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0f40a6800 0x7fb0f40a56b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009e50 tx=0x7fb0e401c8f0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/2832527762 shutdown_connections 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/2832527762 >> v2:172.21.15.190:3300/0 conn(0x7fb0f40a6be0 0x7fb0f40a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/2832527762 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0f41395c0 0x7fb0f4139a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/2832527762 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb0f40a6800 0x7fb0f40a56b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/2832527762 >> 172.21.15.98:0/2832527762 conn(0x7fb0f40abe00 msgr2=0x7fb0f40ac210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/2832527762 shutdown_connections 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/2832527762 wait complete. 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient: build_initial_monmap 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient: monmap: 2024-09-11T02:50:01.260 INFO:teuthology.misc.health.smithi098.stderr:epoch 0 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:created 0.000000 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:01.261 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 librados: starting msgr at 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 librados: starting objecter 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 Processor -- start 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- start start 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 librados: setting wanted keys 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 librados: calling monclient init 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient: init 2024-09-11T02:50:01.262 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient: _add_conns ranks=[0,3,4,6,2,1,5,8,7] 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f40a6800 0x7fb0f41347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): picked mon.noname-a con 0x7fb0f40a6800 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0f40a6be0 0x7fb0f4134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): picked mon.noname-g con 0x7fb0f40a6be0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0f41395c0 0x7fb0f409ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): picked mon.noname-c con 0x7fb0f41395c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f40a5d70 con 0x7fb0f40a6800 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f40a5bf0 con 0x7fb0f40a6be0 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb0f4143240 con 0x7fb0f41395c0 2024-09-11T02:50:01.263 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f8ac3640 10 monclient(hunting): authenticate will time out at 955.131353s 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0f41395c0 0x7fb0f409ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0f41395c0 0x7fb0f409ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55418/0 (socket says 172.21.15.98:55418) 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 -- 172.21.15.98:0/3551748728 learned_addr learned my addr 172.21.15.98:0/3551748728 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): get_auth_request con 0x7fb0f41395c0 auth_method 0 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f40a6800 0x7fb0f41347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 10 monclient(hunting): get_auth_request con 0x7fb0f40a6800 auth_method 0 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.264 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f1d9b640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0f40a6be0 0x7fb0f4134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f1d9b640 10 monclient(hunting): get_auth_request con 0x7fb0f40a6be0 auth_method 0 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2024-09-11T02:50:01.265 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 -- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0f40a6be0 msgr2=0x7fb0f4134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0f40a6be0 0x7fb0f4134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 -- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f40a6800 msgr2=0x7fb0f41347d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f40a6800 0x7fb0f41347d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 monclient: found mon.noname-c 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f2d9d640 1 -- 172.21.15.98:0/3551748728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f40a04c0 con 0x7fb0f41395c0 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.247+0000 7fb0f259c640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f40a6800 0x7fb0f41347d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f2d9d640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0f41395c0 0x7fb0f409ca10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c820 tx=0x7fb0ec00cce0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 client.?.objecter ms_handle_connect 0x7fb0f41395c0 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:01.266 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 1 -- 172.21.15.98:0/3551748728 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0ec015020 con 0x7fb0f41395c0 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: dump: 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.267 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 5 monclient: authenticate success, global_id 4207 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 1 -- 172.21.15.98:0/3551748728 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ec00ae00 con 0x7fb0f41395c0 2024-09-11T02:50:01.268 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 monclient: _renew_subs 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f40a29b0 con 0x7fb0f41395c0 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 1 -- 172.21.15.98:0/3551748728 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0ec004240 con 0x7fb0f41395c0 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f0d99640 4 set_mon_vals no callback set 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 monclient: dump: 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:epoch 1 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.269 INFO:teuthology.misc.health.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr: 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 monclient: _renew_subs 2024-09-11T02:50:01.270 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f409fc20 con 0x7fb0f41395c0 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 1 librados: init done 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f40a6fc0 con 0x7fb0f41395c0 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 1 -- 172.21.15.98:0/3551748728 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb0ec003620 con 0x7fb0f41395c0 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0d403df40 0x7fb0d4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 1 -- 172.21.15.98:0/3551748728 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7fb0ec04ee40 con 0x7fb0f41395c0 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 client.4207.objecter ms_dispatch 0x7fb0f4000ba0 osd_map(15..15 src has 1..15) 2024-09-11T02:50:01.271 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 3 client.4207.objecter handle_osd_map got epochs [15,15] > 0 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 3 client.4207.objecter handle_osd_map decoding full epoch 15 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 20 client.4207.objecter dump_active .. 0 homeless 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f259c640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0d403df40 0x7fb0d4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f259c640 10 monclient: get_auth_request con 0x7fb0d403df40 auth_method 0 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0e2ffd640 10 client.4207.objecter ms_handle_connect 0x7fb0d403df40 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.251+0000 7fb0f259c640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0d403df40 0x7fb0d4040400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40ae000 tx=0x7fb0e40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.255+0000 7fb0e2ffd640 1 -- 172.21.15.98:0/3551748728 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fb0ec01f030 con 0x7fb0f41395c0 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.255+0000 7fb0e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:01.272 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.255+0000 7fb0e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:01.425 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.423+0000 7fb0f8ac3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:01.467 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.423+0000 7fb0f8ac3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:01.467 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.423+0000 7fb0f8ac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:01.467 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.423+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0f409f8f0 con 0x7fb0f41395c0 2024-09-11T02:50:01.468 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0e2ffd640 1 -- 172.21.15.98:0/3551748728 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10497 (secure 0 0 0) 0x7fb0ec0044a0 con 0x7fb0f41395c0 2024-09-11T02:50:01.468 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:01.469 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:01.469 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-11T02:50:01.469 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":15,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T02:49:57.894684+0000","last_up_change":"2024-09-11T02:49:54.844374+0000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T02:49:55.787515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6801","nonce":1456395826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6803","nonce":1456395826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6807","nonce":1456395826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6805","nonce":1456395826}]},"public_addr":"172.21.15.98:6801/1456395826","cluster_addr":"172.21.15.98:6803/1456395826","heartbeat_back_addr":"172.21.15.98:6807/1456395826","heartbeat_front_addr":"172.21.15.98:6805/1456395826","state":["exists","up"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6809","nonce":2313933591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6811","nonce":2313933591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6815","nonce":2313933591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6813","nonce":2313933591}]},"public_addr":"172.21.15.98:6809/2313933591","cluster_addr":"172.21.15.98:6811/2313933591","heartbeat_back_addr":"172.21.15.98:6815/2313933591","heartbeat_front_addr":"172.21.15.98:6813/2313933591","state":["exists","up"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6817","nonce":3987433753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3987433753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3987433753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3987433753}]},"public_addr":"172.21.15.98:6817/3987433753","cluster_addr":"172.21.15.98:6819/3987433753","heartbeat_back_addr":"172.21.15.98:6823/3987433753","heartbeat_front_addr":"172.21.15.98:6821/3987433753","state":["exists","up"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6801","nonce":373809828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6803","nonce":373809828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6807","nonce":373809828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6805","nonce":373809828}]},"public_addr":"172.21.15.190:6801/373809828","cluster_addr":"172.21.15.190:6803/373809828","heartbeat_back_addr":"172.21.15.190:6807/373809828","heartbeat_front_addr":"172.21.15.190:6805/373809828","state":["exists","up"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6817","nonce":1271446291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6819","nonce":1271446291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6823","nonce":1271446291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6821","nonce":1271446291}]},"public_addr":"172.21.15.190:6817/1271446291","cluster_addr":"172.21.15.190:6819/1271446291","heartbeat_back_addr":"172.21.15.190:6823/1271446291","heartbeat_front_addr":"172.21.15.190:6821/1271446291","state":["exists","up"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6809","nonce":1497028198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6811","nonce":1497028198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6815","nonce":1497028198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6813","nonce":1497028198}]},"public_addr":"172.21.15.190:6809/1497028198","cluster_addr":"172.21.15.190:6811/1497028198","heartbeat_back_addr":"172.21.15.190:6815/1497028198","heartbeat_front_addr":"172.21.15.190:6813/1497028198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.670036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.679318+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.766598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.015321+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.338711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.969084+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T02:50:01.472 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 10 librados: watch_flush enter 2024-09-11T02:50:01.472 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 10 librados: watch_flush exit 2024-09-11T02:50:01.473 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 20 client.4207.objecter shutdown clearing up homeless session... 2024-09-11T02:50:01.473 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 10 client.4207.objecter successfully canceled tick 2024-09-11T02:50:01.473 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0d403df40 msgr2=0x7fb0d4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.473 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0d403df40 0x7fb0d4040400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb0e40ae000 tx=0x7fb0e40a7070 comp rx=0 tx=0).stop 2024-09-11T02:50:01.473 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 10 monclient: shutdown 2024-09-11T02:50:01.474 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:01.474 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0f41395c0 msgr2=0x7fb0f409ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.474 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0f41395c0 0x7fb0f409ca10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c820 tx=0x7fb0ec00cce0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.474 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 shutdown_connections 2024-09-11T02:50:01.474 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0d403df40 0x7fb0d4040400 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.475 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0f41395c0 0x7fb0f409ca10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.475 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb0f40a6be0 0x7fb0f4134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.475 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 --2- 172.21.15.98:0/3551748728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0f40a6800 0x7fb0f41347d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.475 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 >> 172.21.15.98:0/3551748728 conn(0x7fb0f40abe00 msgr2=0x7fb0f40ab3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:01.475 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 shutdown_connections 2024-09-11T02:50:01.475 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 -- 172.21.15.98:0/3551748728 wait complete. 2024-09-11T02:50:01.476 INFO:teuthology.misc.health.smithi098.stderr:2024-09-11T02:50:01.427+0000 7fb0f8ac3640 1 librados: shutdown 2024-09-11T02:50:01.477 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-11T02:50:01.477 INFO:tasks.ceph:Creating RBD pool 2024-09-11T02:50:01.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T02:50:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.587+0000 7fde398a8640 10 monclient: get_monmap_and_config 2024-09-11T02:50:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.587+0000 7fde398a8640 10 monclient: build_initial_monmap 2024-09-11T02:50:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.587+0000 7fde398a8640 10 monclient: monmap: 2024-09-11T02:50:01.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:01.592090+0000 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:01.592090+0000 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:01.589 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:01.590 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:01.590 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.587+0000 7fde398a8640 1 Processor -- start 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.587+0000 7fde398a8640 1 -- start start 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.587+0000 7fde398a8640 10 monclient: init 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient: _add_conns ranks=[5,7,8,1,4,10,3,2,0,9,6] 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde34139b10 0x7fde340725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): picked mon.noname-h con 0x7fde34139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde3406fe30 0x7fde34139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): picked mon.noname-i con 0x7fde3406fe30 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde340a7a10 0x7fde340a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): picked mon.noname-e con 0x7fde340a7a10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fde340a9650 con 0x7fde34139b10 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fde34071350 con 0x7fde3406fe30 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fde34071630 con 0x7fde340a7a10 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient(hunting): authenticate will time out at 955.472250s 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde340a7a10 0x7fde340a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde340a7a10 0x7fde340a7e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:41956/0 (socket says 172.21.15.98:41956) 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 -- 172.21.15.98:0/3273310932 learned_addr learned my addr 172.21.15.98:0/3273310932 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): get_auth_request con 0x7fde340a7a10 auth_method 0 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde327fc640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde3406fe30 0x7fde34139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde32ffd640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde34139b10 0x7fde340725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 -- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde3406fe30 msgr2=0x7fde34139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde3406fe30 0x7fde34139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 -- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde34139b10 msgr2=0x7fde340725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde34139b10 0x7fde340725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 monclient: found mon.noname-e 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 -- 172.21.15.98:0/3273310932 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde340702b0 con 0x7fde340a7a10 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde327fc640 10 monclient: get_auth_request con 0x7fde3406fe30 auth_method 0 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde327fc640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde3406fe30 0x7fde34139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde337fe640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde340a7a10 0x7fde340a7e70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009e80 tx=0x7fde2c01c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4b2aa70f0dd24b09 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 1 -- 172.21.15.98:0/3273310932 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde2c0090d0 con 0x7fde340a7a10 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: dump: 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 5 monclient: authenticate success, global_id 4211 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 1 -- 172.21.15.98:0/3273310932 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2c0a5b80 con 0x7fde340a7a10 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 1 -- 172.21.15.98:0/3273310932 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde2c0a5d70 con 0x7fde340a7a10 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 4 set_mon_vals no callback set 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde31ffb640 10 monclient: dump: 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 10 monclient: shutdown 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- 172.21.15.98:0/3273310932 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde340a7a10 msgr2=0x7fde340a7e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde340a7a10 0x7fde340a7e70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009e80 tx=0x7fde2c01c910 comp rx=0 tx=0).stop 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- 172.21.15.98:0/3273310932 shutdown_connections 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde340a7a10 0x7fde340a7e70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde3406fe30 0x7fde34139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 --2- 172.21.15.98:0/3273310932 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde34139b10 0x7fde340725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- 172.21.15.98:0/3273310932 >> 172.21.15.98:0/3273310932 conn(0x7fde340ac770 msgr2=0x7fde340a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- 172.21.15.98:0/3273310932 shutdown_connections 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.591+0000 7fde398a8640 1 -- 172.21.15.98:0/3273310932 wait complete. 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: build_initial_monmap 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: monmap: 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 librados: starting msgr at 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 librados: starting objecter 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 Processor -- start 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 -- start start 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 librados: setting wanted keys 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 librados: calling monclient init 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: init 2024-09-11T02:50:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _add_conns ranks=[2,0,5,1,4,3,8,6,7] 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde340a7a10 0x7fde3413fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): picked mon.noname-b con 0x7fde340a7a10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34139b10 0x7fde34140100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): picked mon.noname-a con 0x7fde34139b10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde341d1840 0x7fde341d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): picked mon.noname-h con 0x7fde341d1840 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fde341435c0 con 0x7fde34139b10 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fde34143440 con 0x7fde340a7a10 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fde34143740 con 0x7fde341d1840 2024-09-11T02:50:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient(hunting): authenticate will time out at 955.477066s 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde340a7a10 0x7fde3413fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34139b10 0x7fde34140100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34139b10 0x7fde34140100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38918/0 (socket says 172.21.15.98:38918) 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 1 -- 172.21.15.98:0/2328526548 learned_addr learned my addr 172.21.15.98:0/2328526548 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient(hunting): get_auth_request con 0x7fde34139b10 auth_method 0 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): get_auth_request con 0x7fde340a7a10 auth_method 0 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde337fe640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde341d1840 0x7fde341d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde337fe640 10 monclient(hunting): get_auth_request con 0x7fde341d1840 auth_method 0 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2024-09-11T02:50:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 -- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde341d1840 msgr2=0x7fde341d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde341d1840 0x7fde341d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 -- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34139b10 msgr2=0x7fde34140100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34139b10 0x7fde34140100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 monclient: found mon.noname-b 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 -- 172.21.15.98:0/2328526548 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde340734d0 con 0x7fde340a7a10 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34139b10 0x7fde34140100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde32ffd640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde340a7a10 0x7fde3413fbc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c540 tx=0x7fde2400ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 client.?.objecter ms_handle_connect 0x7fde340a7a10 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 1 -- 172.21.15.98:0/2328526548 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde24015070 con 0x7fde340a7a10 2024-09-11T02:50:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: dump: 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: _finish_auth 0 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 5 monclient: authenticate success, global_id 4178 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 1 -- 172.21.15.98:0/2328526548 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde24011a70 con 0x7fde340a7a10 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _renew_subs 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde3406e310 con 0x7fde340a7a10 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 1 -- 172.21.15.98:0/2328526548 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde24011c60 con 0x7fde340a7a10 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde317fa640 4 set_mon_vals no callback set 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 monclient: dump: 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _renew_subs 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde340a5c10 con 0x7fde340a7a10 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 librados: init done 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde340a7e70 con 0x7fde340a7a10 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 1 -- 172.21.15.98:0/2328526548 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fde24011600 con 0x7fde340a7a10 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fde2003df40 0x7fde20040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 1 -- 172.21.15.98:0/2328526548 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7fde2404f0a0 con 0x7fde340a7a10 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 10 client.4178.objecter ms_dispatch 0x7fde34000ba0 osd_map(15..15 src has 1..15) 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 3 client.4178.objecter handle_osd_map got epochs [15,15] > 0 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 3 client.4178.objecter handle_osd_map decoding full epoch 15 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde1bfff640 20 client.4178.objecter dump_active .. 0 homeless 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fde2003df40 0x7fde20040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.595+0000 7fde327fc640 10 monclient: get_auth_request con 0x7fde2003df40 auth_method 0 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.599+0000 7fde1bfff640 10 client.4178.objecter ms_handle_connect 0x7fde2003df40 2024-09-11T02:50:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.599+0000 7fde327fc640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fde2003df40 0x7fde20040400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde340a5920 tx=0x7fde28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.603+0000 7fde1bfff640 1 -- 172.21.15.98:0/2328526548 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fde240193b0 con 0x7fde340a7a10 2024-09-11T02:50:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.603+0000 7fde1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.603+0000 7fde1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.759+0000 7fde398a8640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T02:50:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.759+0000 7fde398a8640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T02:50:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.759+0000 7fde398a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.759+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fde34073b20 con 0x7fde340a7a10 2024-09-11T02:50:01.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.931+0000 7fde1bfff640 1 -- 172.21.15.98:0/2328526548 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (secure 0 0 0) 0x7fde24022d60 con 0x7fde340a7a10 2024-09-11T02:50:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.931+0000 7fde1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-11T02:50:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.931+0000 7fde1bfff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-11T02:50:01.933 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2024-09-11T02:50:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 10 librados: watch_flush enter 2024-09-11T02:50:01.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 10 librados: watch_flush exit 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 20 client.4178.objecter shutdown clearing up homeless session... 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 10 client.4178.objecter successfully canceled tick 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fde2003df40 msgr2=0x7fde20040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fde2003df40 0x7fde20040400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fde340a5920 tx=0x7fde28008040 comp rx=0 tx=0).stop 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 10 monclient: shutdown 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde340a7a10 msgr2=0x7fde3413fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde340a7a10 0x7fde3413fbc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c540 tx=0x7fde2400ca00 comp rx=0 tx=0).stop 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 shutdown_connections 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fde341d1840 0x7fde341d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fde34139b10 0x7fde34140100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fde2003df40 0x7fde20040400 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 --2- 172.21.15.98:0/2328526548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde340a7a10 0x7fde3413fbc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 >> 172.21.15.98:0/2328526548 conn(0x7fde340ac770 msgr2=0x7fde340aa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 shutdown_connections 2024-09-11T02:50:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 -- 172.21.15.98:0/2328526548 wait complete. 2024-09-11T02:50:01.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:01.935+0000 7fde398a8640 1 librados: shutdown 2024-09-11T02:50:01.956 DEBUG:teuthology.orchestra.run.smithi098:> rbd --cluster ceph pool init rbd 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.015+0000 7f5da8982100 10 monclient: get_monmap_and_config 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.015+0000 7f5da8982100 10 monclient: build_initial_monmap 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.015+0000 7f5da8982100 10 monclient: monmap: 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:02.019779+0000 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:02.019779+0000 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:02.017 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.015+0000 7f5da8982100 1 Processor -- start 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.015+0000 7f5da8982100 1 -- start start 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.015+0000 7f5da8982100 10 monclient: init 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient: _reopen_session rank -1 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient: _add_conns ranks=[5,7,4,6,9,3,2,10,0,1,8] 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c531f70 0x55d78c532350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): picked mon.noname-h con 0x55d78c531f70 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x55d78c49cff0 0x55d78c49b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): picked mon.noname-i con 0x55d78c49cff0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c49be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): picked mon.noname-c con 0x55d78c49e4d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x55d78c36b160 con 0x55d78c49e4d0 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x55d78c37bd30 con 0x55d78c531f70 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x55d78c36ef70 con 0x55d78c49cff0 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): _renew_subs 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient(hunting): authenticate will time out at 955.900356s 2024-09-11T02:50:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c49be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c49be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55428/0 (socket says 172.21.15.98:55428) 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 -- 172.21.15.98:0/3449548826 learned_addr learned my addr 172.21.15.98:0/3449548826 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): get_auth_request con 0x55d78c49e4d0 auth_method 0 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6c8a640 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x55d78c49cff0 0x55d78c49b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6c8a640 10 monclient(hunting): get_auth_request con 0x55d78c49cff0 auth_method 0 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6c8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da748b640 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c531f70 0x55d78c532350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da748b640 10 monclient(hunting): get_auth_request con 0x55d78c531f70 auth_method 0 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da748b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da748b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:02.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da748b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient(hunting): handle_auth_done global_id 4216 payload 274 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 -- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x55d78c49cff0 msgr2=0x55d78c49b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x55d78c49cff0 0x55d78c49b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 -- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c531f70 msgr2=0x55d78c532350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c531f70 0x55d78c532350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient: _finish_hunting 0 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 monclient: found mon.noname-c 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 -- 172.21.15.98:0/3449548826 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d78c2e9c80 con 0x55d78c49e4d0 2024-09-11T02:50:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da86ed640 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c49be30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800a5f0 tx=0x7f5d9801d2a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=de6fd6dc074a65e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 1 -- 172.21.15.98:0/3449548826 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d980ac880 con 0x55d78c49e4d0 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: dump: 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: _finish_auth 0 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: _check_auth_tickets 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 1 -- 172.21.15.98:0/3449548826 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d980aca20 con 0x55d78c49e4d0 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 1 -- 172.21.15.98:0/3449548826 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d980acc50 con 0x55d78c49e4d0 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 5 monclient: authenticate success, global_id 4216 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 10 monclient: get_monmap_and_config success 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da8982100 4 set_mon_vals no callback set 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.019+0000 7f5da6489640 10 monclient: dump: 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:02.034 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: shutdown 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- 172.21.15.98:0/3449548826 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 msgr2=0x55d78c49be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c49be30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5d9800a5f0 tx=0x7f5d9801d2a0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- 172.21.15.98:0/3449548826 shutdown_connections 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c49be30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x55d78c49cff0 0x55d78c49b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 --2- 172.21.15.98:0/3449548826 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c531f70 0x55d78c532350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- 172.21.15.98:0/3449548826 >> 172.21.15.98:0/3449548826 conn(0x55d78c481b10 msgr2=0x55d78c481f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- 172.21.15.98:0/3449548826 shutdown_connections 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- 172.21.15.98:0/3449548826 wait complete. 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: build_initial_monmap 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: monmap: 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:02.036 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 librados: starting msgr at 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 librados: starting objecter 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 Processor -- start 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- start start 2024-09-11T02:50:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 librados: setting wanted keys 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 librados: calling monclient init 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: init 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: _reopen_session rank -1 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: _add_conns ranks=[4,1,5,2,3,6,8,7,0] 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c495ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): picked mon.noname-c con 0x55d78c49e4d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d78c531f70 0x55d78c496400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): picked mon.noname-f con 0x55d78c531f70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c496940 0x55d78c548ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): picked mon.noname-h con 0x55d78c496940 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x55d78c4a09b0 con 0x55d78c531f70 2024-09-11T02:50:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x55d78c310a70 con 0x55d78c49e4d0 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x55d78c532890 con 0x55d78c496940 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): _renew_subs 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient(hunting): authenticate will time out at 955.905932s 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c495ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c495ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55436/0 (socket says 172.21.15.98:55436) 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 -- 172.21.15.98:0/1686998876 learned_addr learned my addr 172.21.15.98:0/1686998876 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): get_auth_request con 0x55d78c49e4d0 auth_method 0 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:02.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da6c8a640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d78c531f70 0x55d78c496400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da6c8a640 10 monclient(hunting): get_auth_request con 0x55d78c531f70 auth_method 0 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da6c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da6c8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da6c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da86ed640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c496940 0x55d78c548ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient(hunting): handle_auth_done global_id 4225 payload 824 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 -- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c496940 msgr2=0x55d78c548ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c496940 0x55d78c548ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 -- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d78c531f70 msgr2=0x55d78c496400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:02.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d78c531f70 0x55d78c496400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 monclient: found mon.noname-c 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d78c5332f0 con 0x55d78c49e4d0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da86ed640 10 monclient: get_auth_request con 0x55d78c496940 auth_method 0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da86ed640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x55d78c496940 0x55d78c548ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da748b640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c495ec0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0086c0 tx=0x7f5d9c008ac0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 client.?.objecter ms_handle_connect 0x55d78c49e4d0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d9c01a070 con 0x55d78c49e4d0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:02.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: dump: 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:02.042 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d9c006b30 con 0x55d78c49e4d0 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d9c006ce0 con 0x55d78c49e4d0 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5d8f7fe640 10 monclient: dump: 2024-09-11T02:50:02.043 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:02.044 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da5c88640 4 set_mon_vals no callback set 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 5 monclient: authenticate success, global_id 4225 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: _renew_subs 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d78c49bda0 con 0x55d78c49e4d0 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: _renew_subs 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x55d78c533010 con 0x55d78c49e4d0 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 1 librados: init done 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.023+0000 7f5da8982100 10 librados: wait_for_osdmap waiting 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5d9c0066c0 con 0x55d78c49e4d0 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d8803c070 0x7f5d8803e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(16..16 src has 1..16) ==== 4335+0+0 (secure 0 0 0) 0x7f5d9c010030 con 0x55d78c49e4d0 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_map(16..16 src has 1..16) 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map got epochs [16,16] > 0 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map decoding full epoch 16 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 20 client.4225.objecter dump_active .. 0 homeless 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da8982100 10 librados: wait_for_osdmap done waiting 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da8982100 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da8982100 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T02:50:02.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da8982100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55d78c4985e0 con 0x55d78c49e4d0 2024-09-11T02:50:02.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da6c8a640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d8803c070 0x7f5d8803e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da6c8a640 10 monclient: get_auth_request con 0x7f5d8803c070 auth_method 0 2024-09-11T02:50:02.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5da6c8a640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d8803c070 0x7f5d8803e530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5d90005390 tx=0x7f5d90005320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:02.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.027+0000 7f5d8f7fe640 10 client.4225.objecter ms_handle_connect 0x7f5d8803c070 2024-09-11T02:50:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.951+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (secure 0 0 0) 0x7f5d9c01dd00 con 0x55d78c49e4d0 2024-09-11T02:50:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.951+0000 7f5d8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-11T02:50:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.951+0000 7f5d8f7fe640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-11T02:50:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.951+0000 7f5da8982100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.951+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d980aca20 con 0x55d78c49e4d0 2024-09-11T02:50:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f5d9c013510 con 0x55d78c49e4d0 2024-09-11T02:50:02.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da5c88640 10 client.4225.objecter _get_latest_version latest 17, waiting 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da5c88640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da5c88640 10 monclient: _renew_subs 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da5c88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da5c88640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=17}) -- 0x7f5d80003e00 con 0x55d78c49e4d0 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 8 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (secure 0 0 0) 0x7f5d9c012c90 con 0x55d78c49e4d0 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_map(17..17 src has 1..17) 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map got epochs [17,17] > 16 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map decoding incremental epoch 17 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 20 client.4225.objecter dump_active .. 0 homeless 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 10 librados: create oid=rbd_trash nspace= 2024-09-11T02:50:02.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 10 client.4225.objecter _op_submit op 0x55d78c549400 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 20 client.4225.objecter _calc_target epoch 17 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 20 client.4225.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 10 client.4225.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x55d78c549dc0 0x55d78c54c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 20 client.4225.objecter _get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 10 client.4225.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 20 client.4225.objecter get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 15 client.4225.objecter _session_op_assign 5 1 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 15 client.4225.objecter _send_op 1 to 2.3 on osd.5 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55d78c54c6f0 con 0x55d78c549dc0 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:02.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da8982100 5 client.4225.objecter 1 in flight 2024-09-11T02:50:02.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da86ed640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x55d78c549dc0 0x55d78c54c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:02.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da86ed640 10 monclient: get_auth_request con 0x55d78c549dc0 auth_method 0 2024-09-11T02:50:02.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5d8f7fe640 10 client.4225.objecter ms_handle_connect 0x55d78c549dc0 2024-09-11T02:50:02.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.955+0000 7f5da86ed640 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x55d78c549dc0 0x55d78c54c1b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:02.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 1 -- 172.21.15.98:0/1686998876 <== osd.5 v2:172.21.15.190:6808/1497028198 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f5d980b5040 con 0x55d78c549dc0 2024-09-11T02:50:02.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) 2024-09-11T02:50:02.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 10 client.4225.objecter in handle_osd_op_reply 2024-09-11T02:50:02.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 7 client.4225.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-11T02:50:02.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 10 client.4225.objecter op 0 rval 0 len 0 2024-09-11T02:50:02.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 15 client.4225.objecter handle_osd_op_reply completed tid 1 2024-09-11T02:50:02.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 15 client.4225.objecter _finish_op 1 2024-09-11T02:50:02.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:02.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 15 client.4225.objecter _session_op_remove 5 1 2024-09-11T02:50:02.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da86ed640 5 client.4225.objecter 0 in flight 2024-09-11T02:50:02.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.967+0000 7f5da8982100 10 librados: Objecter returned from create r=0 2024-09-11T02:50:02.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 10 client.4225.objecter _op_submit op 0x55d78c59e6d0 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 10 client.4225.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter _get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 10 client.4225.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 15 client.4225.objecter _session_op_assign 5 2 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 15 client.4225.objecter _send_op 2 to 2.2 on osd.5 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55d78c59ec20 con 0x55d78c549dc0 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:02.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 5 client.4225.objecter 1 in flight 2024-09-11T02:50:02.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 1 -- 172.21.15.98:0/1686998876 <== osd.5 v2:172.21.15.190:6808/1497028198 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f5d980b5040 con 0x55d78c549dc0 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter in handle_osd_op_reply 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 7 client.4225.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter op 0 rval 0 len 0 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 15 client.4225.objecter handle_osd_op_reply completed tid 2 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 15 client.4225.objecter _finish_op 2 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 15 client.4225.objecter _session_op_remove 5 2 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 5 client.4225.objecter 0 in flight 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 10 client.4225.objecter _op_submit op 0x55d78c59d830 2024-09-11T02:50:02.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 10 client.4225.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter _get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 10 client.4225.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 15 client.4225.objecter _session_op_assign 5 3 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 15 client.4225.objecter _send_op 3 to 2.2 on osd.5 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55d78c59dce0 con 0x55d78c549dc0 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da8982100 5 client.4225.objecter 1 in flight 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 1 -- 172.21.15.98:0/1686998876 <== osd.5 v2:172.21.15.190:6808/1497028198 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f5d980b5040 con 0x55d78c549dc0 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:50:02.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter in handle_osd_op_reply 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 7 client.4225.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 10 client.4225.objecter op 0 rval 0 len 0 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 15 client.4225.objecter handle_osd_op_reply completed tid 3 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 15 client.4225.objecter _finish_op 3 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 15 client.4225.objecter _session_op_remove 5 3 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da86ed640 5 client.4225.objecter 0 in flight 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da5487640 10 client.4225.objecter allocate_selfmanaged_snap; pool: 2 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da5487640 10 client.4225.objecter pool_op_submit 4 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da5487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:02.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:02.971+0000 7f5da5487640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f5d840030e0 con 0x55d78c49e4d0 2024-09-11T02:50:03.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 9 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (secure 0 0 0) 0x7f5d9c03e1f0 con 0x55d78c49e4d0 2024-09-11T02:50:03.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 pool_op_reply(tid 4 (0) Success v18) 2024-09-11T02:50:03.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 client.4225.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v18) 2024-09-11T02:50:03.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 client.4225.objecter have request 4 at 0x7f5d84003070 Op: create unmanaged snap 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 20 client.4225.objecter waiting for client to reach epoch 18 before calling back 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 monclient: _renew_subs 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=18}) -- 0x7f5d88042140 con 0x55d78c49e4d0 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 client.4225.objecter done 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 10 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (secure 0 0 0) 0x7f5d9c011660 con 0x55d78c49e4d0 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_map(18..18 src has 1..18) 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map got epochs [18,18] > 17 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map decoding incremental epoch 18 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5d8f7fe640 20 client.4225.objecter dump_active .. 0 homeless 2024-09-11T02:50:03.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 20 librados: queue_aio_write 0x55d78c599c60 completion 0x55d78c36aba0 write_seq 1 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 10 client.4225.objecter _op_submit op 0x7f5d800064f0 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 20 client.4225.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 20 client.4225.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 10 client.4225.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 20 client.4225.objecter _get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 10 client.4225.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 20 client.4225.objecter get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 15 client.4225.objecter _session_op_assign 5 5 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 15 client.4225.objecter _send_op 5 to 2.2 on osd.5 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f5d80006b30 con 0x55d78c549dc0 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:03.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.959+0000 7f5da5c88640 5 client.4225.objecter 1 in flight 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 1 -- 172.21.15.98:0/1686998876 <== osd.5 v2:172.21.15.190:6808/1497028198 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f5d980b5040 con 0x55d78c549dc0 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 10 client.4225.objecter in handle_osd_op_reply 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 7 client.4225.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 10 client.4225.objecter op 0 rval 0 len 0 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 10 client.4225.objecter op 1 rval 0 len 0 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 15 client.4225.objecter handle_osd_op_reply completed tid 5 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 15 client.4225.objecter _finish_op 5 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:03.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 15 client.4225.objecter _session_op_remove 5 5 2024-09-11T02:50:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 5 client.4225.objecter 0 in flight 2024-09-11T02:50:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da86ed640 20 librados: complete_aio_write 0x55d78c36aba0 2024-09-11T02:50:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da5487640 10 client.4225.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-11T02:50:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da5487640 10 client.4225.objecter pool_op_submit 6 2024-09-11T02:50:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da5487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:03.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:03.963+0000 7f5da5487640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f5d84004530 con 0x55d78c49e4d0 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 11 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (secure 0 0 0) 0x7f5d9c023070 con 0x55d78c49e4d0 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 pool_op_reply(tid 6 (0) Success v19) 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 client.4225.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v19) 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 client.4225.objecter have request 6 at 0x7f5d84003070 Op: delete unmanaged snap 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 20 client.4225.objecter waiting for client to reach epoch 19 before calling back 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 monclient: _renew_subs 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=19}) -- 0x7f5d880421f0 con 0x55d78c49e4d0 2024-09-11T02:50:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 client.4225.objecter done 2024-09-11T02:50:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 1 -- 172.21.15.98:0/1686998876 <== mon.4 v2:172.21.15.98:3302/0 12 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (secure 0 0 0) 0x7f5d9c014a60 con 0x55d78c49e4d0 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_map(19..19 src has 1..19) 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map got epochs [19,19] > 18 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 3 client.4225.objecter handle_osd_map decoding incremental epoch 19 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5d8f7fe640 20 client.4225.objecter dump_active .. 0 homeless 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 20 librados: queue_aio_write 0x55d78c599c60 completion 0x7f5d84001d70 write_seq 2 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 10 client.4225.objecter _op_submit op 0x7f5d800064f0 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 20 client.4225.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 20 client.4225.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 10 client.4225.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 20 client.4225.objecter _get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 10 client.4225.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 20 client.4225.objecter get_session s=0x55d78c496d80 osd=5 3 2024-09-11T02:50:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 15 client.4225.objecter _session_op_assign 5 7 2024-09-11T02:50:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 15 client.4225.objecter _send_op 7 to 2.2 on osd.5 2024-09-11T02:50:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 1 -- 172.21.15.98:0/1686998876 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f5d80006ed0 con 0x55d78c549dc0 2024-09-11T02:50:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.971+0000 7f5da5c88640 5 client.4225.objecter 1 in flight 2024-09-11T02:50:04.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 1 -- 172.21.15.98:0/1686998876 <== osd.5 v2:172.21.15.190:6808/1497028198 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f5d980b5040 con 0x55d78c549dc0 2024-09-11T02:50:04.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 10 client.4225.objecter ms_dispatch 0x55d78c383a30 osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 10 client.4225.objecter in handle_osd_op_reply 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 7 client.4225.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 10 client.4225.objecter op 0 rval 0 len 0 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 15 client.4225.objecter handle_osd_op_reply completed tid 7 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 15 client.4225.objecter _finish_op 7 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 4 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 15 client.4225.objecter _session_op_remove 5 7 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 5 client.4225.objecter 0 in flight 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da86ed640 20 librados: complete_aio_write 0x7f5d84001d70 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 10 librados: watch_flush enter 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 10 librados: watch_flush exit 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 10 client.4225.objecter close_session for osd.5 2024-09-11T02:50:04.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x55d78c549dc0 msgr2=0x55d78c54c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x55d78c549dc0 0x55d78c54c1b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 20 client.4225.objecter put_session s=0x55d78c496d80 osd=5 1 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 20 client.4225.objecter shutdown clearing up homeless session... 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 10 client.4225.objecter successfully canceled tick 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d8803c070 msgr2=0x7f5d8803e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.975+0000 7f5da8982100 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d8803c070 0x7f5d8803e530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5d90005390 tx=0x7f5d90005320 comp rx=0 tx=0).stop 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 10 monclient: shutdown 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 msgr2=0x55d78c495ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 1 --2- 172.21.15.98:0/1686998876 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x55d78c49e4d0 0x55d78c495ec0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c0086c0 tx=0x7f5d9c008ac0 comp rx=0 tx=0).stop 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da86ed640 1 -- 172.21.15.98:0/1686998876 reap_dead start 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 shutdown_connections 2024-09-11T02:50:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 >> 172.21.15.98:0/1686998876 conn(0x55d78c481b10 msgr2=0x55d78c4a3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 shutdown_connections 2024-09-11T02:50:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 1 -- 172.21.15.98:0/1686998876 wait complete. 2024-09-11T02:50:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:04.979+0000 7f5da8982100 1 librados: shutdown 2024-09-11T02:50:04.984 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T02:50:04.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-11T02:50:04.985 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient: get_monmap_and_config 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient: build_initial_monmap 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient: monmap: 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:05.113576+0000 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:05.113576+0000 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:05.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:05.111 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 1 Processor -- start 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 1 -- start start 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient: init 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient: _add_conns ranks=[1,9,7,4,3,10,2,0,5,8,6] 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f5380139b80 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f538006fe50 0x7f538013ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient(hunting): picked mon.noname-a con 0x7f538006fe50 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5380072640 0x7f5380140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient(hunting): picked mon.noname-i con 0x7f5380072640 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:05.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5380071370 con 0x7f5380139b80 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f53800719c0 con 0x7f538006fe50 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f5380071c10 con 0x7f5380072640 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f5384fe5640 10 monclient(hunting): authenticate will time out at 958.993734s 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f538006fe50 0x7f538013ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f538006fe50 0x7f538013ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38930/0 (socket says 172.21.15.98:38930) 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 -- 172.21.15.98:0/254668942 learned_addr learned my addr 172.21.15.98:0/254668942 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): get_auth_request con 0x7f538006fe50 auth_method 0 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537ed9d640 1 --2- 172.21.15.98:0/254668942 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5380072640 0x7f5380140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537ed9d640 10 monclient(hunting): get_auth_request con 0x7f5380072640 auth_method 0 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient(hunting): handle_auth_done global_id 4275 payload 274 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 -- 172.21.15.98:0/254668942 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5380072640 msgr2=0x7f5380140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 --2- 172.21.15.98:0/254668942 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5380072640 0x7f5380140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 -- 172.21.15.98:0/254668942 >> v1:172.21.15.190:6789/0 conn(0x7f5380139b80 legacy=0x7f5380139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:05.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 monclient: found mon.noname-a 2024-09-11T02:50:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537dd9b640 1 -- 172.21.15.98:0/254668942 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53800a9600 con 0x7f538006fe50 2024-09-11T02:50:05.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.111+0000 7f537ed9d640 1 --2- 172.21.15.98:0/254668942 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5380072640 0x7f5380140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:50:05.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537dd9b640 1 --2- 172.21.15.98:0/254668942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f538013ea00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5368009e50 tx=0x7f536801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b61f2536974778 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 1 -- 172.21.15.98:0/254668942 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53680090d0 con 0x7f538006fe50 2024-09-11T02:50:05.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: dump: 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.122 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: _finish_auth 0 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 1 -- 172.21.15.98:0/254668942 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53680a5b30 con 0x7f538006fe50 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 5 monclient: authenticate success, global_id 4275 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:05.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 1 -- 172.21.15.98:0/254668942 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53680a5d20 con 0x7f538006fe50 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537d59a640 10 monclient: dump: 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.124 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 4 set_mon_vals no callback set 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: shutdown 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- 172.21.15.98:0/254668942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 msgr2=0x7f538013ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 --2- 172.21.15.98:0/254668942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f538013ea00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5368009e50 tx=0x7f536801c8f0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- 172.21.15.98:0/254668942 shutdown_connections 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 --2- 172.21.15.98:0/254668942 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5380072640 0x7f5380140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 --2- 172.21.15.98:0/254668942 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f538013ea00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- 172.21.15.98:0/254668942 >> 172.21.15.98:0/254668942 conn(0x7f53800ac730 msgr2=0x7f53800a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- 172.21.15.98:0/254668942 shutdown_connections 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- 172.21.15.98:0/254668942 wait complete. 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: build_initial_monmap 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: monmap: 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:05.126 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 librados: starting msgr at 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 librados: starting objecter 2024-09-11T02:50:05.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 Processor -- start 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- start start 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 librados: setting wanted keys 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 librados: calling monclient init 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: init 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: _add_conns ranks=[0,1,4,3,7,5,6,8,2] 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f53801d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): picked mon.noname-a con 0x7f538006fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5380072640 0x7f5380138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): picked mon.noname-f con 0x7f5380072640 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5380139b80 0x7f538013d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): picked mon.noname-c con 0x7f5380139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5380143ec0 con 0x7f538006fe50 2024-09-11T02:50:05.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53800702d0 con 0x7f5380072640 2024-09-11T02:50:05.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5380144040 con 0x7f5380139b80 2024-09-11T02:50:05.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:05.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient(hunting): authenticate will time out at 958.998977s 2024-09-11T02:50:05.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f53801d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f53801d10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38940/0 (socket says 172.21.15.98:38940) 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 -- 172.21.15.98:0/2517989719 learned_addr learned my addr 172.21.15.98:0/2517989719 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): get_auth_request con 0x7f538006fe50 auth_method 0 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5380139b80 0x7f538013d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 10 monclient(hunting): get_auth_request con 0x7f5380139b80 auth_method 0 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537dd9b640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5380072640 0x7f5380138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537dd9b640 10 monclient(hunting): get_auth_request con 0x7f5380072640 auth_method 0 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 -- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5380139b80 msgr2=0x7f538013d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5380139b80 0x7f538013d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 -- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5380072640 msgr2=0x7f5380138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5380072640 0x7f5380138940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient: _finish_hunting 0 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 monclient: found mon.noname-a 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:05.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 -- 172.21.15.98:0/2517989719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53800a81b0 con 0x7f538006fe50 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537e59c640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f53801d10c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f537000c3c0 tx=0x7f537000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 client.?.objecter ms_handle_connect 0x7f538006fe50 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 1 -- 172.21.15.98:0/2517989719 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5370003620 con 0x7f538006fe50 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: dump: 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.133 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:05.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 1 -- 172.21.15.98:0/2517989719 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5370003800 con 0x7f538006fe50 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 5 monclient: authenticate success, global_id 4284 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: _renew_subs 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53801395e0 con 0x7f538006fe50 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 1 -- 172.21.15.98:0/2517989719 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5370003e20 con 0x7f538006fe50 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5376ffd640 10 monclient: dump: 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.135 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.136 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f537cd99640 4 set_mon_vals no callback set 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: _renew_subs 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538013f120 con 0x7f538006fe50 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.115+0000 7f5384fe5640 1 librados: init done 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5384fe5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5384fe5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5384fe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5380139f60 con 0x7f538006fe50 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 1 -- 172.21.15.98:0/2517989719 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f53700039b0 con 0x7f538006fe50 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f536003dfa0 0x7f5360040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 1 -- 172.21.15.98:0/2517989719 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f537004e3c0 con 0x7f538006fe50 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 10 client.4284.objecter ms_dispatch 0x7f5380000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 3 client.4284.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 3 client.4284.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 20 client.4284.objecter dump_active .. 0 homeless 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f537dd9b640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f536003dfa0 0x7f5360040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f537dd9b640 10 monclient: get_auth_request con 0x7f536003dfa0 auth_method 0 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f5376ffd640 10 client.4284.objecter ms_handle_connect 0x7f536003dfa0 2024-09-11T02:50:05.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.119+0000 7f537dd9b640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f536003dfa0 0x7f5360040460 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f53680ae000 tx=0x7f53680a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.123+0000 7f5376ffd640 1 -- 172.21.15.98:0/2517989719 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f5370001090 con 0x7f538006fe50 2024-09-11T02:50:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.123+0000 7f5376ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.123+0000 7f5376ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T02:50:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T02:50:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f5380069090 con 0x7f538006fe50 2024-09-11T02:50:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5376ffd640 1 -- 172.21.15.98:0/2517989719 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v3) ==== 86+0+159 (secure 0 0 0) 0x7f5370011590 con 0x7f538006fe50 2024-09-11T02:50:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5376ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-11T02:50:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5376ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:05.295 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 10 librados: watch_flush enter 2024-09-11T02:50:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 10 librados: watch_flush exit 2024-09-11T02:50:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 20 client.4284.objecter shutdown clearing up homeless session... 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 10 client.4284.objecter successfully canceled tick 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f536003dfa0 msgr2=0x7f5360040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f536003dfa0 0x7f5360040460 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f53680ae000 tx=0x7f53680a7070 comp rx=0 tx=0).stop 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 10 monclient: shutdown 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 msgr2=0x7f53801d10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f53801d10c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f537000c3c0 tx=0x7f537000c880 comp rx=0 tx=0).stop 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 shutdown_connections 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f536003dfa0 0x7f5360040460 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5380139b80 0x7f538013d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5380072640 0x7f5380138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 --2- 172.21.15.98:0/2517989719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f538006fe50 0x7f53801d10c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.295+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 >> 172.21.15.98:0/2517989719 conn(0x7f53800ac730 msgr2=0x7f53800aa170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.299+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 shutdown_connections 2024-09-11T02:50:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.299+0000 7f5384fe5640 1 -- 172.21.15.98:0/2517989719 wait complete. 2024-09-11T02:50:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.299+0000 7f5384fe5640 1 librados: shutdown 2024-09-11T02:50:05.313 INFO:teuthology.orchestra.run.smithi098.stdout:[{"version":3,"timestamp":"2024-09-11T02:49:54.663914+0000","name":"","changes":[{"name":"osd.2/osd_mclock_max_capacity_iops_ssd","new_value":"8888.961107"}]}] 2024-09-11T02:50:05.314 INFO:tasks.ceph_manager:config epoch is 3 2024-09-11T02:50:05.314 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T02:50:05.314 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T02:50:05.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.427+0000 7fee67724640 10 monclient: get_monmap_and_config 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.427+0000 7fee67724640 10 monclient: build_initial_monmap 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.427+0000 7fee67724640 10 monclient: monmap: 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:05.432746+0000 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:05.432746+0000 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:05.429 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:05.430 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 Processor -- start 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 -- start start 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient: init 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient: _add_conns ranks=[8,2,4,3,6,7,0,9,1,10,5] 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee601379a0 0x7fee60137d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): picked mon.noname-e con 0x7fee601379a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee6009d7e0 0x7fee600a4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): picked mon.noname-b con 0x7fee6009d7e0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee600a4d60 0x7fee600a5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): picked mon.noname-c con 0x7fee600a4d60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fee600a56a0 con 0x7fee6009d7e0 2024-09-11T02:50:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fee600a3030 con 0x7fee600a4d60 2024-09-11T02:50:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fee60077690 con 0x7fee601379a0 2024-09-11T02:50:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee67724640 10 monclient(hunting): authenticate will time out at 959.313682s 2024-09-11T02:50:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee601379a0 0x7fee60137d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee600a4d60 0x7fee600a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee6009d7e0 0x7fee600a4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee601379a0 0x7fee60137d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:41972/0 (socket says 172.21.15.98:41972) 2024-09-11T02:50:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): get_auth_request con 0x7fee600a4d60 auth_method 0 2024-09-11T02:50:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 1 -- 172.21.15.98:0/447665959 learned_addr learned my addr 172.21.15.98:0/447665959 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 10 monclient(hunting): get_auth_request con 0x7fee601379a0 auth_method 0 2024-09-11T02:50:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 10 monclient(hunting): get_auth_request con 0x7fee6009d7e0 auth_method 0 2024-09-11T02:50:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee654c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee64cbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2024-09-11T02:50:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 1 -- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee601379a0 msgr2=0x7fee60137d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 1 --2- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee601379a0 0x7fee60137d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 1 -- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee6009d7e0 msgr2=0x7fee600a4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 1 --2- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee6009d7e0 0x7fee600a4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient: _finish_hunting 0 2024-09-11T02:50:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 1 monclient: found mon.noname-c 2024-09-11T02:50:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:05.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.431+0000 7fee65cc1640 1 -- 172.21.15.98:0/447665959 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee60077970 con 0x7fee600a4d60 2024-09-11T02:50:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee65cc1640 1 --2- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee600a4d60 0x7fee600a5160 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fee50009dc0 tx=0x7fee5001c850 comp rx=0 tx=0).ready entity=mon.4 client_cookie=fd9be65895f3f701 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 1 -- 172.21.15.98:0/447665959 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee500b2070 con 0x7fee600a4d60 2024-09-11T02:50:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: dump: 2024-09-11T02:50:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.446 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.447 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.448 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: _finish_auth 0 2024-09-11T02:50:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 1 -- 172.21.15.98:0/447665959 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee500a5ac0 con 0x7fee600a4d60 2024-09-11T02:50:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:05.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 1 -- 172.21.15.98:0/447665959 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee500a5cf0 con 0x7fee600a4d60 2024-09-11T02:50:05.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 5 monclient: authenticate success, global_id 4243 2024-09-11T02:50:05.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:05.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 4 set_mon_vals no callback set 2024-09-11T02:50:05.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:05.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5ffff640 10 monclient: dump: 2024-09-11T02:50:05.450 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.450 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.450 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.450 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: shutdown 2024-09-11T02:50:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee600a4d60 msgr2=0x7fee600a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 --2- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee600a4d60 0x7fee600a5160 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fee50009dc0 tx=0x7fee5001c850 comp rx=0 tx=0).stop 2024-09-11T02:50:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- 172.21.15.98:0/447665959 shutdown_connections 2024-09-11T02:50:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 --2- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee600a4d60 0x7fee600a5160 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 --2- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee6009d7e0 0x7fee600a4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 --2- 172.21.15.98:0/447665959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee601379a0 0x7fee60137d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- 172.21.15.98:0/447665959 >> 172.21.15.98:0/447665959 conn(0x7fee6006ed00 msgr2=0x7fee6006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- 172.21.15.98:0/447665959 shutdown_connections 2024-09-11T02:50:05.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- 172.21.15.98:0/447665959 wait complete. 2024-09-11T02:50:05.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: build_initial_monmap 2024-09-11T02:50:05.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: monmap: 2024-09-11T02:50:05.455 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:05.456 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:05.457 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:05.458 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 librados: starting msgr at 2024-09-11T02:50:05.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 librados: starting objecter 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 Processor -- start 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- start start 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 librados: setting wanted keys 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 librados: calling monclient init 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: init 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: _add_conns ranks=[1,6,3,2,8,4,5,7,0] 2024-09-11T02:50:05.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6009d7e0 0x7fee60132640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): picked mon.noname-f con 0x7fee6009d7e0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee600a4d60 0x7fee60132b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): picked mon.noname-d con 0x7fee600a4d60 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee601379a0 0x7fee600a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): picked mon.noname-g con 0x7fee601379a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee60141460 con 0x7fee6009d7e0 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fee600a2e30 con 0x7fee601379a0 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fee601415e0 con 0x7fee600a4d60 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient(hunting): authenticate will time out at 959.318902s 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee600a4d60 0x7fee60132b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee600a4d60 0x7fee60132b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57084/0 (socket says 172.21.15.98:57084) 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 -- 172.21.15.98:0/3500162805 learned_addr learned my addr 172.21.15.98:0/3500162805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): get_auth_request con 0x7fee600a4d60 auth_method 0 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee654c0640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6009d7e0 0x7fee60132640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee654c0640 10 monclient(hunting): get_auth_request con 0x7fee6009d7e0 auth_method 0 2024-09-11T02:50:05.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee654c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee654c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee654c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee65cc1640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee601379a0 0x7fee600a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 -- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee601379a0 msgr2=0x7fee600a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee601379a0 0x7fee600a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 -- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6009d7e0 msgr2=0x7fee60132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6009d7e0 0x7fee60132640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient: _finish_hunting 0 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 monclient: found mon.noname-d 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:05.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 -- 172.21.15.98:0/3500162805 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee60133630 con 0x7fee600a4d60 2024-09-11T02:50:05.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee65cc1640 10 monclient: get_auth_request con 0x7fee601379a0 auth_method 0 2024-09-11T02:50:05.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee65cc1640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee601379a0 0x7fee600a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:05.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee64cbf640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee600a4d60 0x7fee60132b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fee5400e930 tx=0x7fee5400edf0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 client.?.objecter ms_handle_connect 0x7fee600a4d60 2024-09-11T02:50:05.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 1 -- 172.21.15.98:0/3500162805 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee54017070 con 0x7fee600a4d60 2024-09-11T02:50:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: dump: 2024-09-11T02:50:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:05.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:05.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 1 -- 172.21.15.98:0/3500162805 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee54013470 con 0x7fee600a4d60 2024-09-11T02:50:05.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:05.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 1 -- 172.21.15.98:0/3500162805 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee54013a90 con 0x7fee600a4d60 2024-09-11T02:50:05.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:05.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5dffb640 10 monclient: dump: 2024-09-11T02:50:05.475 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.475 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.475 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.475 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.475 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.475 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.476 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee5effd640 4 set_mon_vals no callback set 2024-09-11T02:50:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 5 monclient: authenticate success, global_id 4245 2024-09-11T02:50:05.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: _renew_subs 2024-09-11T02:50:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee60120000 con 0x7fee600a4d60 2024-09-11T02:50:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: _renew_subs 2024-09-11T02:50:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:05.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fee60137f80 con 0x7fee600a4d60 2024-09-11T02:50:05.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.435+0000 7fee67724640 1 librados: init done 2024-09-11T02:50:05.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee67724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee67724640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee67724640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:05.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee6013e1f0 con 0x7fee600a4d60 2024-09-11T02:50:05.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 1 -- 172.21.15.98:0/3500162805 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fee54013c30 con 0x7fee600a4d60 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee4c03df70 0x7fee4c040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 1 -- 172.21.15.98:0/3500162805 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fee5400f040 con 0x7fee600a4d60 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 10 client.4245.objecter ms_dispatch 0x7fee60000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 3 client.4245.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 3 client.4245.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 20 client.4245.objecter dump_active .. 0 homeless 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee654c0640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee4c03df70 0x7fee4c040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee654c0640 10 monclient: get_auth_request con 0x7fee4c03df70 auth_method 0 2024-09-11T02:50:05.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee5dffb640 10 client.4245.objecter ms_handle_connect 0x7fee4c03df70 2024-09-11T02:50:05.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.439+0000 7fee654c0640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee4c03df70 0x7fee4c040430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fee580099c0 tx=0x7fee58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.443+0000 7fee5dffb640 1 -- 172.21.15.98:0/3500162805 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fee54021070 con 0x7fee600a4d60 2024-09-11T02:50:05.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.443+0000 7fee5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.443+0000 7fee5dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:05.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.659+0000 7fee67724640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-11T02:50:05.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.659+0000 7fee67724640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-11T02:50:05.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.659+0000 7fee67724640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:05.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.659+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fee600a32e0 con 0x7fee600a4d60 2024-09-11T02:50:05.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.659+0000 7fee5dffb640 1 -- 172.21.15.98:0/3500162805 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+93242 (secure 0 0 0) 0x7fee54021070 con 0x7fee600a4d60 2024-09-11T02:50:05.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.659+0000 7fee5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-11T02:50:05.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.659+0000 7fee5dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:05.662 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:05.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 10 librados: watch_flush enter 2024-09-11T02:50:05.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 10 librados: watch_flush exit 2024-09-11T02:50:05.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 20 client.4245.objecter shutdown clearing up homeless session... 2024-09-11T02:50:05.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 10 client.4245.objecter successfully canceled tick 2024-09-11T02:50:05.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee4c03df70 msgr2=0x7fee4c040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee4c03df70 0x7fee4c040430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fee580099c0 tx=0x7fee58008040 comp rx=0 tx=0).stop 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 10 monclient: shutdown 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee600a4d60 msgr2=0x7fee60132b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee600a4d60 0x7fee60132b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fee5400e930 tx=0x7fee5400edf0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 shutdown_connections 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee4c03df70 0x7fee4c040430 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee601379a0 0x7fee600a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee600a4d60 0x7fee60132b80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 --2- 172.21.15.98:0/3500162805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee6009d7e0 0x7fee60132640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 >> 172.21.15.98:0/3500162805 conn(0x7fee6006ed00 msgr2=0x7fee6009f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:05.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 shutdown_connections 2024-09-11T02:50:05.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 -- 172.21.15.98:0/3500162805 wait complete. 2024-09-11T02:50:05.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.663+0000 7fee67724640 1 librados: shutdown 2024-09-11T02:50:05.678 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":3356844748},{"type":"v1","addr":"172.21.15.190:6825","nonce":3356844748}]},"active_addr":"172.21.15.190:6825/3356844748","active_change":"2024-09-11T02:49:49.723916+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2853191297}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":2136587179}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":92938946}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.190:0","nonce":4142327862}]}]} 2024-09-11T02:50:05.681 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T02:50:05.681 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T02:50:05.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T02:50:05.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient: get_monmap_and_config 2024-09-11T02:50:05.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient: build_initial_monmap 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient: monmap: 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:05.802255+0000 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:05.802255+0000 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:05.799 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:05.800 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:05.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 1 Processor -- start 2024-09-11T02:50:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 1 -- start start 2024-09-11T02:50:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient: init 2024-09-11T02:50:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient: _add_conns ranks=[0,3,6,9,5,7,10,1,2,8,4] 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f350c06f6c0 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f350c072940 0x7f350c0701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient(hunting): picked mon.noname-g con 0x7f350c072940 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f350c13ec80 0x7f350c141070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient(hunting): picked mon.noname-d con 0x7f350c13ec80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350c0706f0 con 0x7f350c06f6c0 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f350c071ff0 con 0x7f350c072940 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f350c072d20 con 0x7f350c13ec80 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:05.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f3511fbc640 10 monclient(hunting): authenticate will time out at 959.682476s 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350b7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f350c06f6c0 0x7f350c06faa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53884/0 (socket says 172.21.15.98:53884) 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350b7fe640 1 -- 172.21.15.98:0/1163346603 learned_addr learned my addr 172.21.15.98:0/1163346603 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f350c13ec80 0x7f350c141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): get_auth_request con 0x7f350c13ec80 auth_method 0 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350a7fc640 1 -- 172.21.15.98:0/1163346603 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 771251886 0 0) 0x7f34fc001120 con 0x7f350c06f6c0 2024-09-11T02:50:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350a7fc640 10 monclient(hunting): _negotiate 2024-09-11T02:50:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350a7fc640 10 monclient(hunting): my global_id is 4293 2024-09-11T02:50:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350affd640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f350c072940 0x7f350c0701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350a7fc640 1 -- 172.21.15.98:0/1163346603 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34f4003530 con 0x7f350c06f6c0 2024-09-11T02:50:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient(hunting): handle_auth_done global_id 4254 payload 274 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 1 -- 172.21.15.98:0/1163346603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f350c072940 msgr2=0x7f350c0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f350c072940 0x7f350c0701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 1 -- 172.21.15.98:0/1163346603 >> v1:172.21.15.98:6789/0 conn(0x7f350c06f6c0 legacy=0x7f350c06faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 1 monclient: found mon.noname-d 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350bfff640 1 -- 172.21.15.98:0/1163346603 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c0a8750 con 0x7f350c13ec80 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350affd640 10 monclient: get_auth_request con 0x7f350c072940 auth_method 0 2024-09-11T02:50:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.799+0000 7f350affd640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f350c072940 0x7f350c0701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f350c13ec80 0x7f350c141070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3504009dc0 tx=0x7f350401c820 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e54b9d5479617543 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 1 -- 172.21.15.98:0/1163346603 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f35040b2070 con 0x7f350c13ec80 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: dump: 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 5 monclient: authenticate success, global_id 4254 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 1 -- 172.21.15.98:0/1163346603 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35040a5af0 con 0x7f350c13ec80 2024-09-11T02:50:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 4 set_mon_vals no callback set 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 1 -- 172.21.15.98:0/1163346603 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f35040a5ce0 con 0x7f350c13ec80 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350a7fc640 10 monclient: dump: 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient: shutdown 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- 172.21.15.98:0/1163346603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f350c13ec80 msgr2=0x7f350c141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f350c13ec80 0x7f350c141070 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3504009dc0 tx=0x7f350401c820 comp rx=0 tx=0).stop 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- 172.21.15.98:0/1163346603 shutdown_connections 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f350c13ec80 0x7f350c141070 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 --2- 172.21.15.98:0/1163346603 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f350c072940 0x7f350c0701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- 172.21.15.98:0/1163346603 >> 172.21.15.98:0/1163346603 conn(0x7f350c0acac0 msgr2=0x7f350c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:05.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- 172.21.15.98:0/1163346603 shutdown_connections 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- 172.21.15.98:0/1163346603 wait complete. 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient: build_initial_monmap 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient: monmap: 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:05.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:05.822 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 librados: starting msgr at 2024-09-11T02:50:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 librados: starting objecter 2024-09-11T02:50:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 Processor -- start 2024-09-11T02:50:05.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- start start 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 librados: setting wanted keys 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 librados: calling monclient init 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient: init 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient: _add_conns ranks=[8,2,0,3,1,6,5,4,7] 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f350c06f6c0 0x7f350c138a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): picked mon.noname-e con 0x7f350c06f6c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f350c072940 0x7f350c138f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): picked mon.noname-b con 0x7f350c072940 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c13ec80 0x7f350c139490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): picked mon.noname-a con 0x7f350c13ec80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f350c143f50 con 0x7f350c13ec80 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f350c143dd0 con 0x7f350c072940 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f350c1440d0 con 0x7f350c06f6c0 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f3511fbc640 10 monclient(hunting): authenticate will time out at 959.687457s 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c13ec80 0x7f350c139490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f350c06f6c0 0x7f350c138a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c13ec80 0x7f350c139490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38950/0 (socket says 172.21.15.98:38950) 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 1 -- 172.21.15.98:0/2064683598 learned_addr learned my addr 172.21.15.98:0/2064683598 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 10 monclient(hunting): get_auth_request con 0x7f350c13ec80 auth_method 0 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f350c072940 0x7f350c138f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): get_auth_request con 0x7f350c06f6c0 auth_method 0 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 10 monclient(hunting): get_auth_request con 0x7f350c072940 auth_method 0 2024-09-11T02:50:05.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 1 -- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f350c072940 msgr2=0x7f350c138f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f350c072940 0x7f350c138f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 1 -- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c13ec80 msgr2=0x7f350c139490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c13ec80 0x7f350c139490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.803+0000 7f350b7fe640 1 monclient: found mon.noname-e 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f350b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f350b7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f350b7fe640 1 -- 172.21.15.98:0/2064683598 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c0aa6f0 con 0x7f350c06f6c0 2024-09-11T02:50:05.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f350b7fe640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f350c06f6c0 0x7f350c138a10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00c790 tx=0x7f34fc00cc50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 client.?.objecter ms_handle_connect 0x7f350c06f6c0 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 1 -- 172.21.15.98:0/2064683598 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34fc017070 con 0x7f350c06f6c0 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: dump: 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.828 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.829 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.830 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.830 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:05.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 1 -- 172.21.15.98:0/2064683598 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34fc00ce40 con 0x7f350c06f6c0 2024-09-11T02:50:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 1 -- 172.21.15.98:0/2064683598 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34fc005d30 con 0x7f350c06f6c0 2024-09-11T02:50:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 monclient: dump: 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:05.832 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:05.834 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3509ffb640 4 set_mon_vals no callback set 2024-09-11T02:50:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 5 monclient: authenticate success, global_id 4229 2024-09-11T02:50:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 monclient: _renew_subs 2024-09-11T02:50:05.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f350c13a340 con 0x7f350c06f6c0 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 monclient: _renew_subs 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f350c13db80 con 0x7f350c06f6c0 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 1 librados: init done 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f350c06faa0 con 0x7f350c06f6c0 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 1 -- 172.21.15.98:0/2064683598 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f34fc0058c0 con 0x7f350c06f6c0 2024-09-11T02:50:05.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f803df40 0x7f34f8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 1 -- 172.21.15.98:0/2064683598 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f34fc04e210 con 0x7f350c06f6c0 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 client.4229.objecter ms_dispatch 0x7f350c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 3 client.4229.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 3 client.4229.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 20 client.4229.objecter dump_active .. 0 homeless 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f350affd640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f803df40 0x7f34f8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f350affd640 10 monclient: get_auth_request con 0x7f34f803df40 auth_method 0 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f34f3fff640 10 client.4229.objecter ms_handle_connect 0x7f34f803df40 2024-09-11T02:50:05.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.807+0000 7f350affd640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f803df40 0x7f34f8040400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f350c13a9b0 tx=0x7f3500008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:05.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.811+0000 7f34f3fff640 1 -- 172.21.15.98:0/2064683598 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f350c0706f0 con 0x7f350c06f6c0 2024-09-11T02:50:05.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.811+0000 7f34f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:05.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.811+0000 7f34f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:05.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.987+0000 7f3511fbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:05.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.987+0000 7f3511fbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.987+0000 7f3511fbc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.987+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f350c069070 con 0x7f350c06f6c0 2024-09-11T02:50:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.987+0000 7f34f3fff640 1 -- 172.21.15.98:0/2064683598 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12401 (secure 0 0 0) 0x7f34fc007250 con 0x7f350c06f6c0 2024-09-11T02:50:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.987+0000 7f34f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:05.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.987+0000 7f34f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:05.989 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:05.989 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T02:50:04.952025+0000","last_up_change":"2024-09-11T02:49:54.844374+0000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T02:49:55.787515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T02:50:01.768815+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6801","nonce":1456395826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6803","nonce":1456395826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6807","nonce":1456395826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6805","nonce":1456395826}]},"public_addr":"172.21.15.98:6801/1456395826","cluster_addr":"172.21.15.98:6803/1456395826","heartbeat_back_addr":"172.21.15.98:6807/1456395826","heartbeat_front_addr":"172.21.15.98:6805/1456395826","state":["exists","up"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6809","nonce":2313933591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6811","nonce":2313933591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6815","nonce":2313933591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6813","nonce":2313933591}]},"public_addr":"172.21.15.98:6809/2313933591","cluster_addr":"172.21.15.98:6811/2313933591","heartbeat_back_addr":"172.21.15.98:6815/2313933591","heartbeat_front_addr":"172.21.15.98:6813/2313933591","state":["exists","up"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6817","nonce":3987433753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3987433753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3987433753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3987433753}]},"public_addr":"172.21.15.98:6817/3987433753","cluster_addr":"172.21.15.98:6819/3987433753","heartbeat_back_addr":"172.21.15.98:6823/3987433753","heartbeat_front_addr":"172.21.15.98:6821/3987433753","state":["exists","up"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6801","nonce":373809828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6803","nonce":373809828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6807","nonce":373809828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6805","nonce":373809828}]},"public_addr":"172.21.15.190:6801/373809828","cluster_addr":"172.21.15.190:6803/373809828","heartbeat_back_addr":"172.21.15.190:6807/373809828","heartbeat_front_addr":"172.21.15.190:6805/373809828","state":["exists","up"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6817","nonce":1271446291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6819","nonce":1271446291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6823","nonce":1271446291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6821","nonce":1271446291}]},"public_addr":"172.21.15.190:6817/1271446291","cluster_addr":"172.21.15.190:6819/1271446291","heartbeat_back_addr":"172.21.15.190:6823/1271446291","heartbeat_front_addr":"172.21.15.190:6821/1271446291","state":["exists","up"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6809","nonce":1497028198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6811","nonce":1497028198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6815","nonce":1497028198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6813","nonce":1497028198}]},"public_addr":"172.21.15.190:6809/1497028198","cluster_addr":"172.21.15.190:6811/1497028198","heartbeat_back_addr":"172.21.15.190:6815/1497028198","heartbeat_front_addr":"172.21.15.190:6813/1497028198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.670036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.679318+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.766598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.015321+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.338711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.969084+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T02:50:05.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 10 librados: watch_flush enter 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 10 librados: watch_flush exit 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 20 client.4229.objecter shutdown clearing up homeless session... 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 10 client.4229.objecter successfully canceled tick 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f803df40 msgr2=0x7f34f8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f803df40 0x7f34f8040400 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f350c13a9b0 tx=0x7f3500008040 comp rx=0 tx=0).stop 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 10 monclient: shutdown 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f350c06f6c0 msgr2=0x7f350c138a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f350c06f6c0 0x7f350c138a10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00c790 tx=0x7f34fc00cc50 comp rx=0 tx=0).stop 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 shutdown_connections 2024-09-11T02:50:05.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f350c13ec80 0x7f350c139490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f803df40 0x7f34f8040400 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f350c072940 0x7f350c138f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 --2- 172.21.15.98:0/2064683598 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f350c06f6c0 0x7f350c138a10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:05.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 >> 172.21.15.98:0/2064683598 conn(0x7f350c0acac0 msgr2=0x7f350c0aad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:05.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 shutdown_connections 2024-09-11T02:50:05.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 -- 172.21.15.98:0/2064683598 wait complete. 2024-09-11T02:50:05.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:05.991+0000 7f3511fbc640 1 librados: shutdown 2024-09-11T02:50:06.007 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T02:50:06.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient: monmap: 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.127497+0000 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.127497+0000 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.124 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.125 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 1 Processor -- start 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 1 -- start start 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient: init 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient: _add_conns ranks=[3,0,2,10,9,7,6,4,1,8,5] 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49000a37a0 0x7f490009e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient(hunting): picked mon.noname-g con 0x7f49000a37a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49000a3b80 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:06.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f490013c9c0 0x7f490013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient(hunting): picked mon.noname-b con 0x7f490013c9c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f490009d440 con 0x7f49000a3b80 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f49000a1b90 con 0x7f490013c9c0 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f490009d720 con 0x7f49000a37a0 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f49076dc640 10 monclient(hunting): authenticate will time out at 960.007732s 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f4905c79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f490013c9c0 0x7f490013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f4904c77640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49000a3b80 0x7f49000a3f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53898/0 (socket says 172.21.15.98:53898) 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f4904c77640 1 -- 172.21.15.98:0/4102229865 learned_addr learned my addr 172.21.15.98:0/4102229865 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:06.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f4905c79640 10 monclient(hunting): get_auth_request con 0x7f490013c9c0 auth_method 0 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f4905c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f4905c79640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.123+0000 7f4905c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905478640 1 --2- 172.21.15.98:0/4102229865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49000a37a0 0x7f490009e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905478640 10 monclient(hunting): get_auth_request con 0x7f49000a37a0 auth_method 0 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905478640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905478640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 1 -- 172.21.15.98:0/4102229865 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1260157024 0 0) 0x7f490009d440 con 0x7f49000a3b80 2024-09-11T02:50:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient(hunting): _negotiate 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient(hunting): my global_id is 4311 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 1 -- 172.21.15.98:0/4102229865 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48ec003530 con 0x7f49000a3b80 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 10 monclient(hunting): handle_auth_done global_id 4205 payload 274 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 1 -- 172.21.15.98:0/4102229865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49000a37a0 msgr2=0x7f490009e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 1 --2- 172.21.15.98:0/4102229865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49000a37a0 0x7f490009e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 1 -- 172.21.15.98:0/4102229865 >> v1:172.21.15.98:6789/0 conn(0x7f49000a3b80 legacy=0x7f49000a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 1 monclient: found mon.noname-b 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 1 -- 172.21.15.98:0/4102229865 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490009da00 con 0x7f490013c9c0 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f4905c79640 1 --2- 172.21.15.98:0/4102229865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f490013c9c0 0x7f490013edb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009dc0 tx=0x7f48f801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=969b988da80d2914 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 1 -- 172.21.15.98:0/4102229865 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48f80b2070 con 0x7f490013c9c0 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: dump: 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 1 -- 172.21.15.98:0/4102229865 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f80a59d0 con 0x7f490013c9c0 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 1 -- 172.21.15.98:0/4102229865 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48f80a5b80 con 0x7f490013c9c0 2024-09-11T02:50:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 5 monclient: authenticate success, global_id 4205 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 4 set_mon_vals no callback set 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f48f3fff640 10 monclient: dump: 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 10 monclient: shutdown 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 -- 172.21.15.98:0/4102229865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f490013c9c0 msgr2=0x7f490013edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 --2- 172.21.15.98:0/4102229865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f490013c9c0 0x7f490013edb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009dc0 tx=0x7f48f801c8e0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 -- 172.21.15.98:0/4102229865 shutdown_connections 2024-09-11T02:50:06.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 --2- 172.21.15.98:0/4102229865 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f490013c9c0 0x7f490013edb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 --2- 172.21.15.98:0/4102229865 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49000a37a0 0x7f490009e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 -- 172.21.15.98:0/4102229865 >> 172.21.15.98:0/4102229865 conn(0x7f49000a4890 msgr2=0x7f49000a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 -- 172.21.15.98:0/4102229865 shutdown_connections 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 1 -- 172.21.15.98:0/4102229865 wait complete. 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.127+0000 7f49076dc640 10 monclient: monmap: 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 librados: starting msgr at 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 librados: starting objecter 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 Processor -- start 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 -- start start 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 librados: setting wanted keys 2024-09-11T02:50:06.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 librados: calling monclient init 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: init 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _add_conns ranks=[5,6,8,7,0,2,1,4,3] 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f49000a37a0 0x7f49001cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): picked mon.noname-h con 0x7f49000a37a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49000a3b80 0x7f49001cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): picked mon.noname-d con 0x7f49000a3b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f490013c9c0 0x7f49001cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): picked mon.noname-e con 0x7f490013c9c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f4900141ea0 con 0x7f49000a37a0 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4900141d20 con 0x7f49000a3b80 2024-09-11T02:50:06.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4900142020 con 0x7f490013c9c0 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient(hunting): authenticate will time out at 960.012745s 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f490013c9c0 0x7f49001cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49000a3b80 0x7f49001cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f490013c9c0 0x7f49001cf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:41986/0 (socket says 172.21.15.98:41986) 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 -- 172.21.15.98:0/3206032353 learned_addr learned my addr 172.21.15.98:0/3206032353 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): get_auth_request con 0x7f490013c9c0 auth_method 0 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 10 monclient(hunting): get_auth_request con 0x7f49000a3b80 auth_method 0 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905478640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f49000a37a0 0x7f49001cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 -- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49000a3b80 msgr2=0x7f49001cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49000a3b80 0x7f49001cf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 -- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f49000a37a0 msgr2=0x7f49001cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f49000a37a0 0x7f49001cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 monclient: found mon.noname-e 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 -- 172.21.15.98:0/3206032353 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4900124190 con 0x7f490013c9c0 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4904c77640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49000a3b80 0x7f49001cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:06.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905c79640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f490013c9c0 0x7f49001cf9b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48f80b0040 tx=0x7f48f8019680 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 client.?.objecter ms_handle_connect 0x7f490013c9c0 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3206032353 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48f80b2050 con 0x7f490013c9c0 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: dump: 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.147 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3206032353 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48f80b3040 con 0x7f490013c9c0 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 5 monclient: authenticate success, global_id 4238 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _renew_subs 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4900125000 con 0x7f490013c9c0 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3206032353 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48f8003e10 con 0x7f490013c9c0 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f37fe640 4 set_mon_vals no callback set 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 monclient: dump: 2024-09-11T02:50:06.149 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.150 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _renew_subs 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f490013efe0 con 0x7f490013c9c0 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 librados: init done 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f490009e100 con 0x7f490013c9c0 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3206032353 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f48f80039a0 con 0x7f490013c9c0 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f48e403df40 0x7f48e4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3206032353 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f48f80be020 con 0x7f490013c9c0 2024-09-11T02:50:06.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 10 client.4238.objecter ms_dispatch 0x7f4900000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 3 client.4238.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 3 client.4238.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f48f1ffb640 20 client.4238.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905478640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f48e403df40 0x7f48e4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.131+0000 7f4905478640 10 monclient: get_auth_request con 0x7f48e403df40 auth_method 0 2024-09-11T02:50:06.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.135+0000 7f48f1ffb640 10 client.4238.objecter ms_handle_connect 0x7f48e403df40 2024-09-11T02:50:06.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.135+0000 7f4905478640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f48e403df40 0x7f48e4040400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009790 tx=0x7f48f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.135+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3206032353 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f48f80c7060 con 0x7f490013c9c0 2024-09-11T02:50:06.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.135+0000 7f48f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.135+0000 7f48f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:06.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.295+0000 7f49076dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:06.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.295+0000 7f49076dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:06.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.295+0000 7f49076dc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.295+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4900122ca0 con 0x7f490013c9c0 2024-09-11T02:50:06.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.295+0000 7f48f1ffb640 1 -- 172.21.15.98:0/3206032353 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12401 (secure 0 0 0) 0x7f48f8016560 con 0x7f490013c9c0 2024-09-11T02:50:06.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.295+0000 7f48f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:06.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.295+0000 7f48f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:06.297 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:06.297 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T02:50:04.952025+0000","last_up_change":"2024-09-11T02:49:54.844374+0000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T02:49:55.787515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T02:50:01.768815+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6801","nonce":1456395826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6803","nonce":1456395826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6807","nonce":1456395826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6805","nonce":1456395826}]},"public_addr":"172.21.15.98:6801/1456395826","cluster_addr":"172.21.15.98:6803/1456395826","heartbeat_back_addr":"172.21.15.98:6807/1456395826","heartbeat_front_addr":"172.21.15.98:6805/1456395826","state":["exists","up"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6809","nonce":2313933591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6811","nonce":2313933591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6815","nonce":2313933591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6813","nonce":2313933591}]},"public_addr":"172.21.15.98:6809/2313933591","cluster_addr":"172.21.15.98:6811/2313933591","heartbeat_back_addr":"172.21.15.98:6815/2313933591","heartbeat_front_addr":"172.21.15.98:6813/2313933591","state":["exists","up"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6817","nonce":3987433753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3987433753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3987433753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3987433753}]},"public_addr":"172.21.15.98:6817/3987433753","cluster_addr":"172.21.15.98:6819/3987433753","heartbeat_back_addr":"172.21.15.98:6823/3987433753","heartbeat_front_addr":"172.21.15.98:6821/3987433753","state":["exists","up"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6801","nonce":373809828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6803","nonce":373809828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6807","nonce":373809828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6805","nonce":373809828}]},"public_addr":"172.21.15.190:6801/373809828","cluster_addr":"172.21.15.190:6803/373809828","heartbeat_back_addr":"172.21.15.190:6807/373809828","heartbeat_front_addr":"172.21.15.190:6805/373809828","state":["exists","up"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6817","nonce":1271446291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6819","nonce":1271446291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6823","nonce":1271446291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6821","nonce":1271446291}]},"public_addr":"172.21.15.190:6817/1271446291","cluster_addr":"172.21.15.190:6819/1271446291","heartbeat_back_addr":"172.21.15.190:6823/1271446291","heartbeat_front_addr":"172.21.15.190:6821/1271446291","state":["exists","up"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6809","nonce":1497028198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6811","nonce":1497028198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6815","nonce":1497028198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6813","nonce":1497028198}]},"public_addr":"172.21.15.190:6809/1497028198","cluster_addr":"172.21.15.190:6811/1497028198","heartbeat_back_addr":"172.21.15.190:6815/1497028198","heartbeat_front_addr":"172.21.15.190:6813/1497028198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.670036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.679318+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.766598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.015321+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.338711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.969084+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T02:50:06.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 10 librados: watch_flush enter 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 10 librados: watch_flush exit 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 20 client.4238.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 10 client.4238.objecter successfully canceled tick 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f48e403df40 msgr2=0x7f48e4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f48e403df40 0x7f48e4040400 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009790 tx=0x7f48f4008040 comp rx=0 tx=0).stop 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 10 monclient: shutdown 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f490013c9c0 msgr2=0x7f49001cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f490013c9c0 0x7f49001cf9b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48f80b0040 tx=0x7f48f8019680 comp rx=0 tx=0).stop 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 shutdown_connections 2024-09-11T02:50:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f48e403df40 0x7f48e4040400 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f490013c9c0 0x7f49001cf9b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49000a3b80 0x7f49001cf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 --2- 172.21.15.98:0/3206032353 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f49000a37a0 0x7f49001cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 >> 172.21.15.98:0/3206032353 conn(0x7f49000a4890 msgr2=0x7f4900073b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 shutdown_connections 2024-09-11T02:50:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 -- 172.21.15.98:0/3206032353 wait complete. 2024-09-11T02:50:06.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.299+0000 7f49076dc640 1 librados: shutdown 2024-09-11T02:50:06.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T02:50:06.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T02:50:06.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T02:50:06.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T02:50:06.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T02:50:06.316 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T02:50:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.427+0000 7f819cfaa640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.427+0000 7f819cfaa640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.427+0000 7f819cfaa640 10 monclient: monmap: 2024-09-11T02:50:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.432959+0000 2024-09-11T02:50:06.429 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.432959+0000 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.430 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 1 Processor -- start 2024-09-11T02:50:06.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 1 -- start start 2024-09-11T02:50:06.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient: init 2024-09-11T02:50:06.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient: _add_conns ranks=[0,6,2,8,9,3,5,4,7,10,1] 2024-09-11T02:50:06.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8198139b80 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:06.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f819806fe50 0x7f819813ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient(hunting): picked mon.noname-d con 0x7f819806fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8198072640 0x7f8198140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient(hunting): picked mon.noname-b con 0x7f8198072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8198071370 con 0x7f8198139b80 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f81980719c0 con 0x7f8198072640 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8198071c10 con 0x7f819806fe50 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f819cfaa640 10 monclient(hunting): authenticate will time out at 960.313261s 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8198072640 0x7f8198140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8198072640 0x7f8198140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60974/0 (socket says 172.21.15.98:60974) 2024-09-11T02:50:06.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 1 -- 172.21.15.98:0/2633961924 learned_addr learned my addr 172.21.15.98:0/2633961924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 10 monclient(hunting): get_auth_request con 0x7f8198072640 auth_method 0 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 1 --2- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f819806fe50 0x7f819813ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): get_auth_request con 0x7f819806fe50 auth_method 0 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8196d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-09-11T02:50:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 1 -- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8198072640 msgr2=0x7f8198140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 1 --2- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8198072640 0x7f8198140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 1 -- 172.21.15.98:0/2633961924 >> v1:172.21.15.98:6789/0 conn(0x7f8198139b80 legacy=0x7f8198139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 1 monclient: found mon.noname-d 2024-09-11T02:50:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 1 -- 172.21.15.98:0/2633961924 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81980a9600 con 0x7f819806fe50 2024-09-11T02:50:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.431+0000 7f8195d9b640 1 --2- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f819806fe50 0x7f819813ea00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f818c0097f0 tx=0x7f818c01c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=70c7e3aee34f6f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 1 -- 172.21.15.98:0/2633961924 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f818c0b0070 con 0x7f819806fe50 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: dump: 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.438 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient: monmap: 2024-09-11T02:50:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.437500+0000 2024-09-11T02:50:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.437500+0000 2024-09-11T02:50:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.442 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 5 monclient: authenticate success, global_id 4272 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 1 -- 172.21.15.98:0/2633961924 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f818c0aa3a0 con 0x7f819806fe50 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 1 -- 172.21.15.98:0/2633961924 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f818c0a4e60 con 0x7f819806fe50 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819559a640 10 monclient: dump: 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 4 set_mon_vals no callback set 2024-09-11T02:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 1 Processor -- start 2024-09-11T02:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 1 -- start start 2024-09-11T02:50:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient: init 2024-09-11T02:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 10 monclient: shutdown 2024-09-11T02:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 -- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f819806fe50 msgr2=0x7f819813ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 --2- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f819806fe50 0x7f819813ea00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f818c0097f0 tx=0x7f818c01c3a0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient: _add_conns ranks=[1,7,8,6,10,9,2,3,4,0,5] 2024-09-11T02:50:06.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 -- 172.21.15.98:0/2633961924 shutdown_connections 2024-09-11T02:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fde08136650 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 --2- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8198072640 0x7f8198140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 --2- 172.21.15.98:0/2633961924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f819806fe50 0x7f819813ea00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde08070f40 0x7fde0813b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 -- 172.21.15.98:0/2633961924 >> 172.21.15.98:0/2633961924 conn(0x7f81980ac730 msgr2=0x7f81980a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient(hunting): picked mon.noname-i con 0x7fde08070f40 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 -- 172.21.15.98:0/2633961924 shutdown_connections 2024-09-11T02:50:06.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde08073730 0x7fde0813d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 -- 172.21.15.98:0/2633961924 wait complete. 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient(hunting): picked mon.noname-e con 0x7fde08073730 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde08073730 0x7fde0813d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde08073730 0x7fde0813d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:42002/0 (socket says 172.21.15.98:42002) 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 -- 172.21.15.98:0/2000056627 learned_addr learned my addr 172.21.15.98:0/2000056627 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde08072460 con 0x7fde08136650 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 1 -- 172.21.15.98:0/2000056627 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fde08072ab0 con 0x7fde08070f40 2024-09-11T02:50:06.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 1 -- 172.21.15.98:0/2000056627 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fde08072d00 con 0x7fde08073730 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde07fff640 10 monclient(hunting): authenticate will time out at 960.318067s 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): get_auth_request con 0x7fde08073730 auth_method 0 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 1 --2- 172.21.15.98:0/2000056627 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde08070f40 0x7fde0813b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 10 monclient(hunting): get_auth_request con 0x7fde08070f40 auth_method 0 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 10 monclient: monmap: 2024-09-11T02:50:06.453 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.453 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.454 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 librados: starting msgr at 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 librados: starting objecter 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 Processor -- start 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 -- start start 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 librados: setting wanted keys 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 librados: calling monclient init 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 10 monclient: init 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 10 monclient: _add_conns ranks=[2,3,5,4,8,7,0,1,6] 2024-09-11T02:50:06.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7f819cfaa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f819806fe50 0x7f819813ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 -- 172.21.15.98:0/2000056627 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde08070f40 msgr2=0x7fde0813b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 --2- 172.21.15.98:0/2000056627 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde08070f40 0x7fde0813b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 -- 172.21.15.98:0/2000056627 >> v1:172.21.15.190:6789/0 conn(0x7fde08136650 legacy=0x7fde08136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 monclient: found mon.noname-e 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.435+0000 7fde077fe640 1 -- 172.21.15.98:0/2000056627 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0806fe00 con 0x7fde08073730 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): picked mon.noname-b con 0x7f819806fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8198072640 0x7f819813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): picked mon.noname-g con 0x7f8198072640 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8198139b80 0x7f81981d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): picked mon.noname-h con 0x7f8198139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8198143ec0 con 0x7f819806fe50 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f81980702d0 con 0x7f8198072640 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8198144040 con 0x7f8198139b80 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient(hunting): authenticate will time out at 960.320138s 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f819806fe50 0x7f819813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f819806fe50 0x7f819813ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60980/0 (socket says 172.21.15.98:60980) 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 -- 172.21.15.98:0/1979574226 learned_addr learned my addr 172.21.15.98:0/1979574226 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): get_auth_request con 0x7f819806fe50 auth_method 0 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8198072640 0x7f819813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient(hunting): get_auth_request con 0x7f8198072640 auth_method 0 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde077fe640 1 --2- 172.21.15.98:0/2000056627 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde08073730 0x7fde0813d920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde00009eb0 tx=0x7fde0001c980 comp rx=0 tx=0).ready entity=mon.8 client_cookie=8a43690ff641e7fe server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8196d9d640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8198139b80 0x7f81981d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 -- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8198139b80 msgr2=0x7f81981d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8198139b80 0x7f81981d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 -- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8198072640 msgr2=0x7f819813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8198072640 0x7f819813e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 monclient: found mon.noname-b 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 -- 172.21.15.98:0/1979574226 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81980a6a00 con 0x7f819806fe50 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819659c640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f819806fe50 0x7f819813ffd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f818800b410 tx=0x7f818800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 client.?.objecter ms_handle_connect 0x7f819806fe50 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 1 -- 172.21.15.98:0/2000056627 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde000090d0 con 0x7fde08073730 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: dump: 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 5 monclient: authenticate success, global_id 4247 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 1 -- 172.21.15.98:0/2000056627 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde000a5b80 con 0x7fde08073730 2024-09-11T02:50:06.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 1 -- 172.21.15.98:0/2000056627 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde000a5d70 con 0x7fde08073730 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 4 set_mon_vals no callback set 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde05ffb640 10 monclient: dump: 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 1 -- 172.21.15.98:0/1979574226 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8188015020 con 0x7f819806fe50 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: dump: 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.466 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 1 -- 172.21.15.98:0/1979574226 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8188011aa0 con 0x7f819806fe50 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 1 -- 172.21.15.98:0/1979574226 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8188011cd0 con 0x7f819806fe50 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 monclient: dump: 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.468 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8194d99640 4 set_mon_vals no callback set 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 5 monclient: authenticate success, global_id 4223 2024-09-11T02:50:06.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient: _renew_subs 2024-09-11T02:50:06.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f819813f470 con 0x7f819806fe50 2024-09-11T02:50:06.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient: _renew_subs 2024-09-11T02:50:06.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8198140730 con 0x7f819806fe50 2024-09-11T02:50:06.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 librados: init done 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 10 monclient: shutdown 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 -- 172.21.15.98:0/2000056627 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde08073730 msgr2=0x7fde0813d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 --2- 172.21.15.98:0/2000056627 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde08073730 0x7fde0813d920 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fde00009eb0 tx=0x7fde0001c980 comp rx=0 tx=0).stop 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 -- 172.21.15.98:0/2000056627 shutdown_connections 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 --2- 172.21.15.98:0/2000056627 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fde08073730 0x7fde0813d920 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 --2- 172.21.15.98:0/2000056627 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fde08070f40 0x7fde0813b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 -- 172.21.15.98:0/2000056627 >> 172.21.15.98:0/2000056627 conn(0x7fde080a61e0 msgr2=0x7fde0806ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 -- 172.21.15.98:0/2000056627 shutdown_connections 2024-09-11T02:50:06.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 -- 172.21.15.98:0/2000056627 wait complete. 2024-09-11T02:50:06.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 1 -- 172.21.15.98:0/1979574226 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8188011630 con 0x7f819806fe50 2024-09-11T02:50:06.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f817803df40 0x7f8178040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 1 -- 172.21.15.98:0/1979574226 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f818804ef50 con 0x7f819806fe50 2024-09-11T02:50:06.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 10 client.4223.objecter ms_dispatch 0x7f8198000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 3 client.4223.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 3 client.4223.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8186ffd640 20 client.4223.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 10 monclient: monmap: 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.473 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 librados: starting msgr at 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7fde07fff640 1 librados: starting objecter 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f817803df40 0x7f8178040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f8195d9b640 10 monclient: get_auth_request con 0x7f817803df40 auth_method 0 2024-09-11T02:50:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 10 client.4223.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 15 client.4223.objecter _session_command_op_assign -1 1 2024-09-11T02:50:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.439+0000 7f819cfaa640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f819813dec0 0x7f819802d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 Processor -- start 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 20 client.4223.objecter _get_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 20 client.4223.objecter put_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 20 client.4223.objecter _get_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 15 client.4223.objecter _session_command_op_remove -1 1 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 20 client.4223.objecter get_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 15 client.4223.objecter _session_command_op_assign 2 1 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 20 client.4223.objecter put_session s=0x7f81980a6e30 osd=2 4 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 10 client.4223.objecter _send_command 1 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 --> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8198123220 con 0x7f819813dec0 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8196d9d640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f819813dec0 0x7f819802d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8196d9d640 10 monclient: get_auth_request con 0x7f819813dec0 auth_method 0 2024-09-11T02:50:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 -- start start 2024-09-11T02:50:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 librados: setting wanted keys 2024-09-11T02:50:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 librados: calling monclient init 2024-09-11T02:50:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient: init 2024-09-11T02:50:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8196d9d640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f819813dec0 0x7f819802d960 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8186ffd640 10 client.4223.objecter ms_handle_connect 0x7f819813dec0 2024-09-11T02:50:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8186ffd640 10 client.4223.objecter ms_handle_connect 0x7f817803df40 2024-09-11T02:50:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8195d9b640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f817803df40 0x7f8178040400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ad000 tx=0x7f818c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient: _add_conns ranks=[1,6,2,5,3,7,8,4,0] 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde08070f40 0x7fde081ceb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): picked mon.noname-f con 0x7fde08070f40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fde08073730 0x7fde08133160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): picked mon.noname-d con 0x7fde08073730 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde08136650 0x7fde081336a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fde08073730 0x7fde08133160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fde08073730 0x7fde08133160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57114/0 (socket says 172.21.15.98:57114) 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde08136650 0x7fde081336a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde08136650 0x7fde081336a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60996/0 (socket says 172.21.15.98:60996) 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde08070f40 0x7fde081ceb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): picked mon.noname-b con 0x7fde08136650 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 1 -- 172.21.15.98:0/1687558004 learned_addr learned my addr 172.21.15.98:0/1687558004 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fde08140a10 con 0x7fde08070f40 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fde08140890 con 0x7fde08136650 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fde08140b90 con 0x7fde08073730 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde07fff640 10 monclient(hunting): authenticate will time out at 960.325664s 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 10 monclient(hunting): get_auth_request con 0x7fde08070f40 auth_method 0 2024-09-11T02:50:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): get_auth_request con 0x7fde08136650 auth_method 0 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 10 monclient(hunting): get_auth_request con 0x7fde08073730 auth_method 0 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8186ffd640 1 -- 172.21.15.98:0/1979574226 <== osd.2 v2:172.21.15.98:6816/3987433753 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81900091b0 con 0x7f819813dec0 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8186ffd640 10 client.4223.objecter ms_dispatch 0x7f8198000ba0 command_reply(tid 1: 0 ) 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8186ffd640 10 client.4223.objecter _finish_command 1 = system:0 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8186ffd640 20 client.4223.objecter put_session s=0x7f81980a6e30 osd=2 4 2024-09-11T02:50:06.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7f8186ffd640 15 client.4223.objecter _session_command_op_remove 2 1 2024-09-11T02:50:06.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 -- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fde08073730 msgr2=0x7fde08133160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fde08073730 0x7fde08133160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 -- 172.21.15.98:0/1687558004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde08070f40 msgr2=0x7fde081ceb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde08070f40 0x7fde081ceb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 monclient: found mon.noname-b 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde080a8540 con 0x7fde08136650 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.443+0000 7fde077fe640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde08136650 0x7fde081336a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde000ab000 tx=0x7fde00019720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.?.objecter ms_handle_connect 0x7fde08136650 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde000bc020 con 0x7fde08136650 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: dump: 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde00003bb0 con 0x7fde08136650 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 5 monclient: authenticate success, global_id 4232 2024-09-11T02:50:06.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 monclient: _renew_subs 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0813ca70 con 0x7fde08136650 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde00003d60 con 0x7fde08136650 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde057fa640 4 set_mon_vals no callback set 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: dump: 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 monclient: _renew_subs 2024-09-11T02:50:06.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde08133f70 con 0x7fde08136650 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 1 librados: init done 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 client.4232.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 15 client.4232.objecter _session_command_op_assign -1 1 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 20 client.4232.objecter _get_session osd=-1 returning homeless 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fde08075070 con 0x7fde08136650 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fde000c9060 con 0x7fde08136650 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fddf403df40 0x7fddf4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fde000eaae0 con 0x7fde08136650 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.4232.objecter ms_dispatch 0x7fde08000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 3 client.4232.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 3 client.4232.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.4232.objecter checking command 1 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7fddf4043db0 0x7fddf40461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 20 client.4232.objecter _get_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 20 client.4232.objecter put_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 15 client.4232.objecter _session_command_op_remove -1 1 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 20 client.4232.objecter _get_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 20 client.4232.objecter get_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 15 client.4232.objecter _session_command_op_assign 0 1 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 20 client.4232.objecter put_session s=0x7fddf40436e0 osd=0 4 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.4232.objecter _send_command 1 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde000bee70 con 0x7fddf4043db0 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 20 client.4232.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fde000eae10 con 0x7fde08136650 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde06ffd640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fddf403df40 0x7fddf4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde06ffd640 10 monclient: get_auth_request con 0x7fddf403df40 auth_method 0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde067fc640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7fddf4043db0 0x7fddf40461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde067fc640 10 monclient: get_auth_request con 0x7fddf4043db0 auth_method 0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.4232.objecter ms_handle_connect 0x7fddf4043db0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde067fc640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7fddf4043db0 0x7fddf40461f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.4232.objecter ms_handle_connect 0x7fddf403df40 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fde06ffd640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fddf403df40 0x7fddf4040400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fddf8007fd0 tx=0x7fddf8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== osd.0 v2:172.21.15.98:6800/1456395826 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde000bee70 con 0x7fddf4043db0 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.4232.objecter ms_dispatch 0x7fde08000ba0 command_reply(tid 1: 0 ) 2024-09-11T02:50:06.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.451+0000 7fddef7fe640 10 client.4232.objecter _finish_command 1 = system:0 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.455+0000 7fddef7fe640 20 client.4232.objecter put_session s=0x7fddf40436e0 osd=0 4 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.455+0000 7fddef7fe640 15 client.4232.objecter _session_command_op_remove 0 1 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 10 client.4223.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 15 client.4223.objecter _session_command_op_assign -1 2 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 client.4223.objecter _get_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 client.4223.objecter put_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 client.4223.objecter _get_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 15 client.4223.objecter _session_command_op_remove -1 2 2024-09-11T02:50:06.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 client.4223.objecter get_session s=0x7f81980a6e30 osd=2 3 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 15 client.4223.objecter _session_command_op_assign 2 2 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 client.4223.objecter put_session s=0x7f81980a6e30 osd=2 4 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 10 client.4223.objecter _send_command 2 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 --> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f81980a6ff0 con 0x7f819813dec0 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f8186ffd640 1 -- 172.21.15.98:0/1979574226 <== osd.2 v2:172.21.15.98:6816/3987433753 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f81980a6ff0 con 0x7f819813dec0 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f8186ffd640 10 client.4223.objecter ms_dispatch 0x7f8198000ba0 command_reply(tid 2: 0 ) 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f8186ffd640 10 client.4223.objecter _finish_command 2 = system:0 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f8186ffd640 20 client.4223.objecter put_session s=0x7f81980a6e30 osd=2 4 2024-09-11T02:50:06.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f8186ffd640 15 client.4223.objecter _session_command_op_remove 2 2 2024-09-11T02:50:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 10 librados: watch_flush enter 2024-09-11T02:50:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 10 librados: watch_flush exit 2024-09-11T02:50:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 10 client.4223.objecter close_session for osd.2 2024-09-11T02:50:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f819813dec0 msgr2=0x7f819802d960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f819813dec0 0x7f819802d960 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 client.4223.objecter put_session s=0x7f81980a6e30 osd=2 1 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 client.4223.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 10 client.4223.objecter successfully canceled tick 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f817803df40 msgr2=0x7f8178040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f817803df40 0x7f8178040400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ad000 tx=0x7f818c0a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 10 monclient: shutdown 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f819806fe50 msgr2=0x7f819813ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 --2- 172.21.15.98:0/1979574226 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f819806fe50 0x7f819813ffd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f818800b410 tx=0x7f818800b8d0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f8196d9d640 1 -- 172.21.15.98:0/1979574226 reap_dead start 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 shutdown_connections 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.459+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 >> 172.21.15.98:0/1979574226 conn(0x7f81980ac730 msgr2=0x7f81980a72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.463+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 shutdown_connections 2024-09-11T02:50:06.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.463+0000 7f819cfaa640 1 -- 172.21.15.98:0/1979574226 wait complete. 2024-09-11T02:50:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.463+0000 7f819cfaa640 1 librados: shutdown 2024-09-11T02:50:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 10 client.4232.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T02:50:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 15 client.4232.objecter _session_command_op_assign -1 2 2024-09-11T02:50:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 client.4232.objecter _get_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 client.4232.objecter put_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 client.4232.objecter _get_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 15 client.4232.objecter _session_command_op_remove -1 2 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 client.4232.objecter get_session s=0x7fddf40436e0 osd=0 3 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 15 client.4232.objecter _session_command_op_assign 0 2 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 client.4232.objecter put_session s=0x7fddf40436e0 osd=0 4 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 10 client.4232.objecter _send_command 2 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 --> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fde08136a30 con 0x7fddf4043db0 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fddef7fe640 1 -- 172.21.15.98:0/1687558004 <== osd.0 v2:172.21.15.98:6800/1456395826 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fde08136a30 con 0x7fddf4043db0 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fddef7fe640 10 client.4232.objecter ms_dispatch 0x7fde08000ba0 command_reply(tid 2: 0 ) 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fddef7fe640 10 client.4232.objecter _finish_command 2 = system:0 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fddef7fe640 20 client.4232.objecter put_session s=0x7fddf40436e0 osd=0 4 2024-09-11T02:50:06.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fddef7fe640 15 client.4232.objecter _session_command_op_remove 0 2 2024-09-11T02:50:06.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 10 librados: watch_flush enter 2024-09-11T02:50:06.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 10 librados: watch_flush exit 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 10 client.4232.objecter close_session for osd.0 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7fddf4043db0 msgr2=0x7fddf40461f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7fddf4043db0 0x7fddf40461f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 client.4232.objecter put_session s=0x7fddf40436e0 osd=0 1 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 client.4232.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 10 client.4232.objecter successfully canceled tick 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fddf403df40 msgr2=0x7fddf4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fddf403df40 0x7fddf4040400 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fddf8007fd0 tx=0x7fddf8009040 comp rx=0 tx=0).stop 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 10 monclient: shutdown 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde08136650 msgr2=0x7fde081336a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 --2- 172.21.15.98:0/1687558004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fde08136650 0x7fde081336a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde000ab000 tx=0x7fde00019720 comp rx=0 tx=0).stop 2024-09-11T02:50:06.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde077fe640 1 -- 172.21.15.98:0/1687558004 reap_dead start 2024-09-11T02:50:06.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 shutdown_connections 2024-09-11T02:50:06.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 >> 172.21.15.98:0/1687558004 conn(0x7fde080a61e0 msgr2=0x7fde080a3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 shutdown_connections 2024-09-11T02:50:06.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.475+0000 7fde07fff640 1 -- 172.21.15.98:0/1687558004 wait complete. 2024-09-11T02:50:06.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.479+0000 7fde07fff640 1 librados: shutdown 2024-09-11T02:50:06.503 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-09-11T02:50:06.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.483+0000 7f5e11656640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.483+0000 7f5e11656640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.483+0000 7f5e11656640 10 monclient: monmap: 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.488780+0000 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.488780+0000 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.505 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 1 Processor -- start 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 1 -- start start 2024-09-11T02:50:06.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient: init 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient: monmap: 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.489920+0000 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.489920+0000 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.508 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient: _add_conns ranks=[0,5,8,6,3,10,7,9,4,1,2] 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e0c0a46a0 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5e0c0a5190 0x7f5e0c13ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient(hunting): picked mon.noname-h con 0x7f5e0c0a5190 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c141510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient(hunting): picked mon.noname-e con 0x7f5e0c13f120 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0c0a17b0 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5e0c0a0750 con 0x7f5e0c0a5190 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5e0c0775d0 con 0x7f5e0c13f120 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e11656640 10 monclient(hunting): authenticate will time out at 960.369173s 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5e0c0a46a0 0x7f5e0c0a4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53908/0 (socket says 172.21.15.98:53908) 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e0c13f120 auth_method 0 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0affd640 1 -- 172.21.15.98:0/132135517 learned_addr learned my addr 172.21.15.98:0/132135517 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 1 Processor -- start 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 1 -- start start 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient: init 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0a7fc640 1 --2- 172.21.15.98:0/132135517 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5e0c0a5190 0x7f5e0c13ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e0c0a5190 auth_method 0 2024-09-11T02:50:06.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 1 -- 172.21.15.98:0/132135517 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5e0c0a5190 msgr2=0x7f5e0c13ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 1 --2- 172.21.15.98:0/132135517 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5e0c0a5190 0x7f5e0c13ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 1 -- 172.21.15.98:0/132135517 >> v1:172.21.15.98:6789/0 conn(0x7f5e0c0a46a0 legacy=0x7f5e0c0a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 1 monclient: found mon.noname-e 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7f5e0b7fe640 1 -- 172.21.15.98:0/132135517 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0c0a0200 con 0x7f5e0c13f120 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient: _add_conns ranks=[1,4,10,0,8,6,2,9,5,3,7] 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd87009e870 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd870068e60 0x7fd87009bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient(hunting): picked mon.noname-c con 0x7fd870068e60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd87009ec50 0x7fd8700a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient(hunting): picked mon.noname-f con 0x7fd87009ec50 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd87009d430 con 0x7fd87009e870 2024-09-11T02:50:06.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd87009d710 con 0x7fd87009ec50 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd87009d9f0 con 0x7fd870068e60 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd8768eb640 10 monclient(hunting): authenticate will time out at 960.370530s 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd870068e60 0x7fd87009bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd870068e60 0x7fd87009bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55454/0 (socket says 172.21.15.98:55454) 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 -- 172.21.15.98:0/180081118 learned_addr learned my addr 172.21.15.98:0/180081118 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): get_auth_request con 0x7fd870068e60 auth_method 0 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 1 --2- 172.21.15.98:0/180081118 >> v2:172.21.15.190:3300/0 conn(0x7fd87009ec50 0x7fd8700a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 10 monclient(hunting): get_auth_request con 0x7fd87009ec50 auth_method 0 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd874e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient(hunting): handle_auth_done global_id 4252 payload 274 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 -- 172.21.15.98:0/180081118 >> v2:172.21.15.190:3300/0 conn(0x7fd87009ec50 msgr2=0x7fd8700a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 --2- 172.21.15.98:0/180081118 >> v2:172.21.15.190:3300/0 conn(0x7fd87009ec50 0x7fd8700a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 -- 172.21.15.98:0/180081118 >> v1:172.21.15.190:6789/0 conn(0x7fd87009e870 legacy=0x7fd8700a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 monclient: found mon.noname-c 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.487+0000 7fd86f7fe640 1 -- 172.21.15.98:0/180081118 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87009c460 con 0x7fd870068e60 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient: monmap: 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.493901+0000 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.493901+0000 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.519 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.520 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.520 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e0b7fe640 1 --2- 172.21.15.98:0/132135517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c141510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009dc0 tx=0x7f5e0401c890 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3e9399b29e51ca3b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 1 -- 172.21.15.98:0/132135517 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5e040a88a0 con 0x7f5e0c13f120 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: dump: 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.521 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 1 -- 172.21.15.98:0/132135517 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e040a8a60 con 0x7f5e0c13f120 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 5 monclient: authenticate success, global_id 4256 2024-09-11T02:50:06.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 4 set_mon_vals no callback set 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 1 -- 172.21.15.98:0/132135517 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5e040a8c50 con 0x7f5e0c13f120 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e09ffb640 10 monclient: dump: 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.523 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 1 Processor -- start 2024-09-11T02:50:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 1 -- start start 2024-09-11T02:50:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient: init 2024-09-11T02:50:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 10 monclient: shutdown 2024-09-11T02:50:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 -- 172.21.15.98:0/132135517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 msgr2=0x7f5e0c141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 --2- 172.21.15.98:0/132135517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c141510 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009dc0 tx=0x7f5e0401c890 comp rx=0 tx=0).stop 2024-09-11T02:50:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 -- 172.21.15.98:0/132135517 shutdown_connections 2024-09-11T02:50:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 --2- 172.21.15.98:0/132135517 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5e0c0a5190 0x7f5e0c13ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 --2- 172.21.15.98:0/132135517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c141510 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 -- 172.21.15.98:0/132135517 >> 172.21.15.98:0/132135517 conn(0x7f5e0c06ed00 msgr2=0x7f5e0c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient: _add_conns ranks=[10,0,9,2,8,7,3,5,1,6,4] 2024-09-11T02:50:06.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f49d00a46a0 0x7f49d00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient(hunting): picked mon.noname-f con 0x7f49d00a46a0 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49d00a4fc0 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49d013a040 0x7f49d0141c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient(hunting): picked mon.noname-a con 0x7f49d013a040 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d00740c0 con 0x7f49d00a4fc0 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f49d00a1540 con 0x7f49d013a040 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f49d00a1790 con 0x7f49d00a46a0 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d6f00640 10 monclient(hunting): authenticate will time out at 960.373995s 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49cffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49d00a4fc0 0x7f49d0139930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53914/0 (socket says 172.21.15.98:53914) 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49cffff640 1 -- 172.21.15.98:0/1461847782 learned_addr learned my addr 172.21.15.98:0/1461847782 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 1 --2- 172.21.15.98:0/1461847782 >> v2:172.21.15.98:3300/0 conn(0x7f49d013a040 0x7f49d0141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 10 monclient(hunting): get_auth_request con 0x7f49d013a040 auth_method 0 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 1 --2- 172.21.15.98:0/1461847782 >> v2:172.21.15.190:3300/0 conn(0x7f49d00a46a0 0x7f49d00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 -- 172.21.15.98:0/132135517 shutdown_connections 2024-09-11T02:50:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86f7fe640 1 --2- 172.21.15.98:0/180081118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd870068e60 0x7fd87009bbe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd868009e80 tx=0x7fd86801c960 comp rx=0 tx=0).ready entity=mon.4 client_cookie=9dda47b453cc8523 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f5e11656640 1 -- 172.21.15.98:0/132135517 wait complete. 2024-09-11T02:50:06.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): get_auth_request con 0x7f49d00a46a0 auth_method 0 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 1 -- 172.21.15.98:0/180081118 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8680090d0 con 0x7fd870068e60 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 10 monclient: dump: 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 5 monclient: authenticate success, global_id 4252 2024-09-11T02:50:06.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 1 -- 172.21.15.98:0/1461847782 >> v2:172.21.15.98:3300/0 conn(0x7f49d013a040 msgr2=0x7f49d0141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 1 --2- 172.21.15.98:0/1461847782 >> v2:172.21.15.98:3300/0 conn(0x7f49d013a040 0x7f49d0141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 1 -- 172.21.15.98:0/1461847782 >> v1:172.21.15.98:6789/0 conn(0x7f49d00a4fc0 legacy=0x7f49d0139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 1 monclient: found mon.noname-f 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d4c9c640 1 -- 172.21.15.98:0/1461847782 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d009fe60 con 0x7f49d00a46a0 2024-09-11T02:50:06.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7f49d549d640 1 --2- 172.21.15.98:0/1461847782 >> v2:172.21.15.98:3300/0 conn(0x7f49d013a040 0x7f49d0141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:06.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 1 -- 172.21.15.98:0/180081118 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8700a4c40 con 0x7fd870068e60 2024-09-11T02:50:06.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 4 set_mon_vals no callback set 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 10 monclient: shutdown 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 1 -- 172.21.15.98:0/180081118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd870068e60 msgr2=0x7fd87009bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 1 -- 172.21.15.98:0/180081118 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8680090d0 con 0x7fd870068e60 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 1 --2- 172.21.15.98:0/180081118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd870068e60 0x7fd87009bbe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd868009e80 tx=0x7fd86801c960 comp rx=0 tx=0).stop 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd86effd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 1 -- 172.21.15.98:0/180081118 shutdown_connections 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 1 --2- 172.21.15.98:0/180081118 >> v2:172.21.15.190:3300/0 conn(0x7fd87009ec50 0x7fd8700a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 1 --2- 172.21.15.98:0/180081118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd870068e60 0x7fd87009bbe0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.491+0000 7fd8768eb640 1 -- 172.21.15.98:0/180081118 >> 172.21.15.98:0/180081118 conn(0x7fd8700a4ed0 msgr2=0x7fd8700a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 -- 172.21.15.98:0/180081118 shutdown_connections 2024-09-11T02:50:06.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 -- 172.21.15.98:0/180081118 wait complete. 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient: monmap: 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.497555+0000 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.497555+0000 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.537 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 Processor -- start 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient: monmap: 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.539 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 -- start start 2024-09-11T02:50:06.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient: init 2024-09-11T02:50:06.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 librados: starting msgr at 2024-09-11T02:50:06.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 librados: starting objecter 2024-09-11T02:50:06.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 Processor -- start 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient: _add_conns ranks=[5,7,9,3,8,0,2,10,4,6,1] 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f69bc076b00 0x7f69bc0a4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 -- start start 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 librados: setting wanted keys 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 librados: calling monclient init 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient: init 2024-09-11T02:50:06.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): picked mon.noname-h con 0x7f69bc076b00 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f69bc076ee0 0x7f69bc0772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): picked mon.noname-i con 0x7f69bc076ee0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69bc137360 0x7f69bc137780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d4c9c640 1 --2- 172.21.15.98:0/1461847782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d00a46a0 0x7f49d00a4a80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f49d00a2170 tx=0x7f49c401c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32f2db3b80638942 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f69bc137360 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f69bc0a21f0 con 0x7f69bc137360 2024-09-11T02:50:06.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f69bc0a2590 con 0x7f69bc076b00 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f69bc073450 con 0x7f69bc076ee0 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bb59e640 10 monclient(hunting): authenticate will time out at 960.377765s 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69bc137360 0x7f69bc137780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f69bc137360 0x7f69bc137780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38960/0 (socket says 172.21.15.98:38960) 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 -- 172.21.15.98:0/1548524357 learned_addr learned my addr 172.21.15.98:0/1548524357 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): get_auth_request con 0x7f69bc137360 auth_method 0 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f69bc076ee0 0x7f69bc0772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 10 monclient(hunting): get_auth_request con 0x7f69bc076ee0 auth_method 0 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient: monmap: 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.545 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.546 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 1 -- 172.21.15.98:0/1461847782 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49c40b0070 con 0x7f49d00a46a0 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: dump: 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.547 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69ba59c640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f69bc076b00 0x7f69bc0a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 -- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f69bc076ee0 msgr2=0x7f69bc0772c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f69bc076ee0 0x7f69bc0772c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 -- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f69bc076b00 msgr2=0x7f69bc0a4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f69bc076b00 0x7f69bc0a4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 monclient: found mon.noname-a 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69bad9d640 1 -- 172.21.15.98:0/1548524357 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69bc073730 con 0x7f69bc137360 2024-09-11T02:50:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 librados: starting msgr at 2024-09-11T02:50:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 5 monclient: authenticate success, global_id 4303 2024-09-11T02:50:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 1 -- 172.21.15.98:0/1461847782 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c40aa3a0 con 0x7f49d00a46a0 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 1 -- 172.21.15.98:0/1461847782 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49c40aaa40 con 0x7f49d00a46a0 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49cf7fe640 10 monclient: dump: 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 4 set_mon_vals no callback set 2024-09-11T02:50:06.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 librados: starting objecter 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient: _add_conns ranks=[6,4,7,0,8,2,5,3,1] 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd87009e870 0x7fd870072650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): picked mon.noname-d con 0x7fd87009e870 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd87009ec50 0x7fd870072b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): picked mon.noname-c con 0x7fd87009ec50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69ba59c640 10 monclient: get_auth_request con 0x7f69bc076b00 auth_method 0 2024-09-11T02:50:06.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f69ba59c640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f69bc076b00 0x7f69bc0a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 Processor -- start 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd8700730d0 0x7fd8701d1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): picked mon.noname-i con 0x7fd8700730d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd870144150 con 0x7fd87009ec50 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd870143fd0 con 0x7fd87009e870 2024-09-11T02:50:06.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fd8701442d0 con 0x7fd8700730d0 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd8768eb640 10 monclient(hunting): authenticate will time out at 960.378277s 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd87009e870 0x7fd870072650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd87009e870 0x7fd870072650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57120/0 (socket says 172.21.15.98:57120) 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 -- 172.21.15.98:0/2978954752 learned_addr learned my addr 172.21.15.98:0/2978954752 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): get_auth_request con 0x7fd87009e870 auth_method 0 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 -- start start 2024-09-11T02:50:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 librados: setting wanted keys 2024-09-11T02:50:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 librados: calling monclient init 2024-09-11T02:50:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient: init 2024-09-11T02:50:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd87009ec50 0x7fd870072b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 10 monclient(hunting): get_auth_request con 0x7fd87009ec50 auth_method 0 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 -- 172.21.15.98:0/2978954752 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd8700730d0 msgr2=0x7fd8701d1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd8700730d0 0x7fd8701d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 -- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd87009ec50 msgr2=0x7fd870072b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd87009ec50 0x7fd870072b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 monclient: found mon.noname-d 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 -- 172.21.15.98:0/2978954752 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd870093ce0 con 0x7fd87009e870 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 10 monclient: shutdown 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 -- 172.21.15.98:0/1461847782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d00a46a0 msgr2=0x7f49d00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 --2- 172.21.15.98:0/1461847782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d00a46a0 0x7f49d00a4a80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f49d00a2170 tx=0x7f49c401c340 comp rx=0 tx=0).stop 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86ffff640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd87009e870 0x7fd870072650 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c4d0 tx=0x7fd85800c990 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86cff9640 10 client.?.objecter ms_handle_connect 0x7fd87009e870 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient: _add_conns ranks=[0,1,8,5,6,4,2,7,3] 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e0c0a46a0 0x7f5e0c1d15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0c0a46a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e0c0a5190 0x7f5e0c1d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): picked mon.noname-f con 0x7f5e0c0a5190 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c13a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): picked mon.noname-e con 0x7f5e0c13f120 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c144270 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c077a60 con 0x7f5e0c0a5190 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e0c0a46a0 0x7f5e0c1d15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5e0c1443f0 con 0x7f5e0c13f120 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e0c0a46a0 0x7f5e0c1d15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38974/0 (socket says 172.21.15.98:38974) 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 1 -- 172.21.15.98:0/2490040146 learned_addr learned my addr 172.21.15.98:0/2490040146 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c13a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e11656640 10 monclient(hunting): authenticate will time out at 960.379608s 2024-09-11T02:50:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 -- 172.21.15.98:0/1461847782 shutdown_connections 2024-09-11T02:50:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 --2- 172.21.15.98:0/1461847782 >> v2:172.21.15.98:3300/0 conn(0x7f49d013a040 0x7f49d0141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 --2- 172.21.15.98:0/1461847782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d00a46a0 0x7f49d00a4a80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 -- 172.21.15.98:0/1461847782 >> 172.21.15.98:0/1461847782 conn(0x7f49d006ed00 msgr2=0x7f49d006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 -- 172.21.15.98:0/1461847782 shutdown_connections 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 10 monclient(hunting): get_auth_request con 0x7f5e0c0a46a0 auth_method 0 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e0c0a5190 0x7f5e0c1d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e0c13f120 auth_method 0 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 10 monclient(hunting): get_auth_request con 0x7f5e0c0a5190 auth_method 0 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd858012630 con 0x7fd87009e870 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86cff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:06.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7fd86cff9640 10 monclient: dump: 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.564 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f5e0a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8580127f0 con 0x7fd87009e870 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd858012e10 con 0x7fd87009e870 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 monclient: dump: 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.566 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86e7fc640 4 set_mon_vals no callback set 2024-09-11T02:50:06.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-11T02:50:06.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 1 -- 172.21.15.98:0/2490040146 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 msgr2=0x7f5e0c13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e0c13f120 0x7f5e0c13a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 1 -- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e0c0a5190 msgr2=0x7f5e0c1d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e0c0a5190 0x7f5e0c1d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 1 monclient: found mon.noname-a 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 1 -- 172.21.15.98:0/2490040146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0c13d080 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 5 monclient: authenticate success, global_id 4290 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 monclient: _renew_subs 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87006e3b0 con 0x7fd87009e870 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 monclient: _renew_subs 2024-09-11T02:50:06.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd870093a00 con 0x7fd87009e870 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 1 librados: init done 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.495+0000 7f49d6f00640 1 -- 172.21.15.98:0/1461847782 wait complete. 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: monmap: 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.570 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 librados: starting msgr at 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 librados: starting objecter 2024-09-11T02:50:06.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 Processor -- start 2024-09-11T02:50:06.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e0affd640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e0c0a46a0 0x7f5e0c1d15c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00c4d0 tx=0x7f5dfc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 client.?.objecter ms_handle_connect 0x7f5e0c0a46a0 2024-09-11T02:50:06.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 client.4290.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 15 client.4290.objecter _session_command_op_assign -1 1 2024-09-11T02:50:06.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 20 client.4290.objecter _get_session osd=-1 returning homeless 2024-09-11T02:50:06.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 1 -- 172.21.15.98:0/2490040146 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dfc017070 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:06.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:06.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd8700a75e0 con 0x7fd87009e870 2024-09-11T02:50:06.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: dump: 2024-09-11T02:50:06.574 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.574 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.574 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.574 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.574 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.574 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 1 -- 172.21.15.98:0/2490040146 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dfc007070 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 1 -- 172.21.15.98:0/2490040146 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dfc0076d0 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 10 monclient: dump: 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.576 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.577 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.578 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-09-11T02:50:06.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T02:50:06.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bad9d640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc137360 0x7f69bc137780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009e50 tx=0x7f69b401c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4e662e18f8c23 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 -- start start 2024-09-11T02:50:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 librados: setting wanted keys 2024-09-11T02:50:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 librados: calling monclient init 2024-09-11T02:50:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: init 2024-09-11T02:50:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: _add_conns ranks=[2,6,8,3,0,5,1,7,4] 2024-09-11T02:50:06.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49d00a46a0 0x7f49d0140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): picked mon.noname-b con 0x7f49d00a46a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49d00a4fc0 0x7f49d0140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): picked mon.noname-d con 0x7f49d00a4fc0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f49d013a040 0x7f49d011f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): picked mon.noname-e con 0x7f49d013a040 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f49d0144960 con 0x7f49d00a46a0 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f49d0077860 con 0x7f49d00a4fc0 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f49d0144ae0 con 0x7f49d013a040 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient(hunting): authenticate will time out at 960.381506s 2024-09-11T02:50:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e11656640 5 monclient: authenticate success, global_id 4356 2024-09-11T02:50:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e11656640 10 monclient: _renew_subs 2024-09-11T02:50:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e11656640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0c13e440 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5e097fa640 4 set_mon_vals no callback set 2024-09-11T02:50:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49d00a46a0 0x7f49d0140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49d00a46a0 0x7f49d0140580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:32768/0 (socket says 172.21.15.98:32768) 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 -- 172.21.15.98:0/110356458 learned_addr learned my addr 172.21.15.98:0/110356458 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): get_auth_request con 0x7f49d00a46a0 auth_method 0 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d549d640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f49d013a040 0x7f49d011f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d549d640 10 monclient(hunting): get_auth_request con 0x7f49d013a040 auth_method 0 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d549d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d549d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d549d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd8580129a0 con 0x7fd87009e870 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd85003df70 0x7fd850040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fd85804ebe0 con 0x7fd87009e870 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 client.4290.objecter ms_dispatch 0x7fd870000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 3 client.4290.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 3 client.4290.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 client.4290.objecter checking command 1 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7fd850043d70 0x7fd8500461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 client.4290.objecter _get_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 client.4290.objecter put_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 15 client.4290.objecter _session_command_op_remove -1 1 2024-09-11T02:50:06.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 client.4290.objecter _get_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 client.4290.objecter get_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 15 client.4290.objecter _session_command_op_assign 1 1 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 client.4290.objecter put_session s=0x7fd8500436a0 osd=1 4 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 client.4290.objecter _send_command 1 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd850046880 con 0x7fd850043d70 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 client.4290.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86f7fe640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd85003df70 0x7fd850040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86f7fe640 10 monclient: get_auth_request con 0x7fd85003df70 auth_method 0 2024-09-11T02:50:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49d00a4fc0 0x7f49d0140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 10 monclient(hunting): get_auth_request con 0x7f49d00a4fc0 auth_method 0 2024-09-11T02:50:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd874e88640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7fd850043d70 0x7fd8500461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd874e88640 10 monclient: get_auth_request con 0x7fd850043d70 auth_method 0 2024-09-11T02:50:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fd858019d30 con 0x7fd87009e870 2024-09-11T02:50:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T02:50:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 1 -- 172.21.15.98:0/1548524357 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f69b40ac020 con 0x7f69bc137360 2024-09-11T02:50:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: dump: 2024-09-11T02:50:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.587 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.588 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 1 -- 172.21.15.98:0/1548524357 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b4009110 con 0x7f69bc137360 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 1 -- 172.21.15.98:0/1548524357 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f69b40a6d10 con 0x7f69bc137360 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69b9d9b640 10 monclient: dump: 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.590 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 -- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f49d013a040 msgr2=0x7f49d011f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f49d013a040 0x7f49d011f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 -- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49d00a4fc0 msgr2=0x7f49d0140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f49d00a4fc0 0x7f49d0140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 monclient: found mon.noname-b 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 -- 172.21.15.98:0/110356458 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d0120dd0 con 0x7f49d00a46a0 2024-09-11T02:50:06.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 5 monclient: authenticate success, global_id 4347 2024-09-11T02:50:06.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 4 set_mon_vals no callback set 2024-09-11T02:50:06.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 client.4290.objecter ms_handle_connect 0x7fd850043d70 2024-09-11T02:50:06.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd874e88640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7fd850043d70 0x7fd8500461b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d4c9c640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49d00a46a0 0x7f49d0140580 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009710 tx=0x7f49c40190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86f7fe640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd85003df70 0x7fd850040430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8680ae000 tx=0x7fd8680a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 client.4290.objecter ms_handle_connect 0x7fd85003df70 2024-09-11T02:50:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 10 monclient: shutdown 2024-09-11T02:50:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 1 -- 172.21.15.98:0/1548524357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc137360 msgr2=0x7f69bc137780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc137360 0x7f69bc137780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009e50 tx=0x7f69b401c8f0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 client.?.objecter ms_handle_connect 0x7f49d00a46a0 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49c40b0040 con 0x7f49d00a46a0 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: dump: 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.595 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 5 monclient: authenticate success, global_id 4241 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: _renew_subs 2024-09-11T02:50:06.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c40b9070 con 0x7f49d00a46a0 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d00a71e0 con 0x7f49d00a46a0 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49c40aa550 con 0x7f49d00a46a0 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49ceffd640 4 set_mon_vals no callback set 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49cd7fa640 10 monclient: dump: 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 1 -- 172.21.15.98:0/1548524357 shutdown_connections 2024-09-11T02:50:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f69bc076ee0 0x7f69bc0772c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc137360 0x7f69bc137780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 1 --2- 172.21.15.98:0/1548524357 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f69bc076b00 0x7f69bc0a4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f69bb59e640 1 -- 172.21.15.98:0/1548524357 >> 172.21.15.98:0/1548524357 conn(0x7f69bc06eed0 msgr2=0x7f69bc06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: _renew_subs 2024-09-11T02:50:06.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d0122890 con 0x7f49d00a46a0 2024-09-11T02:50:06.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f49d6f00640 1 librados: init done 2024-09-11T02:50:06.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 1 -- 172.21.15.98:0/2490040146 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5dfc007260 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7f5df37fe640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5df803df40 0x7f5df8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== osd.1 v2:172.21.15.98:6808/2313933591 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd850046880 con 0x7fd850043d70 2024-09-11T02:50:06.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 client.4290.objecter ms_dispatch 0x7fd870000ba0 command_reply(tid 1: 0 ) 2024-09-11T02:50:06.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 10 client.4290.objecter _finish_command 1 = system:0 2024-09-11T02:50:06.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 20 client.4290.objecter put_session s=0x7fd8500436a0 osd=1 4 2024-09-11T02:50:06.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.499+0000 7fd86cff9640 15 client.4290.objecter _session_command_op_remove 1 1 2024-09-11T02:50:06.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 10 monclient: _renew_subs 2024-09-11T02:50:06.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e0c13e160 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e0a7fc640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5df803df40 0x7f5df8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 1 librados: init done 2024-09-11T02:50:06.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e0a7fc640 10 monclient: get_auth_request con 0x7f5df803df40 auth_method 0 2024-09-11T02:50:06.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- 172.21.15.98:0/1548524357 shutdown_connections 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5df37fe640 1 -- 172.21.15.98:0/2490040146 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f5dfc01ad40 con 0x7f5e0c0a46a0 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5df37fe640 10 client.4356.objecter ms_dispatch 0x7f5e0c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5df37fe640 3 client.4356.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5df37fe640 3 client.4356.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5df37fe640 20 client.4356.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d6f00640 10 client.4241.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d6f00640 15 client.4241.objecter _session_command_op_assign -1 1 2024-09-11T02:50:06.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d6f00640 20 client.4241.objecter _get_session osd=-1 returning homeless 2024-09-11T02:50:06.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d6f00640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d6f00640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f49d0122d00 con 0x7f49d00a46a0 2024-09-11T02:50:06.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f49c40b3070 con 0x7f49d00a46a0 2024-09-11T02:50:06.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 10 client.4356.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e0a7fc640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5df803df40 0x7f5df8040400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c09fb60 tx=0x7f5e00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5df37fe640 10 client.4356.objecter ms_handle_connect 0x7f5df803df40 2024-09-11T02:50:06.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 15 client.4356.objecter _session_command_op_assign -1 1 2024-09-11T02:50:06.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f5e0c13e860 0x7f5e0c13b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 20 client.4356.objecter _get_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 20 client.4356.objecter put_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 20 client.4356.objecter _get_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 15 client.4356.objecter _session_command_op_remove -1 1 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 20 client.4356.objecter get_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 15 client.4356.objecter _session_command_op_assign 4 1 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 20 client.4356.objecter put_session s=0x7f5e0c06eb30 osd=4 4 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 10 client.4356.objecter _send_command 1 2024-09-11T02:50:06.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 --> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e0c13d810 con 0x7f5e0c13e860 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- 172.21.15.98:0/1548524357 wait complete. 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: monmap: 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.607 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 librados: starting msgr at 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 librados: starting objecter 2024-09-11T02:50:06.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 Processor -- start 2024-09-11T02:50:06.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- start start 2024-09-11T02:50:06.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 librados: setting wanted keys 2024-09-11T02:50:06.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 librados: calling monclient init 2024-09-11T02:50:06.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: init 2024-09-11T02:50:06.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b403df40 0x7f49b4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f49c40e88d0 con 0x7f49d00a46a0 2024-09-11T02:50:06.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 10 client.4241.objecter ms_dispatch 0x7f49d0000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 3 client.4241.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 3 client.4241.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 10 client.4241.objecter checking command 1 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7f49b4043db0 0x7f49b40461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 20 client.4241.objecter _get_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 20 client.4241.objecter put_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 15 client.4241.objecter _session_command_op_remove -1 1 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 20 client.4241.objecter _get_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cffff640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b403df40 0x7f49b4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 20 client.4241.objecter get_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 15 client.4241.objecter _session_command_op_assign 5 1 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 20 client.4241.objecter put_session s=0x7f49b40436e0 osd=5 4 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 10 client.4241.objecter _send_command 1 2024-09-11T02:50:06.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cffff640 10 monclient: get_auth_request con 0x7f49b403df40 auth_method 0 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: _add_conns ranks=[1,0,4,3,5,2,6,8,7] 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69bc076b00 0x7f69bc11f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): picked mon.noname-f con 0x7f69bc076b00 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc076ee0 0x7f69bc11f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f69bc076ee0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69bc137360 0x7f69bc11fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f69bc137360 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1435f0 con 0x7f69bc076ee0 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc072a30 con 0x7f69bc076b00 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f69bc143770 con 0x7f69bc137360 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient(hunting): authenticate will time out at 960.386682s 2024-09-11T02:50:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49c40aa3a0 con 0x7f49b4043db0 2024-09-11T02:50:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 20 client.4241.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f49c40ed240 con 0x7f49d00a46a0 2024-09-11T02:50:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T02:50:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc076ee0 0x7f69bc11f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc076ee0 0x7f69bc11f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38976/0 (socket says 172.21.15.98:38976) 2024-09-11T02:50:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 -- 172.21.15.98:0/3073104276 learned_addr learned my addr 172.21.15.98:0/3073104276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): get_auth_request con 0x7f69bc076ee0 auth_method 0 2024-09-11T02:50:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69bc076b00 0x7f69bc11f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d549d640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7f49b4043db0 0x7f49b40461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 10 monclient(hunting): get_auth_request con 0x7f69bc076b00 auth_method 0 2024-09-11T02:50:06.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49d549d640 10 monclient: get_auth_request con 0x7f49b4043db0 auth_method 0 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-09-11T02:50:06.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 -- 172.21.15.98:0/3073104276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69bc137360 msgr2=0x7f69bc11fc60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f69bc137360 0x7f69bc11fc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 -- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69bc076b00 msgr2=0x7f69bc11f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f69bc076b00 0x7f69bc11f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 monclient: found mon.noname-a 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 -- 172.21.15.98:0/3073104276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69bc13cf10 con 0x7f69bc076ee0 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b3fff640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc076ee0 0x7f69bc11f720 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f69a400e930 tx=0x7f69a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 10 client.4241.objecter ms_handle_connect 0x7f49b403df40 2024-09-11T02:50:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 client.?.objecter ms_handle_connect 0x7f69bc076ee0 2024-09-11T02:50:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f49cd7fa640 10 client.4241.objecter ms_handle_connect 0x7f49b4043db0 2024-09-11T02:50:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f69a4017070 con 0x7f69bc076ee0 2024-09-11T02:50:06.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: dump: 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.621 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a4013470 con 0x7f69bc076ee0 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f69a4013a90 con 0x7f69bc076ee0 2024-09-11T02:50:06.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b2ffd640 10 monclient: dump: 2024-09-11T02:50:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.623 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 5 monclient: authenticate success, global_id 4365 2024-09-11T02:50:06.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: _renew_subs 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69b959a640 4 set_mon_vals no callback set 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69bc121bf0 con 0x7f69bc076ee0 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: _renew_subs 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69bc13d8e0 con 0x7f69bc076ee0 2024-09-11T02:50:06.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.503+0000 7f69bb59e640 1 librados: init done 2024-09-11T02:50:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f49cffff640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b403df40 0x7f49b4040400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49d0120a40 tx=0x7f49bc007ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f49d549d640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7f49b4043db0 0x7f49b40461f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bb59e640 10 client.4365.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bb59e640 15 client.4365.objecter _session_command_op_assign -1 1 2024-09-11T02:50:06.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bb59e640 20 client.4365.objecter _get_session osd=-1 returning homeless 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bb59e640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f69bc0a4980 con 0x7f69bc076ee0 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f69a4013c30 con 0x7f69bc076ee0 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f699c03df40 0x7f699c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f69a400f040 con 0x7f69bc076ee0 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 10 client.4365.objecter ms_dispatch 0x7f69bc000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 3 client.4365.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 3 client.4365.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 10 client.4365.objecter checking command 1 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7f699c043db0 0x7f699c0461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 20 client.4365.objecter _get_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 20 client.4365.objecter put_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 15 client.4365.objecter _session_command_op_remove -1 1 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 20 client.4365.objecter _get_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 20 client.4365.objecter get_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 15 client.4365.objecter _session_command_op_assign 3 1 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 20 client.4365.objecter put_session s=0x7f699c0436e0 osd=3 4 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 10 client.4365.objecter _send_command 1 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 --> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69a40044c0 con 0x7f699c043db0 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 20 client.4365.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f69a4056050 con 0x7f69bc076ee0 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69ba59c640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f699c03df40 0x7f699c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69ba59c640 10 monclient: get_auth_request con 0x7f699c03df40 auth_method 0 2024-09-11T02:50:06.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bad9d640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7f699c043db0 0x7f699c0461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bad9d640 10 monclient: get_auth_request con 0x7f699c043db0 auth_method 0 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 10 client.4365.objecter ms_handle_connect 0x7f699c03df40 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69ba59c640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f699c03df40 0x7f699c040400 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00ab60 tx=0x7f69ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69b2ffd640 10 client.4365.objecter ms_handle_connect 0x7f699c043db0 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f69bad9d640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7f699c043db0 0x7f699c0461f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== osd.5 v2:172.21.15.190:6808/1497028198 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f49c40aa3a0 con 0x7f49b4043db0 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f49cd7fa640 10 client.4241.objecter ms_dispatch 0x7f49d0000ba0 command_reply(tid 1: 0 ) 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f49cd7fa640 10 client.4241.objecter _finish_command 1 = system:0 2024-09-11T02:50:06.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f49cd7fa640 20 client.4241.objecter put_session s=0x7f49b40436e0 osd=5 4 2024-09-11T02:50:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.507+0000 7f49cd7fa640 15 client.4241.objecter _session_command_op_remove 5 1 2024-09-11T02:50:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5e0b7fe640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f5e0c13e860 0x7f5e0c13b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5e0b7fe640 10 monclient: get_auth_request con 0x7f5e0c13e860 auth_method 0 2024-09-11T02:50:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5df37fe640 10 client.4356.objecter ms_handle_connect 0x7f5e0c13e860 2024-09-11T02:50:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5e0b7fe640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f5e0c13e860 0x7f5e0c13b400 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5df37fe640 1 -- 172.21.15.98:0/2490040146 <== osd.4 v2:172.21.15.190:6816/1271446291 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e0c13d810 con 0x7f5e0c13e860 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5df37fe640 10 client.4356.objecter ms_dispatch 0x7f5e0c000ba0 command_reply(tid 1: 0 ) 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5df37fe640 10 client.4356.objecter _finish_command 1 = system:0 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5df37fe640 20 client.4356.objecter put_session s=0x7f5e0c06eb30 osd=4 4 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f5df37fe640 15 client.4356.objecter _session_command_op_remove 4 1 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== osd.3 v2:172.21.15.190:6800/373809828 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f69a40044c0 con 0x7f699c043db0 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f69b2ffd640 10 client.4365.objecter ms_dispatch 0x7f69bc000ba0 command_reply(tid 1: 0 ) 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f69b2ffd640 10 client.4365.objecter _finish_command 1 = system:0 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f69b2ffd640 20 client.4365.objecter put_session s=0x7f699c0436e0 osd=3 4 2024-09-11T02:50:06.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.511+0000 7f69b2ffd640 15 client.4365.objecter _session_command_op_remove 3 1 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 10 client.4290.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 15 client.4290.objecter _session_command_op_assign -1 2 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 20 client.4290.objecter _get_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 20 client.4290.objecter put_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 20 client.4290.objecter _get_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 15 client.4290.objecter _session_command_op_remove -1 2 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 20 client.4290.objecter get_session s=0x7fd8500436a0 osd=1 3 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 15 client.4290.objecter _session_command_op_assign 1 2 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 20 client.4290.objecter put_session s=0x7fd8500436a0 osd=1 4 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 10 client.4290.objecter _send_command 2 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.519+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd8700a4c40 con 0x7fd850043d70 2024-09-11T02:50:06.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd86cff9640 1 -- 172.21.15.98:0/2978954752 <== osd.1 v2:172.21.15.98:6808/2313933591 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd8700a4c40 con 0x7fd850043d70 2024-09-11T02:50:06.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd86cff9640 10 client.4290.objecter ms_dispatch 0x7fd870000ba0 command_reply(tid 2: 0 ) 2024-09-11T02:50:06.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd86cff9640 10 client.4290.objecter _finish_command 2 = system:0 2024-09-11T02:50:06.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd86cff9640 20 client.4290.objecter put_session s=0x7fd8500436a0 osd=1 4 2024-09-11T02:50:06.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd86cff9640 15 client.4290.objecter _session_command_op_remove 1 2 2024-09-11T02:50:06.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 10 librados: watch_flush enter 2024-09-11T02:50:06.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 10 librados: watch_flush exit 2024-09-11T02:50:06.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 10 client.4290.objecter close_session for osd.1 2024-09-11T02:50:06.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7fd850043d70 msgr2=0x7fd8500461b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7fd850043d70 0x7fd8500461b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 20 client.4290.objecter put_session s=0x7fd8500436a0 osd=1 1 2024-09-11T02:50:06.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 20 client.4290.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 10 client.4290.objecter successfully canceled tick 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd85003df70 msgr2=0x7fd850040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd85003df70 0x7fd850040430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd8680ae000 tx=0x7fd8680a7070 comp rx=0 tx=0).stop 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 10 monclient: shutdown 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd87009e870 msgr2=0x7fd870072650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 --2- 172.21.15.98:0/2978954752 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd87009e870 0x7fd870072650 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c4d0 tx=0x7fd85800c990 comp rx=0 tx=0).stop 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd874e88640 1 -- 172.21.15.98:0/2978954752 reap_dead start 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 shutdown_connections 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 >> 172.21.15.98:0/2978954752 conn(0x7fd8700a4ed0 msgr2=0x7fd8700a1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 shutdown_connections 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 -- 172.21.15.98:0/2978954752 wait complete. 2024-09-11T02:50:06.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7fd8768eb640 1 librados: shutdown 2024-09-11T02:50:06.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 10 client.4241.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T02:50:06.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 15 client.4241.objecter _session_command_op_assign -1 2 2024-09-11T02:50:06.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 client.4241.objecter _get_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 client.4241.objecter put_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 client.4241.objecter _get_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 15 client.4241.objecter _session_command_op_remove -1 2 2024-09-11T02:50:06.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 client.4241.objecter get_session s=0x7f49b40436e0 osd=5 3 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 15 client.4241.objecter _session_command_op_assign 5 2 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 client.4241.objecter put_session s=0x7f49b40436e0 osd=5 4 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 10 client.4241.objecter _send_command 2 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f49d00a4a80 con 0x7f49b4043db0 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49cd7fa640 1 -- 172.21.15.98:0/110356458 <== osd.5 v2:172.21.15.190:6808/1497028198 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f49d00a4a80 con 0x7f49b4043db0 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49cd7fa640 10 client.4241.objecter ms_dispatch 0x7f49d0000ba0 command_reply(tid 2: 0 ) 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49cd7fa640 10 client.4241.objecter _finish_command 2 = system:0 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49cd7fa640 20 client.4241.objecter put_session s=0x7f49b40436e0 osd=5 4 2024-09-11T02:50:06.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49cd7fa640 15 client.4241.objecter _session_command_op_remove 5 2 2024-09-11T02:50:06.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 10 librados: watch_flush enter 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 10 librados: watch_flush exit 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 10 client.4241.objecter close_session for osd.5 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7f49b4043db0 msgr2=0x7f49b40461f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7f49b4043db0 0x7f49b40461f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 client.4241.objecter put_session s=0x7f49b40436e0 osd=5 1 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 client.4241.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 10 client.4241.objecter successfully canceled tick 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b403df40 msgr2=0x7f49b4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b403df40 0x7f49b4040400 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49d0120a40 tx=0x7f49bc007ef0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 10 monclient: shutdown 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49d00a46a0 msgr2=0x7f49d0140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 --2- 172.21.15.98:0/110356458 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f49d00a46a0 0x7f49d0140580 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009710 tx=0x7f49c40190e0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d549d640 1 -- 172.21.15.98:0/110356458 reap_dead start 2024-09-11T02:50:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 shutdown_connections 2024-09-11T02:50:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 >> 172.21.15.98:0/110356458 conn(0x7f49d006ed00 msgr2=0x7f49d009f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 shutdown_connections 2024-09-11T02:50:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 -- 172.21.15.98:0/110356458 wait complete. 2024-09-11T02:50:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f49d6f00640 1 librados: shutdown 2024-09-11T02:50:06.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 10 client.4365.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 15 client.4365.objecter _session_command_op_assign -1 2 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 20 client.4365.objecter _get_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 20 client.4365.objecter put_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 20 client.4365.objecter _get_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 15 client.4365.objecter _session_command_op_remove -1 2 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 20 client.4365.objecter get_session s=0x7f699c0436e0 osd=3 3 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 15 client.4365.objecter _session_command_op_assign 3 2 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 20 client.4365.objecter put_session s=0x7f699c0436e0 osd=3 4 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 10 client.4365.objecter _send_command 2 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.523+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 --> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f69bc0772c0 con 0x7f699c043db0 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69b2ffd640 1 -- 172.21.15.98:0/3073104276 <== osd.3 v2:172.21.15.190:6800/373809828 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f69bc0772c0 con 0x7f699c043db0 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69b2ffd640 10 client.4365.objecter ms_dispatch 0x7f69bc000ba0 command_reply(tid 2: 0 ) 2024-09-11T02:50:06.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69b2ffd640 10 client.4365.objecter _finish_command 2 = system:0 2024-09-11T02:50:06.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69b2ffd640 20 client.4365.objecter put_session s=0x7f699c0436e0 osd=3 4 2024-09-11T02:50:06.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69b2ffd640 15 client.4365.objecter _session_command_op_remove 3 2 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 10 librados: watch_flush enter 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 10 librados: watch_flush exit 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 10 client.4365.objecter close_session for osd.3 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7f699c043db0 msgr2=0x7f699c0461f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7f699c043db0 0x7f699c0461f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 20 client.4365.objecter put_session s=0x7f699c0436e0 osd=3 1 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 20 client.4365.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 10 client.4365.objecter successfully canceled tick 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f699c03df40 msgr2=0x7f699c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f699c03df40 0x7f699c040400 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00ab60 tx=0x7f69ac008040 comp rx=0 tx=0).stop 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 10 monclient: shutdown 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc076ee0 msgr2=0x7f69bc11f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 --2- 172.21.15.98:0/3073104276 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69bc076ee0 0x7f69bc11f720 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f69a400e930 tx=0x7f69a400edf0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bad9d640 1 -- 172.21.15.98:0/3073104276 reap_dead start 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 shutdown_connections 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 >> 172.21.15.98:0/3073104276 conn(0x7f69bc06eed0 msgr2=0x7f69bc0a7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 shutdown_connections 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 -- 172.21.15.98:0/3073104276 wait complete. 2024-09-11T02:50:06.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.527+0000 7f69bb59e640 1 librados: shutdown 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 10 client.4356.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 15 client.4356.objecter _session_command_op_assign -1 2 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 20 client.4356.objecter _get_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 20 client.4356.objecter put_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 20 client.4356.objecter _get_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 15 client.4356.objecter _session_command_op_remove -1 2 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 20 client.4356.objecter get_session s=0x7f5e0c06eb30 osd=4 3 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 15 client.4356.objecter _session_command_op_assign 4 2 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 20 client.4356.objecter put_session s=0x7f5e0c06eb30 osd=4 4 2024-09-11T02:50:06.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 10 client.4356.objecter _send_command 2 2024-09-11T02:50:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 --> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e0c13b940 con 0x7f5e0c13e860 2024-09-11T02:50:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5df37fe640 1 -- 172.21.15.98:0/2490040146 <== osd.4 v2:172.21.15.190:6816/1271446291 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5e0c13b940 con 0x7f5e0c13e860 2024-09-11T02:50:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5df37fe640 10 client.4356.objecter ms_dispatch 0x7f5e0c000ba0 command_reply(tid 2: 0 ) 2024-09-11T02:50:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5df37fe640 10 client.4356.objecter _finish_command 2 = system:0 2024-09-11T02:50:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5df37fe640 20 client.4356.objecter put_session s=0x7f5e0c06eb30 osd=4 4 2024-09-11T02:50:06.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.531+0000 7f5df37fe640 15 client.4356.objecter _session_command_op_remove 4 2 2024-09-11T02:50:06.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 10 librados: watch_flush enter 2024-09-11T02:50:06.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 10 librados: watch_flush exit 2024-09-11T02:50:06.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 10 client.4356.objecter close_session for osd.4 2024-09-11T02:50:06.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f5e0c13e860 msgr2=0x7f5e0c13b400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f5e0c13e860 0x7f5e0c13b400 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 20 client.4356.objecter put_session s=0x7f5e0c06eb30 osd=4 1 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 20 client.4356.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 10 client.4356.objecter successfully canceled tick 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5df803df40 msgr2=0x7f5df8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5df803df40 0x7f5df8040400 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c09fb60 tx=0x7f5e00008040 comp rx=0 tx=0).stop 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 10 monclient: shutdown 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e0c0a46a0 msgr2=0x7f5e0c1d15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 --2- 172.21.15.98:0/2490040146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e0c0a46a0 0x7f5e0c1d15c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00c4d0 tx=0x7f5dfc00c990 comp rx=0 tx=0).stop 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e0b7fe640 1 -- 172.21.15.98:0/2490040146 reap_dead start 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 shutdown_connections 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 >> 172.21.15.98:0/2490040146 conn(0x7f5e0c06ed00 msgr2=0x7f5e0c0739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 shutdown_connections 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 -- 172.21.15.98:0/2490040146 wait complete. 2024-09-11T02:50:06.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.535+0000 7f5e11656640 1 librados: shutdown 2024-09-11T02:50:06.651 INFO:teuthology.orchestra.run.smithi098.stdout:47244640260 2024-09-11T02:50:06.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T02:50:06.652 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-09-11T02:50:06.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T02:50:06.653 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-09-11T02:50:06.653 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T02:50:06.654 INFO:teuthology.orchestra.run.smithi098.stdout:51539607557 2024-09-11T02:50:06.655 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T02:50:06.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.735+0000 7f3f42d84640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.735+0000 7f3f42d84640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.735+0000 7f3f42d84640 10 monclient: monmap: 2024-09-11T02:50:06.736 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.740120+0000 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.740120+0000 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.737 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.738 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.738 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.735+0000 7f3f42d84640 1 Processor -- start 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.735+0000 7f3f42d84640 1 -- start start 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.735+0000 7f3f42d84640 10 monclient: init 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient: _add_conns ranks=[3,4,1,8,5,7,9,10,6,2,0] 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f3c0a09d0 0x7f3f3c0a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient(hunting): picked mon.noname-g con 0x7f3f3c0a09d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1365c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c0a0200 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3f3c136b00 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:06.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f3c0a50c0 con 0x7f3f3c136b00 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3c09ff40 con 0x7f3f3c0a09d0 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3f3c0a5340 con 0x7f3f3c0a0200 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f42d84640 10 monclient(hunting): authenticate will time out at 960.620913s 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1365c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55470/0 (socket says 172.21.15.98:55470) 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 -- 172.21.15.98:0/2725688837 learned_addr learned my addr 172.21.15.98:0/2725688837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): get_auth_request con 0x7f3f3c0a0200 auth_method 0 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f40b20640 1 --2- 172.21.15.98:0/2725688837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f3c0a09d0 0x7f3f3c0a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f40b20640 10 monclient(hunting): get_auth_request con 0x7f3f3c0a09d0 auth_method 0 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f40b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f40b20640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f40b20640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 -- 172.21.15.98:0/2725688837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f3c0a09d0 msgr2=0x7f3f3c0a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 --2- 172.21.15.98:0/2725688837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f3c0a09d0 0x7f3f3c0a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 -- 172.21.15.98:0/2725688837 >> v1:172.21.15.190:6789/0 conn(0x7f3f3c136b00 legacy=0x7f3f3c13d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 monclient: found mon.noname-c 2024-09-11T02:50:06.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:06.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.739+0000 7f3f3bfff640 1 -- 172.21.15.98:0/2725688837 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c0a1f20 con 0x7f3f3c0a0200 2024-09-11T02:50:06.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3bfff640 1 --2- 172.21.15.98:0/2725688837 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1365c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f280098a0 tx=0x7f3f2801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=c94614fb8b022225 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 1 -- 172.21.15.98:0/2725688837 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f280b0070 con 0x7f3f3c0a0200 2024-09-11T02:50:06.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: dump: 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.746 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 1 -- 172.21.15.98:0/2725688837 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f280aa810 con 0x7f3f3c0a0200 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 5 monclient: authenticate success, global_id 4270 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 4 set_mon_vals no callback set 2024-09-11T02:50:06.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 1 -- 172.21.15.98:0/2725688837 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f280a9300 con 0x7f3f3c0a0200 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f3b7fe640 10 monclient: dump: 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.748 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 10 monclient: shutdown 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 -- 172.21.15.98:0/2725688837 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a0200 msgr2=0x7f3f3c1365c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 --2- 172.21.15.98:0/2725688837 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1365c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f280098a0 tx=0x7f3f2801c440 comp rx=0 tx=0).stop 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 -- 172.21.15.98:0/2725688837 shutdown_connections 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 --2- 172.21.15.98:0/2725688837 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1365c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 --2- 172.21.15.98:0/2725688837 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f3c0a09d0 0x7f3f3c0a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 -- 172.21.15.98:0/2725688837 >> 172.21.15.98:0/2725688837 conn(0x7f3f3c06eed0 msgr2=0x7f3f3c06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 -- 172.21.15.98:0/2725688837 shutdown_connections 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 -- 172.21.15.98:0/2725688837 wait complete. 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 10 monclient: monmap: 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 librados: starting msgr at 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.743+0000 7f3f42d84640 1 librados: starting objecter 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 Processor -- start 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 -- start start 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 librados: setting wanted keys 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 librados: calling monclient init 2024-09-11T02:50:06.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient: init 2024-09-11T02:50:06.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient: _add_conns ranks=[5,4,8,6,7,3,2,0,1] 2024-09-11T02:50:06.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1225a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): picked mon.noname-h con 0x7f3f3c0a0200 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a09d0 0x7f3f3c1200e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3c0a09d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3f3c136b00 0x7f3f3c120620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a09d0 0x7f3f3c1200e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a09d0 0x7f3f3c1200e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55480/0 (socket says 172.21.15.98:55480) 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1225a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1225a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:51814/0 (socket says 172.21.15.98:51814) 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 1 -- 172.21.15.98:0/1327090001 learned_addr learned my addr 172.21.15.98:0/1327090001 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3f3c136b00 0x7f3f3c120620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): picked mon.noname-e con 0x7f3f3c136b00 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3f3c0a07c0 con 0x7f3f3c0a09d0 2024-09-11T02:50:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3f3c0a0640 con 0x7f3f3c0a0200 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3f3c077310 con 0x7f3f3c136b00 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient(hunting): authenticate will time out at 960.629756s 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 10 monclient(hunting): get_auth_request con 0x7f3f3c0a0200 auth_method 0 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): get_auth_request con 0x7f3f3c136b00 auth_method 0 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3bfff640 10 monclient(hunting): get_auth_request con 0x7f3f3c0a09d0 auth_method 0 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 -- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f3c0a0200 msgr2=0x7f3f3c1225a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1225a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 -- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a09d0 msgr2=0x7f3f3c1200e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a09d0 0x7f3f3c1200e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 monclient: found mon.noname-e 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c123da0 con 0x7f3f3c136b00 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f40b20640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1225a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f41321640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3f3c136b00 0x7f3f3c120620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400be00 tx=0x7f3f3400bef0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 client.?.objecter ms_handle_connect 0x7f3f3c136b00 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3bfff640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a09d0 0x7f3f3c1200e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 1 -- 172.21.15.98:0/1327090001 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f3400cc70 con 0x7f3f3c136b00 2024-09-11T02:50:06.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: dump: 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.760 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 1 -- 172.21.15.98:0/1327090001 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f3400ce50 con 0x7f3f3c136b00 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 1 -- 172.21.15.98:0/1327090001 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f34014d20 con 0x7f3f3c136b00 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f397fa640 10 monclient: dump: 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.762 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 5 monclient: authenticate success, global_id 4283 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient: _renew_subs 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3c13eb90 con 0x7f3f3c136b00 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f3affd640 4 set_mon_vals no callback set 2024-09-11T02:50:06.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient: _renew_subs 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3c1367b0 con 0x7f3f3c136b00 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.747+0000 7f3f42d84640 1 librados: init done 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f42d84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f42d84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f42d84640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f3c077640 con 0x7f3f3c136b00 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 1 -- 172.21.15.98:0/1327090001 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3f34003980 con 0x7f3f3c136b00 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f2003df40 0x7f3f20040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 1 -- 172.21.15.98:0/1327090001 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f3f3404f750 con 0x7f3f3c136b00 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 10 client.4283.objecter ms_dispatch 0x7f3f3c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 3 client.4283.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 3 client.4283.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 20 client.4283.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f40b20640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f2003df40 0x7f3f20040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f40b20640 10 monclient: get_auth_request con 0x7f3f2003df40 auth_method 0 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f397fa640 10 client.4283.objecter ms_handle_connect 0x7f3f2003df40 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.751+0000 7f3f40b20640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f2003df40 0x7f3f20040400 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000bfb0 tx=0x7f3f30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.759+0000 7f3f397fa640 1 -- 172.21.15.98:0/1327090001 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3f3404d2d0 con 0x7f3f3c136b00 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.759+0000 7f3f397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.759+0000 7f3f397fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient: monmap: 2024-09-11T02:50:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.814821+0000 2024-09-11T02:50:06.811 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.814821+0000 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.812 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.813 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 1 Processor -- start 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 1 -- start start 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient: init 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient: _add_conns ranks=[3,4,1,8,6,9,5,0,10,2,7] 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient(hunting): picked mon.noname-g con 0x7f68f0139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f006fe50 0x7f68f00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient(hunting): picked mon.noname-c con 0x7f68f006fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f68f0072640 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f0071370 con 0x7f68f0072640 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f68f00719c0 con 0x7f68f0139b80 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f68f0071c10 con 0x7f68f006fe50 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f7a6d640 10 monclient(hunting): authenticate will time out at 960.695053s 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f006fe50 0x7f68f00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f006fe50 0x7f68f00a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55494/0 (socket says 172.21.15.98:55494) 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 -- 172.21.15.98:0/2389705825 learned_addr learned my addr 172.21.15.98:0/2389705825 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): get_auth_request con 0x7f68f006fe50 auth_method 0 2024-09-11T02:50:06.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5809640 1 --2- 172.21.15.98:0/2389705825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 -- 172.21.15.98:0/2389705825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 msgr2=0x7f68f0139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 --2- 172.21.15.98:0/2389705825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f0139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 -- 172.21.15.98:0/2389705825 >> v1:172.21.15.190:6789/0 conn(0x7f68f0072640 legacy=0x7f68f00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 monclient: found mon.noname-c 2024-09-11T02:50:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5008640 1 -- 172.21.15.98:0/2389705825 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f00a9600 con 0x7f68f006fe50 2024-09-11T02:50:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5809640 10 monclient: get_auth_request con 0x7f68f0139b80 auth_method 0 2024-09-11T02:50:06.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.811+0000 7f68f5809640 1 --2- 172.21.15.98:0/2389705825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f0139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f5008640 1 --2- 172.21.15.98:0/2389705825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f006fe50 0x7f68f00a7a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f68e00098a0 tx=0x7f68e001c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1006e5a0a97fd3fc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 1 -- 172.21.15.98:0/2389705825 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68e00b1070 con 0x7f68f006fe50 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: dump: 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.818 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 1 -- 172.21.15.98:0/2389705825 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e00aa7b0 con 0x7f68f006fe50 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 1 -- 172.21.15.98:0/2389705825 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68e00aae10 con 0x7f68f006fe50 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f4807640 10 monclient: dump: 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f7a6d640 5 monclient: authenticate success, global_id 4279 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f7a6d640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.815+0000 7f68f7a6d640 4 set_mon_vals no callback set 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient: shutdown 2024-09-11T02:50:06.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2389705825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f006fe50 msgr2=0x7f68f00a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2389705825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f006fe50 0x7f68f00a7a10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f68e00098a0 tx=0x7f68e001c440 comp rx=0 tx=0).stop 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2389705825 shutdown_connections 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2389705825 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f006fe50 0x7f68f00a7a10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2389705825 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f0139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2389705825 >> 172.21.15.98:0/2389705825 conn(0x7f68f00ac730 msgr2=0x7f68f00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2389705825 shutdown_connections 2024-09-11T02:50:06.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2389705825 wait complete. 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient: monmap: 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.826 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 librados: starting msgr at 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 librados: starting objecter 2024-09-11T02:50:06.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 Processor -- start 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- start start 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 librados: setting wanted keys 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 librados: calling monclient init 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient: init 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient: _add_conns ranks=[1,4,3,8,6,2,7,0,5] 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f006fe50 0x7f68f013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): picked mon.noname-f con 0x7f68f006fe50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f0072640 0x7f68f013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): picked mon.noname-c con 0x7f68f0072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f013b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): picked mon.noname-g con 0x7f68f0139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f68f0144450 con 0x7f68f006fe50 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f68f00a8870 con 0x7f68f0139b80 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f68f01445d0 con 0x7f68f0072640 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f7a6d640 10 monclient(hunting): authenticate will time out at 960.703145s 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f006fe50 0x7f68f013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f0072640 0x7f68f013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f0072640 0x7f68f013ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55496/0 (socket says 172.21.15.98:55496) 2024-09-11T02:50:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f013b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 1 -- 172.21.15.98:0/2827894003 learned_addr learned my addr 172.21.15.98:0/2827894003 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 10 monclient(hunting): get_auth_request con 0x7f68f0072640 auth_method 0 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): get_auth_request con 0x7f68f0139b80 auth_method 0 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 10 monclient(hunting): get_auth_request con 0x7f68f006fe50 auth_method 0 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5809640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 1 -- 172.21.15.98:0/2827894003 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f0072640 msgr2=0x7f68f013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f0072640 0x7f68f013ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 1 -- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f006fe50 msgr2=0x7f68f013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f006fe50 0x7f68f013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 1 monclient: found mon.noname-g 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f600a640 1 -- 172.21.15.98:0/2827894003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f013d070 con 0x7f68f0139b80 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.819+0000 7f68f5008640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f0072640 0x7f68f013ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f600a640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f013b250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68e800c730 tx=0x7f68e800cbf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 client.?.objecter ms_handle_connect 0x7f68f0139b80 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 1 -- 172.21.15.98:0/2827894003 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68e8001760 con 0x7f68f0139b80 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:50:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: dump: 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 1 -- 172.21.15.98:0/2827894003 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e8003df0 con 0x7f68f0139b80 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e7fff640 4 set_mon_vals no callback set 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 5 monclient: authenticate success, global_id 4287 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 monclient: _renew_subs 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 1 -- 172.21.15.98:0/2827894003 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68e80050d0 con 0x7f68f0139b80 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f00abbd0 con 0x7f68f0139b80 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 monclient: dump: 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 monclient: _renew_subs 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f013cd90 con 0x7f68f0139b80 2024-09-11T02:50:06.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 1 librados: init done 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient: monmap: 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.826324+0000 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.826324+0000 2024-09-11T02:50:06.837 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 1 Processor -- start 2024-09-11T02:50:06.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 1 -- start start 2024-09-11T02:50:06.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient: init 2024-09-11T02:50:06.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f0139f60 con 0x7f68f0139b80 2024-09-11T02:50:06.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient: _add_conns ranks=[3,1,8,4,9,7,10,5,2,6,0] 2024-09-11T02:50:06.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657c139b80 0x7f657c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient(hunting): picked mon.noname-g con 0x7f657c139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f657c06fe50 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:06.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c072640 0x7f657c0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient(hunting): picked mon.noname-e con 0x7f657c072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 1 -- 172.21.15.98:0/2827894003 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f68e8003980 con 0x7f68f0139b80 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f657c071370 con 0x7f657c06fe50 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c072640 0x7f657c0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f657c0719c0 con 0x7f657c139b80 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657c139b80 0x7f657c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f657c071c10 con 0x7f657c072640 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f657c06fe50 0x7f657c0a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:49396/0 (socket says 172.21.15.98:49396) 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657affd640 1 -- 172.21.15.98:0/149975282 learned_addr learned my addr 172.21.15.98:0/149975282 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f6581cfe640 10 monclient(hunting): authenticate will time out at 960.706797s 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c072640 0x7f657c0a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:42054/0 (socket says 172.21.15.98:42054) 2024-09-11T02:50:06.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 10 monclient(hunting): get_auth_request con 0x7f657c139b80 auth_method 0 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): get_auth_request con 0x7f657c072640 auth_method 0 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f68d403df40 0x7f68d4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 1 -- 172.21.15.98:0/2827894003 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f68e804ed90 con 0x7f68f0139b80 2024-09-11T02:50:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 10 client.4287.objecter ms_dispatch 0x7f68f0000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 3 client.4287.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 3 client.4287.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68e67fc640 20 client.4287.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 1 -- 172.21.15.98:0/149975282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657c139b80 msgr2=0x7f657c139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 1 --2- 172.21.15.98:0/149975282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657c139b80 0x7f657c139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 1 -- 172.21.15.98:0/149975282 >> v1:172.21.15.190:6789/0 conn(0x7f657c06fe50 legacy=0x7f657c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 1 monclient: found mon.noname-e 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657a7fc640 1 -- 172.21.15.98:0/149975282 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895808350 0 0) 0x7f657c071370 con 0x7f657c06fe50 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657bfff640 1 -- 172.21.15.98:0/149975282 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c0a9600 con 0x7f657c072640 2024-09-11T02:50:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f657a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:50:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f5809640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f68d403df40 0x7f68d4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.823+0000 7f68f5809640 10 monclient: get_auth_request con 0x7f68d403df40 auth_method 0 2024-09-11T02:50:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f68e67fc640 10 client.4287.objecter ms_handle_connect 0x7f68d403df40 2024-09-11T02:50:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f68f5809640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f68d403df40 0x7f68d4040400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68ec006fd0 tx=0x7f68ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657bfff640 1 --2- 172.21.15.98:0/149975282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c072640 0x7f657c0a8120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6574009dc0 tx=0x7f657401c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b6fe9adaaed99d34 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 1 -- 172.21.15.98:0/149975282 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65740b2070 con 0x7f657c072640 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: dump: 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 1 -- 172.21.15.98:0/149975282 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65740a5a50 con 0x7f657c072640 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 1 -- 172.21.15.98:0/149975282 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65740a5c40 con 0x7f657c072640 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f657a7fc640 10 monclient: dump: 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f6581cfe640 5 monclient: authenticate success, global_id 4292 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f6581cfe640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f6581cfe640 4 set_mon_vals no callback set 2024-09-11T02:50:06.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f6581cfe640 10 monclient: shutdown 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f6581cfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f6581cfe640 1 -- 172.21.15.98:0/149975282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c072640 msgr2=0x7f657c0a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.827+0000 7f6581cfe640 1 --2- 172.21.15.98:0/149975282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c072640 0x7f657c0a8120 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6574009dc0 tx=0x7f657401c850 comp rx=0 tx=0).stop 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- 172.21.15.98:0/149975282 shutdown_connections 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 --2- 172.21.15.98:0/149975282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c072640 0x7f657c0a8120 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 --2- 172.21.15.98:0/149975282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f657c139b80 0x7f657c139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- 172.21.15.98:0/149975282 >> 172.21.15.98:0/149975282 conn(0x7f657c0ac730 msgr2=0x7f657c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- 172.21.15.98:0/149975282 shutdown_connections 2024-09-11T02:50:06.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- 172.21.15.98:0/149975282 wait complete. 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient: monmap: 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.861 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 librados: starting msgr at 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 librados: starting objecter 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 Processor -- start 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- start start 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 librados: setting wanted keys 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 librados: calling monclient init 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient: init 2024-09-11T02:50:06.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient: _add_conns ranks=[5,1,8,2,4,6,7,3,0] 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f657c06fe50 0x7f657c0a1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): picked mon.noname-h con 0x7f657c06fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657c072640 0x7f657c0a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): picked mon.noname-f con 0x7f657c072640 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c139b80 0x7f657c0a2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): picked mon.noname-e con 0x7f657c139b80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f657c144450 con 0x7f657c072640 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f657c0a8870 con 0x7f657c06fe50 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f657c1445d0 con 0x7f657c139b80 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f6581cfe640 10 monclient(hunting): authenticate will time out at 960.715037s 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c139b80 0x7f657c0a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c139b80 0x7f657c0a2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:42070/0 (socket says 172.21.15.98:42070) 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 -- 172.21.15.98:0/2022672692 learned_addr learned my addr 172.21.15.98:0/2022672692 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): get_auth_request con 0x7f657c139b80 auth_method 0 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f657c06fe50 0x7f657c0a1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 10 monclient(hunting): get_auth_request con 0x7f657c06fe50 auth_method 0 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 -- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f657c06fe50 msgr2=0x7f657c0a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f657c06fe50 0x7f657c0a1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 -- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657c072640 msgr2=0x7f657c0a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657c072640 0x7f657c0a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 monclient: found mon.noname-e 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.831+0000 7f657bfff640 1 -- 172.21.15.98:0/2022672692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657c0a5360 con 0x7f657c139b80 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f657bfff640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c139b80 0x7f657c0a2690 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f657401af00 tx=0x7f65740195f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 client.?.objecter ms_handle_connect 0x7f657c139b80 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 1 -- 172.21.15.98:0/2022672692 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65740b2050 con 0x7f657c139b80 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:06.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: dump: 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.867 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 1 -- 172.21.15.98:0/2022672692 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65740b3040 con 0x7f657c139b80 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 5 monclient: authenticate success, global_id 4301 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 monclient: _renew_subs 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f657c0a7fb0 con 0x7f657c139b80 2024-09-11T02:50:06.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 1 -- 172.21.15.98:0/2022672692 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6574003c70 con 0x7f657c139b80 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 monclient: dump: 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6579ffb640 4 set_mon_vals no callback set 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 monclient: _renew_subs 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f657c0a4460 con 0x7f657c139b80 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 1 -- 172.21.15.98:0/2022672692 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6574004040 con 0x7f657c139b80 2024-09-11T02:50:06.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 1 librados: init done 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f656803df40 0x7f6568040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 1 -- 172.21.15.98:0/2022672692 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f65740be020 con 0x7f657c139b80 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 client.4301.objecter ms_dispatch 0x7f657c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 3 client.4301.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 3 client.4301.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 20 client.4301.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f657b7fe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f656803df40 0x7f6568040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f657b7fe640 10 monclient: get_auth_request con 0x7f656803df40 auth_method 0 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f657c0a3000 con 0x7f657c139b80 2024-09-11T02:50:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.839+0000 7f68e67fc640 1 -- 172.21.15.98:0/2827894003 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f68e800a640 con 0x7f68f0139b80 2024-09-11T02:50:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.839+0000 7f68e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.839+0000 7f68e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f6563fff640 10 client.4301.objecter ms_handle_connect 0x7f656803df40 2024-09-11T02:50:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.835+0000 7f657b7fe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f656803df40 0x7f6568040400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f656c009ac0 tx=0x7f656c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.843+0000 7f6563fff640 1 -- 172.21.15.98:0/2022672692 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f65740c7060 con 0x7f657c139b80 2024-09-11T02:50:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.843+0000 7f6563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.843+0000 7f6563fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:06.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.847+0000 7f1c01b39640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.847+0000 7f1c01b39640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.847+0000 7f1c01b39640 10 monclient: monmap: 2024-09-11T02:50:06.880 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.850355+0000 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.850355+0000 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.881 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 1 Processor -- start 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 1 -- start start 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient: init 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient: _add_conns ranks=[1,2,0,5,3,10,6,8,4,7,9] 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1bfc139b80 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc06fe50 0x7f1bfc13ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient(hunting): picked mon.noname-b con 0x7f1bfc06fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bfc072640 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:06.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bfc071370 con 0x7f1bfc072640 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bfc0a9600 con 0x7f1bfc139b80 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1bfc0719c0 con 0x7f1bfc06fe50 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1c01b39640 10 monclient(hunting): authenticate will time out at 960.732992s 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1bfc072640 0x7f1bfc140f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53920/0 (socket says 172.21.15.98:53920) 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfbfff640 1 -- 172.21.15.98:0/3725283117 learned_addr learned my addr 172.21.15.98:0/3725283117 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 1 --2- 172.21.15.98:0/3725283117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc06fe50 0x7f1bfc13ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): get_auth_request con 0x7f1bfc06fe50 auth_method 0 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3394374080 0 0) 0x7f1bfc071370 con 0x7f1bfc072640 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): _negotiate 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): my global_id is 4374 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be4003530 con 0x7f1bfc072640 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1735716807 0 0) 0x7f1bfc0a9600 con 0x7f1bfc139b80 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): _negotiate 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient(hunting): my global_id is 4357 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bfc071370 con 0x7f1bfc139b80 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4013508742 0 0) 0x7f1be4003530 con 0x7f1bfc072640 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 1 -- 172.21.15.98:0/3725283117 >> v1:172.21.15.190:6789/0 conn(0x7f1bfc139b80 legacy=0x7f1bfc139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 1 -- 172.21.15.98:0/3725283117 >> v1:172.21.15.98:6789/0 conn(0x7f1bfc072640 legacy=0x7f1bfc140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 1 monclient: found mon.noname-b 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfaffd640 1 -- 172.21.15.98:0/3725283117 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc144690 con 0x7f1bfc06fe50 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.851+0000 7f1bfa7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfaffd640 1 --2- 172.21.15.98:0/3725283117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc06fe50 0x7f1bfc13ea00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00097f0 tx=0x7f1bf001c2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b38f00dd14ca1c41 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bf00b0070 con 0x7f1bfc06fe50 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: dump: 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.886 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf00aa810 con 0x7f1bfc06fe50 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 1 -- 172.21.15.98:0/3725283117 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bf4001cf0 con 0x7f1bfc06fe50 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1bfa7fc640 10 monclient: dump: 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.888 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 5 monclient: authenticate success, global_id 4250 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 4 set_mon_vals no callback set 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 10 monclient: shutdown 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 1 -- 172.21.15.98:0/3725283117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc06fe50 msgr2=0x7f1bfc13ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3725283117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc06fe50 0x7f1bfc13ea00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00097f0 tx=0x7f1bf001c2e0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 1 -- 172.21.15.98:0/3725283117 shutdown_connections 2024-09-11T02:50:06.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3725283117 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1bfc06fe50 0x7f1bfc13ea00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 1 -- 172.21.15.98:0/3725283117 >> 172.21.15.98:0/3725283117 conn(0x7f1bfc0ac730 msgr2=0x7f1bfc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 1 -- 172.21.15.98:0/3725283117 shutdown_connections 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.855+0000 7f1c01b39640 1 -- 172.21.15.98:0/3725283117 wait complete. 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: monmap: 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 librados: starting msgr at 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 librados: starting objecter 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 Processor -- start 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 -- start start 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 librados: setting wanted keys 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 librados: calling monclient init 2024-09-11T02:50:06.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: init 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: _add_conns ranks=[7,5,3,8,1,0,2,4,6] 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1bfc06fe50 0x7f1bfc1d1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): picked mon.noname-i con 0x7f1bfc06fe50 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1bfc072640 0x7f1bfc1d1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): picked mon.noname-h con 0x7f1bfc072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bfc139b80 0x7f1bfc1d2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): picked mon.noname-g con 0x7f1bfc139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1bfc0702d0 con 0x7f1bfc139b80 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1bfc145690 con 0x7f1bfc072640 2024-09-11T02:50:06.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f1bfc145810 con 0x7f1bfc06fe50 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient(hunting): authenticate will time out at 960.741891s 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1bfc06fe50 0x7f1bfc1d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1bfc06fe50 0x7f1bfc1d1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:34064/0 (socket says 172.21.15.98:34064) 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 -- 172.21.15.98:0/3338097613 learned_addr learned my addr 172.21.15.98:0/3338097613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): get_auth_request con 0x7f1bfc06fe50 auth_method 0 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bfc139b80 0x7f1bfc1d2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1bfc072640 0x7f1bfc1d1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 10 monclient(hunting): get_auth_request con 0x7f1bfc139b80 auth_method 0 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 10 monclient(hunting): get_auth_request con 0x7f1bfc072640 auth_method 0 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient(hunting): handle_auth_done global_id 4237 payload 824 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 -- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1bfc072640 msgr2=0x7f1bfc1d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1bfc072640 0x7f1bfc1d1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 -- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bfc139b80 msgr2=0x7f1bfc1d2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bfc139b80 0x7f1bfc1d2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 monclient: found mon.noname-i 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:50:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 -- 172.21.15.98:0/3338097613 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc0a6b20 con 0x7f1bfc06fe50 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bfb7fe640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1bfc06fe50 0x7f1bfc1d1710 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00c400 tx=0x7f1bec00c8c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 client.?.objecter ms_handle_connect 0x7f1bfc06fe50 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 1 -- 172.21.15.98:0/3338097613 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bec017070 con 0x7f1bfc06fe50 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: dump: 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 5 monclient: authenticate success, global_id 4237 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 1 -- 172.21.15.98:0/3338097613 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bec003600 con 0x7f1bfc06fe50 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: _renew_subs 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bfc140e90 con 0x7f1bfc06fe50 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 1 -- 172.21.15.98:0/3338097613 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1bec003ca0 con 0x7f1bfc06fe50 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1be3fff640 10 monclient: dump: 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.898 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.899 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: _renew_subs 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bfc140bb0 con 0x7f1bfc06fe50 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1c01b39640 1 librados: init done 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.859+0000 7f1bf9ffb640 4 set_mon_vals no callback set 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1c01b39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1c01b39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1c01b39640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bfc071c10 con 0x7f1bfc06fe50 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 1 -- 172.21.15.98:0/3338097613 <== mon.7 v2:172.21.15.190:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1bec003830 con 0x7f1bfc06fe50 2024-09-11T02:50:06.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1be803df40 0x7f1be8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 1 -- 172.21.15.98:0/3338097613 <== mon.7 v2:172.21.15.190:3303/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f1bec04e3b0 con 0x7f1bfc06fe50 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 10 client.4237.objecter ms_dispatch 0x7f1bfc000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 3 client.4237.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 3 client.4237.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 20 client.4237.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1bfaffd640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1be803df40 0x7f1be8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1bfaffd640 10 monclient: get_auth_request con 0x7f1be803df40 auth_method 0 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1be3fff640 10 client.4237.objecter ms_handle_connect 0x7f1be803df40 2024-09-11T02:50:06.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.863+0000 7f1bfaffd640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1be803df40 0x7f1be8040400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00ad000 tx=0x7f1bf00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient: monmap: 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.870257+0000 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.870257+0000 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.902 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 Processor -- start 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 -- start start 2024-09-11T02:50:06.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient: init 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient: get_monmap_and_config 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient: monmap: 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:06.871514+0000 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:06.871514+0000 2024-09-11T02:50:06.904 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:06.905 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient: _add_conns ranks=[5,9,3,1,8,0,7,6,2,4,10] 2024-09-11T02:50:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa0500a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): picked mon.noname-h con 0x7fa0500a3de0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0500a4700 0x7fa05009b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): picked mon.noname-a con 0x7fa0500a4700 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:06.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 Processor -- start 2024-09-11T02:50:06.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 -- start start 2024-09-11T02:50:06.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient: init 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa05009b680 0x7fa05009bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): picked mon.noname-g con 0x7fa05009b680 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa0500a2200 con 0x7fa0500a4700 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa050068fa0 con 0x7fa05009b680 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa0500a18d0 con 0x7fa0500a3de0 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa057af3640 10 monclient(hunting): authenticate will time out at 960.750733s 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0500a4700 0x7fa05009b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa0500a4700 0x7fa05009b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38980/0 (socket says 172.21.15.98:38980) 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 1 -- 172.21.15.98:0/571516335 learned_addr learned my addr 172.21.15.98:0/571516335 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 10 monclient(hunting): get_auth_request con 0x7fa0500a4700 auth_method 0 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 1 --2- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa0500a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): get_auth_request con 0x7fa0500a3de0 auth_method 0 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 1 --2- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa05009b680 0x7fa05009bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 10 monclient(hunting): get_auth_request con 0x7fa05009b680 auth_method 0 2024-09-11T02:50:06.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient: _add_conns ranks=[8,7,3,5,6,10,9,4,0,1,2] 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f35140a46a0 0x7f35140a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): picked mon.noname-e con 0x7f35140a46a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f35140a13a0 0x7f35140a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): picked mon.noname-i con 0x7f35140a13a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3514136450 0x7f35141368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): picked mon.noname-g con 0x7f3514136450 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f35140a5500 con 0x7f3514136450 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f35140a11a0 con 0x7f35140a13a0 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f35140a28a0 con 0x7f35140a46a0 2024-09-11T02:50:06.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351a2ef640 10 monclient(hunting): authenticate will time out at 960.751799s 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f3513fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f35140a46a0 0x7f35140a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f3513fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f35140a46a0 0x7f35140a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:42082/0 (socket says 172.21.15.98:42082) 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f3513fff640 1 -- 172.21.15.98:0/932713808 learned_addr learned my addr 172.21.15.98:0/932713808 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05508e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa056090640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f3513fff640 10 monclient(hunting): get_auth_request con 0x7f35140a46a0 auth_method 0 2024-09-11T02:50:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f3513fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f3513fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f3513fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 1 -- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa05009b680 msgr2=0x7fa05009bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 1 --2- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa05009b680 0x7fa05009bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 1 -- 172.21.15.98:0/571516335 >> v2:172.21.15.98:3300/0 conn(0x7fa0500a4700 msgr2=0x7fa05009b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 1 --2- 172.21.15.98:0/571516335 >> v2:172.21.15.98:3300/0 conn(0x7fa0500a4700 0x7fa05009b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 1 monclient: found mon.noname-h 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:50:06.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7fa05588f640 1 -- 172.21.15.98:0/571516335 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0500a1bb0 con 0x7fa0500a3de0 2024-09-11T02:50:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f35137fe640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f35140a13a0 0x7f35140a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.867+0000 7f351888c640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3514136450 0x7f35141368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f351888c640 10 monclient(hunting): get_auth_request con 0x7f3514136450 auth_method 0 2024-09-11T02:50:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f351888c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f351888c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f351888c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f35137fe640 10 monclient(hunting): get_auth_request con 0x7f35140a13a0 auth_method 0 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f35137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f35137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f35137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f35137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f35137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f35137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f351888c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f351888c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f351888c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2024-09-11T02:50:06.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 1 -- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f35140a13a0 msgr2=0x7f35140a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f35140a13a0 0x7f35140a4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 1 -- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3514136450 msgr2=0x7f35141368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3514136450 0x7f35141368b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 1 monclient: found mon.noname-e 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 1 -- 172.21.15.98:0/932713808 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35140a25c0 con 0x7f35140a46a0 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05588f640 1 --2- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa0500a41c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00dbe0 tx=0x7fa04c020670 comp rx=0 tx=0).ready entity=mon.5 client_cookie=2069a44bed9cc4de server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 1 -- 172.21.15.98:0/571516335 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa04c0b5070 con 0x7fa0500a3de0 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: dump: 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.919 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.920 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 1 -- 172.21.15.98:0/571516335 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa04c00d0d0 con 0x7fa0500a3de0 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa057af3640 5 monclient: authenticate success, global_id 4280 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa057af3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa057af3640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa057af3640 4 set_mon_vals no callback set 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 1 -- 172.21.15.98:0/571516335 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa04c00bd30 con 0x7fa0500a3de0 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7fa05488d640 10 monclient: dump: 2024-09-11T02:50:06.921 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.922 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.923 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.871+0000 7f3513fff640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f35140a46a0 0x7f35140a4a80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f35040098a0 tx=0x7f3504020850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a734926ebc239824 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 1 -- 172.21.15.98:0/932713808 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f35040af8f0 con 0x7f35140a46a0 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: dump: 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.924 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 1 -- 172.21.15.98:0/932713808 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35040b4070 con 0x7f35140a46a0 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 1 -- 172.21.15.98:0/932713808 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f35040afa90 con 0x7f35140a46a0 2024-09-11T02:50:06.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f3512ffd640 10 monclient: dump: 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.926 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 5 monclient: authenticate success, global_id 4310 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient: shutdown 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 msgr2=0x7fa0500a41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 --2- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa0500a41c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00dbe0 tx=0x7fa04c020670 comp rx=0 tx=0).stop 2024-09-11T02:50:06.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 4 set_mon_vals no callback set 2024-09-11T02:50:06.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- 172.21.15.98:0/571516335 shutdown_connections 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 --2- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa05009b680 0x7fa05009bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 --2- 172.21.15.98:0/571516335 >> v2:172.21.15.98:3300/0 conn(0x7fa0500a4700 0x7fa05009b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 --2- 172.21.15.98:0/571516335 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa0500a41c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- 172.21.15.98:0/571516335 >> 172.21.15.98:0/571516335 conn(0x7fa0500a4ed0 msgr2=0x7fa0500a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- 172.21.15.98:0/571516335 shutdown_connections 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- 172.21.15.98:0/571516335 wait complete. 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient: monmap: 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.929 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient: shutdown 2024-09-11T02:50:06.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- 172.21.15.98:0/932713808 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f35140a46a0 msgr2=0x7f35140a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f35140a46a0 0x7f35140a4a80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f35040098a0 tx=0x7f3504020850 comp rx=0 tx=0).stop 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- 172.21.15.98:0/932713808 shutdown_connections 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3514136450 0x7f35141368b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f35140a13a0 0x7f35140a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 --2- 172.21.15.98:0/932713808 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f35140a46a0 0x7f35140a4a80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- 172.21.15.98:0/932713808 >> 172.21.15.98:0/932713808 conn(0x7f351406ed00 msgr2=0x7f351406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 librados: starting msgr at 2024-09-11T02:50:06.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 librados: starting objecter 2024-09-11T02:50:06.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 Processor -- start 2024-09-11T02:50:06.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- 172.21.15.98:0/932713808 shutdown_connections 2024-09-11T02:50:06.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- start start 2024-09-11T02:50:06.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 librados: setting wanted keys 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 librados: calling monclient init 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient: init 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient: _add_conns ranks=[7,5,3,2,1,6,8,4,0] 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa05009b680 0x7fa050136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): picked mon.noname-i con 0x7fa05009b680 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa05013b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): picked mon.noname-h con 0x7fa0500a3de0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0500a4700 0x7fa05013b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): picked mon.noname-g con 0x7fa0500a4700 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa0500a27f0 con 0x7fa0500a4700 2024-09-11T02:50:06.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa0500a2670 con 0x7fa0500a3de0 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa0500a00e0 con 0x7fa05009b680 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa057af3640 10 monclient(hunting): authenticate will time out at 960.759046s 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0500a4700 0x7fa05013b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa05013b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0500a4700 0x7fa05013b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:35886/0 (socket says 172.21.15.98:35886) 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 1 -- 172.21.15.98:0/4013485964 learned_addr learned my addr 172.21.15.98:0/4013485964 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 10 monclient(hunting): get_auth_request con 0x7fa0500a4700 auth_method 0 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa05013b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:51852/0 (socket says 172.21.15.98:51852) 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 10 monclient(hunting): get_auth_request con 0x7fa0500a3de0 auth_method 0 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- 172.21.15.98:0/932713808 wait complete. 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient: build_initial_monmap 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient: monmap: 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:06.936 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:06.937 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 librados: starting msgr at 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 librados: starting objecter 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 Processor -- start 2024-09-11T02:50:06.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa056090640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7fa05508e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- start start 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 librados: setting wanted keys 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 librados: calling monclient init 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient: init 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient: _add_conns ranks=[0,4,2,6,5,3,7,1,8] 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35140a13a0 0x7f351413a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): picked mon.noname-a con 0x7f35140a13a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35140a46a0 0x7f351413a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): picked mon.noname-c con 0x7f35140a46a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3514136450 0x7f351413ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): picked mon.noname-b con 0x7f3514136450 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3514077490 con 0x7f35140a13a0 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3514077310 con 0x7f3514136450 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3514077610 con 0x7f35140a46a0 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.875+0000 7f351a2ef640 10 monclient(hunting): authenticate will time out at 960.759966s 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35140a46a0 0x7f351413a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35140a46a0 0x7f351413a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:55500/0 (socket says 172.21.15.98:55500) 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 -- 172.21.15.98:0/4076752283 learned_addr learned my addr 172.21.15.98:0/4076752283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:06.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): get_auth_request con 0x7f35140a46a0 auth_method 0 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3514136450 0x7f351413ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 1 -- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa05009b680 msgr2=0x7fa050136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa05009b680 0x7fa050136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 1 -- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 msgr2=0x7fa05013b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa05013b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 1 monclient: found mon.noname-g 2024-09-11T02:50:06.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 1 -- 172.21.15.98:0/4013485964 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05013d230 con 0x7fa0500a4700 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 10 monclient(hunting): get_auth_request con 0x7f3514136450 auth_method 0 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:06.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:06.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351888c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:06.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient(hunting): handle_auth_done global_id 4297 payload 824 2024-09-11T02:50:06.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 -- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3514136450 msgr2=0x7f351413ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3514136450 0x7f351413ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 -- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35140a13a0 msgr2=0x7f351413a3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:06.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa056090640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0500a4700 0x7fa05013b970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa048009dd0 tx=0x7fa04800c490 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 client.?.objecter ms_handle_connect 0x7fa0500a4700 2024-09-11T02:50:06.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3513fff640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35140a13a0 0x7f351413a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 1 -- 172.21.15.98:0/4013485964 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa048018020 con 0x7fa0500a4700 2024-09-11T02:50:06.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35140a13a0 0x7f351413a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 monclient: found mon.noname-c 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 -- 172.21.15.98:0/4076752283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f351413c5d0 con 0x7f35140a46a0 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: dump: 2024-09-11T02:50:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 5 monclient: authenticate success, global_id 4332 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 monclient: _renew_subs 2024-09-11T02:50:06.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0500a5790 con 0x7fa0500a4700 2024-09-11T02:50:06.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3513fff640 10 monclient: get_auth_request con 0x7f35140a13a0 auth_method 0 2024-09-11T02:50:06.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 monclient: _renew_subs 2024-09-11T02:50:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05012e620 con 0x7fa0500a4700 2024-09-11T02:50:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 1 librados: init done 2024-09-11T02:50:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3513fff640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35140a13a0 0x7f351413a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 1 -- 172.21.15.98:0/4013485964 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa04800cd30 con 0x7fa0500a4700 2024-09-11T02:50:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 1 -- 172.21.15.98:0/4013485964 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0480148f0 con 0x7fa0500a4700 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa047fff640 4 set_mon_vals no callback set 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 monclient: dump: 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.953 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35137fe640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35140a46a0 0x7f351413a910 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f350800e8d0 tx=0x7f350800ed90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 client.?.objecter ms_handle_connect 0x7f35140a46a0 2024-09-11T02:50:06.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 1 -- 172.21.15.98:0/4076752283 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3508017070 con 0x7f35140a46a0 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: dump: 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.955 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: _finish_auth 0 2024-09-11T02:50:06.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 1 -- 172.21.15.98:0/4076752283 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f350800cdb0 con 0x7f35140a46a0 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 1 -- 172.21.15.98:0/4076752283 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f35080138e0 con 0x7f35140a46a0 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 10 monclient: dump: 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:06.957 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:06.958 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f35127fc640 4 set_mon_vals no callback set 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 5 monclient: authenticate success, global_id 4297 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 monclient: _renew_subs 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3514130f30 con 0x7f35140a46a0 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 monclient: _renew_subs 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3514073210 con 0x7f35140a46a0 2024-09-11T02:50:06.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 1 librados: init done 2024-09-11T02:50:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa038005090 con 0x7fa0500a4700 2024-09-11T02:50:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 1 -- 172.21.15.98:0/4013485964 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa048014480 con 0x7fa0500a4700 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa03403df40 0x7fa034040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 1 -- 172.21.15.98:0/4013485964 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fa04804fe80 con 0x7fa0500a4700 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 10 client.4332.objecter ms_dispatch 0x7fa050000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 3 client.4332.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 3 client.4332.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa0467fc640 20 client.4332.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa05588f640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa03403df40 0x7fa034040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7fa05588f640 10 monclient: get_auth_request con 0x7fa03403df40 auth_method 0 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35141368b0 con 0x7f35140a46a0 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.879+0000 7f3510ff9640 1 -- 172.21.15.98:0/4076752283 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3508024060 con 0x7f35140a46a0 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3510ff9640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f350003df40 0x7f3500040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3510ff9640 1 -- 172.21.15.98:0/4076752283 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f350800f040 con 0x7f35140a46a0 2024-09-11T02:50:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3510ff9640 10 client.4297.objecter ms_dispatch 0x7f3514000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3510ff9640 3 client.4297.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3510ff9640 3 client.4297.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3510ff9640 20 client.4297.objecter dump_active .. 0 homeless 2024-09-11T02:50:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7fa05588f640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa03403df40 0x7fa034040400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0b2000 tx=0x7fa04c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7fa0467fc640 10 client.4332.objecter ms_handle_connect 0x7fa03403df40 2024-09-11T02:50:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3513fff640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f350003df40 0x7f3500040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3513fff640 10 monclient: get_auth_request con 0x7f350003df40 auth_method 0 2024-09-11T02:50:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3510ff9640 10 client.4297.objecter ms_handle_connect 0x7f350003df40 2024-09-11T02:50:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.883+0000 7f3513fff640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f350003df40 0x7f3500040400 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35040b3000 tx=0x7f3504007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.887+0000 7f3510ff9640 1 -- 172.21.15.98:0/4076752283 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f350800c940 con 0x7f35140a46a0 2024-09-11T02:50:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.887+0000 7f3510ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.887+0000 7f3510ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.887+0000 7f1be3fff640 1 -- 172.21.15.98:0/3338097613 <== mon.7 v2:172.21.15.190:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f1bfc0a6b20 con 0x7f1bfc06fe50 2024-09-11T02:50:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.887+0000 7f1be3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.887+0000 7f1be3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.895+0000 7fa0467fc640 1 -- 172.21.15.98:0/4013485964 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fa048022070 con 0x7fa0500a4700 2024-09-11T02:50:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.895+0000 7fa0467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.895+0000 7fa0467fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.975+0000 7f3f42d84640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T02:50:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.975+0000 7f3f42d84640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T02:50:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.975+0000 7f3f42d84640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.975+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3f3c121a40 con 0x7f3f3c136b00 2024-09-11T02:50:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f397fa640 1 -- 172.21.15.98:0/1327090001 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3f3404d650 con 0x7f3f3c136b00 2024-09-11T02:50:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T02:50:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f397fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:06.978 INFO:teuthology.orchestra.run.smithi098.stdout:51539607556 2024-09-11T02:50:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 10 librados: watch_flush enter 2024-09-11T02:50:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 10 librados: watch_flush exit 2024-09-11T02:50:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 20 client.4283.objecter shutdown clearing up homeless session... 2024-09-11T02:50:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 10 client.4283.objecter successfully canceled tick 2024-09-11T02:50:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f2003df40 msgr2=0x7f3f20040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f2003df40 0x7f3f20040400 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3f3000bfb0 tx=0x7f3f30008040 comp rx=0 tx=0).stop 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 10 monclient: shutdown 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3f3c136b00 msgr2=0x7f3f3c120620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3f3c136b00 0x7f3f3c120620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400be00 tx=0x7f3f3400bef0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 shutdown_connections 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f2003df40 0x7f3f20040400 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3f3c136b00 0x7f3f3c120620 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3f3c0a09d0 0x7f3f3c1200e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 --2- 172.21.15.98:0/1327090001 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f3c0a0200 0x7f3f3c1225a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.979+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 >> 172.21.15.98:0/1327090001 conn(0x7f3f3c06eed0 msgr2=0x7f3f3c09d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.983+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 shutdown_connections 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.983+0000 7f3f42d84640 1 -- 172.21.15.98:0/1327090001 wait complete. 2024-09-11T02:50:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:06.983+0000 7f3f42d84640 1 librados: shutdown 2024-09-11T02:50:06.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.2 2024-09-11T02:50:07.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T02:50:07.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T02:50:07.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:07.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f351413ba70 con 0x7f35140a46a0 2024-09-11T02:50:07.058 INFO:teuthology.orchestra.run.smithi098.stdout:51539607555 2024-09-11T02:50:07.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f3510ff9640 1 -- 172.21.15.98:0/4076752283 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f350801b3b0 con 0x7f35140a46a0 2024-09-11T02:50:07.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f3510ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T02:50:07.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f3510ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:07.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 10 librados: watch_flush enter 2024-09-11T02:50:07.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 10 librados: watch_flush exit 2024-09-11T02:50:07.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 20 client.4297.objecter shutdown clearing up homeless session... 2024-09-11T02:50:07.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 10 client.4297.objecter successfully canceled tick 2024-09-11T02:50:07.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f350003df40 msgr2=0x7f3500040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f350003df40 0x7f3500040400 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35040b3000 tx=0x7f3504007f90 comp rx=0 tx=0).stop 2024-09-11T02:50:07.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 10 monclient: shutdown 2024-09-11T02:50:07.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35140a46a0 msgr2=0x7f351413a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35140a46a0 0x7f351413a910 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f350800e8d0 tx=0x7f350800ed90 comp rx=0 tx=0).stop 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 shutdown_connections 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f350003df40 0x7f3500040400 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3514136450 0x7f351413ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f35140a46a0 0x7f351413a910 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 --2- 172.21.15.98:0/4076752283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35140a13a0 0x7f351413a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 >> 172.21.15.98:0/4076752283 conn(0x7f351406ed00 msgr2=0x7f351413d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 shutdown_connections 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 -- 172.21.15.98:0/4076752283 wait complete. 2024-09-11T02:50:07.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.059+0000 7f351a2ef640 1 librados: shutdown 2024-09-11T02:50:07.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.4 2024-09-11T02:50:07.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68f7a6d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T02:50:07.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68f7a6d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T02:50:07.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68f7a6d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:07.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f68f00a5830 con 0x7f68f0139b80 2024-09-11T02:50:07.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68e67fc640 1 -- 172.21.15.98:0/2827894003 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f68e800ad50 con 0x7f68f0139b80 2024-09-11T02:50:07.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T02:50:07.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:07.088 INFO:teuthology.orchestra.run.smithi098.stdout:51539607556 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68f7a6d640 10 librados: watch_flush enter 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68f7a6d640 10 librados: watch_flush exit 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.087+0000 7f68f7a6d640 20 client.4287.objecter shutdown clearing up homeless session... 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 10 client.4287.objecter successfully canceled tick 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f68d403df40 msgr2=0x7f68d4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f68d403df40 0x7f68d4040400 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68ec006fd0 tx=0x7f68ec008040 comp rx=0 tx=0).stop 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 10 monclient: shutdown 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:07.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 msgr2=0x7f68f013b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f013b250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f68e800c730 tx=0x7f68e800cbf0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 shutdown_connections 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f68d403df40 0x7f68d4040400 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68f0139b80 0x7f68f013b250 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f68f0072640 0x7f68f013ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 --2- 172.21.15.98:0/2827894003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68f006fe50 0x7f68f013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 >> 172.21.15.98:0/2827894003 conn(0x7f68f00ac730 msgr2=0x7f68f00aa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 shutdown_connections 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 -- 172.21.15.98:0/2827894003 wait complete. 2024-09-11T02:50:07.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f68f7a6d640 1 librados: shutdown 2024-09-11T02:50:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f6581cfe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T02:50:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f6581cfe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T02:50:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f6581cfe640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:07.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.091+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f657c069090 con 0x7f657c139b80 2024-09-11T02:50:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6563fff640 1 -- 172.21.15.98:0/2022672692 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6574016560 con 0x7f657c139b80 2024-09-11T02:50:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T02:50:07.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6563fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:07.094 INFO:teuthology.orchestra.run.smithi098.stdout:47244640259 2024-09-11T02:50:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 10 librados: watch_flush enter 2024-09-11T02:50:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 10 librados: watch_flush exit 2024-09-11T02:50:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 20 client.4301.objecter shutdown clearing up homeless session... 2024-09-11T02:50:07.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 10 client.4301.objecter successfully canceled tick 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f656803df40 msgr2=0x7f6568040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f656803df40 0x7f6568040400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f656c009ac0 tx=0x7f656c009190 comp rx=0 tx=0).stop 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 10 monclient: shutdown 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c139b80 msgr2=0x7f657c0a2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c139b80 0x7f657c0a2690 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f657401af00 tx=0x7f65740195f0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 shutdown_connections 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f656803df40 0x7f6568040400 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f657c139b80 0x7f657c0a2690 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f657c072640 0x7f657c0a2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 --2- 172.21.15.98:0/2022672692 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f657c06fe50 0x7f657c0a1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 >> 172.21.15.98:0/2022672692 conn(0x7f657c0ac730 msgr2=0x7f657c0aaf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:07.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 shutdown_connections 2024-09-11T02:50:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 -- 172.21.15.98:0/2022672692 wait complete. 2024-09-11T02:50:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.095+0000 7f6581cfe640 1 librados: shutdown 2024-09-11T02:50:07.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.3 2024-09-11T02:50:07.112 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.1 2024-09-11T02:50:07.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T02:50:07.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T02:50:07.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:07.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1bfc0a8ec0 con 0x7f1bfc06fe50 2024-09-11T02:50:07.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1be3fff640 1 -- 172.21.15.98:0/3338097613 <== mon.7 v2:172.21.15.190:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1bec0102e0 con 0x7f1bfc06fe50 2024-09-11T02:50:07.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1be3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T02:50:07.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1be3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:07.166 INFO:teuthology.orchestra.run.smithi098.stdout:47244640260 2024-09-11T02:50:07.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 10 librados: watch_flush enter 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 10 librados: watch_flush exit 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 20 client.4237.objecter shutdown clearing up homeless session... 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 10 client.4237.objecter successfully canceled tick 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1be803df40 msgr2=0x7f1be8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1be803df40 0x7f1be8040400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1bf00ad000 tx=0x7f1bf00a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 10 monclient: shutdown 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1bfc06fe50 msgr2=0x7f1bfc1d1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1bfc06fe50 0x7f1bfc1d1710 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00c400 tx=0x7f1bec00c8c0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 shutdown_connections 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1be803df40 0x7f1be8040400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1bfc072640 0x7f1bfc1d1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1bfc139b80 0x7f1bfc1d2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 --2- 172.21.15.98:0/3338097613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1bfc06fe50 0x7f1bfc1d1710 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 >> 172.21.15.98:0/3338097613 conn(0x7f1bfc0ac730 msgr2=0x7f1bfc0a7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:07.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 shutdown_connections 2024-09-11T02:50:07.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 -- 172.21.15.98:0/3338097613 wait complete. 2024-09-11T02:50:07.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.167+0000 7f1c01b39640 1 librados: shutdown 2024-09-11T02:50:07.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.171+0000 7fa057af3640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T02:50:07.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.171+0000 7fa057af3640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T02:50:07.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.171+0000 7fa057af3640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:07.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.171+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa038005380 con 0x7fa0500a4700 2024-09-11T02:50:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.171+0000 7fa0467fc640 1 -- 172.21.15.98:0/4013485964 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa0500a5790 con 0x7fa0500a4700 2024-09-11T02:50:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.171+0000 7fa0467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T02:50:07.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.171+0000 7fa0467fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:07.172 INFO:teuthology.orchestra.run.smithi098.stdout:51539607556 2024-09-11T02:50:07.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 10 librados: watch_flush enter 2024-09-11T02:50:07.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 10 librados: watch_flush exit 2024-09-11T02:50:07.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 20 client.4332.objecter shutdown clearing up homeless session... 2024-09-11T02:50:07.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 10 client.4332.objecter successfully canceled tick 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa03403df40 msgr2=0x7fa034040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa03403df40 0x7fa034040400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0b2000 tx=0x7fa04c004740 comp rx=0 tx=0).stop 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 10 monclient: shutdown 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0500a4700 msgr2=0x7fa05013b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0500a4700 0x7fa05013b970 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa048009dd0 tx=0x7fa04800c490 comp rx=0 tx=0).stop 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 shutdown_connections 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa03403df40 0x7fa034040400 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0500a3de0 0x7fa05013b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa0500a4700 0x7fa05013b970 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 --2- 172.21.15.98:0/4013485964 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa05009b680 0x7fa050136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 >> 172.21.15.98:0/4013485964 conn(0x7fa0500a4ed0 msgr2=0x7fa0500a6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:07.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 shutdown_connections 2024-09-11T02:50:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 -- 172.21.15.98:0/4013485964 wait complete. 2024-09-11T02:50:07.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:07.175+0000 7fa057af3640 1 librados: shutdown 2024-09-11T02:50:07.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.0 2024-09-11T02:50:07.190 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.5 2024-09-11T02:50:07.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T02:50:08.077 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T02:50:08.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T02:50:08.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T02:50:08.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient: get_monmap_and_config 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient: monmap: 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:08.129776+0000 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:08.129776+0000 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 1 Processor -- start 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 1 -- start start 2024-09-11T02:50:08.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient: init 2024-09-11T02:50:08.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient: _add_conns ranks=[1,0,3,10,9,8,5,7,2,4,6] 2024-09-11T02:50:08.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe8600a7030 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:08.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe86006f720 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:08.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe86013a2b0 0x7fe860142130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient(hunting): picked mon.noname-g con 0x7fe86013a2b0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:08.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86006ee20 con 0x7fe86006f720 2024-09-11T02:50:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86006f4e0 con 0x7fe8600a7030 2024-09-11T02:50:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe8600a7410 con 0x7fe86013a2b0 2024-09-11T02:50:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 10 monclient(hunting): authenticate will time out at 962.010012s 2024-09-11T02:50:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe864c09640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe86006f720 0x7fe86013ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50904/0 (socket says 172.21.15.98:50904) 2024-09-11T02:50:08.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe864c09640 1 -- 172.21.15.98:0/310981475 learned_addr learned my addr 172.21.15.98:0/310981475 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe86013a2b0 0x7fe860142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 10 monclient(hunting): get_auth_request con 0x7fe86013a2b0 auth_method 0 2024-09-11T02:50:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2406428969 0 0) 0x7fe86006ee20 con 0x7fe86006f720 2024-09-11T02:50:08.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): _negotiate 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): my global_id is 4392 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe84c003530 con 0x7fe86006f720 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe865c0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4083696319 0 0) 0x7fe84c003530 con 0x7fe86006f720 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe86013a2b0 msgr2=0x7fe860142130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe86013a2b0 0x7fe860142130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 >> v1:172.21.15.190:6789/0 conn(0x7fe8600a7030 legacy=0x7fe860139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe860070580 con 0x7fe86006f720 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fe850001990 con 0x7fe86006f720 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: dump: 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.142 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.143 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:50:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 >> v1:172.21.15.98:6789/0 conn(0x7fe86006f720 legacy=0x7fe86013ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient: _add_conns ranks=[8,5,3,7,2,0,1,6,4] 2024-09-11T02:50:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe84c004240 0x7fe84c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): picked mon.e con 0x7fe84c004240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004da0 0x7fe84c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): picked mon.h con 0x7fe84c004da0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe86013a2b0 0x7fe84c00bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): picked mon.g con 0x7fe86013a2b0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe8600a7410 con 0x7fe86013a2b0 2024-09-11T02:50:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fe84c00c410 con 0x7fe84c004da0 2024-09-11T02:50:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe84c00c5c0 con 0x7fe84c004240 2024-09-11T02:50:08.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe857fff640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:50:08.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe84c004240 0x7fe84c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): get_auth_request con 0x7fe84c004240 auth_method 0 2024-09-11T02:50:08.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:08.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:50:08.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe864c09640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004da0 0x7fe84c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient(hunting): handle_auth_done global_id 4392 payload 293 2024-09-11T02:50:08.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 1 -- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004da0 msgr2=0x7fe84c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004da0 0x7fe84c005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 1 -- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe86013a2b0 msgr2=0x7fe84c00bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe86013a2b0 0x7fe84c00bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 1 monclient: found mon.e 2024-09-11T02:50:08.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 1 -- 172.21.15.98:0/310981475 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c003530 con 0x7fe84c004240 2024-09-11T02:50:08.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86540a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 5 monclient: authenticate success, global_id 4392 2024-09-11T02:50:08.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe86766e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:08.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe864c09640 10 monclient: get_auth_request con 0x7fe84c004da0 auth_method 0 2024-09-11T02:50:08.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.127+0000 7fe864c09640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004da0 0x7fe84c005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:08.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86540a640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe84c004240 0x7fe84c004810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe858001790 tx=0x7fe85801c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a762b20163a0f1d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe8580013e0 con 0x7fe84c004240 2024-09-11T02:50:08.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:08.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 10 monclient: dump: 2024-09-11T02:50:08.152 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.152 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.152 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.152 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.152 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.153 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.153 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.153 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.153 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.153 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.153 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.154 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe858003600 con 0x7fe84c004240 2024-09-11T02:50:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 1 -- 172.21.15.98:0/310981475 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe858003ca0 con 0x7fe84c004240 2024-09-11T02:50:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 4 set_mon_vals no callback set 2024-09-11T02:50:08.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe857fff640 10 monclient: dump: 2024-09-11T02:50:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.157 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 10 monclient: shutdown 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 -- 172.21.15.98:0/310981475 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe84c004240 msgr2=0x7fe84c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 --2- 172.21.15.98:0/310981475 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe84c004240 0x7fe84c004810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe858001790 tx=0x7fe85801c3a0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe865c0b640 1 -- 172.21.15.98:0/310981475 reap_dead start 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 -- 172.21.15.98:0/310981475 shutdown_connections 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 -- 172.21.15.98:0/310981475 >> 172.21.15.98:0/310981475 conn(0x7fe8600aa420 msgr2=0x7fe8600a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 -- 172.21.15.98:0/310981475 shutdown_connections 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 -- 172.21.15.98:0/310981475 wait complete. 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 10 monclient: monmap: 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:08.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 librados: starting msgr at 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 librados: starting objecter 2024-09-11T02:50:08.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.131+0000 7fe86766e640 1 Processor -- start 2024-09-11T02:50:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 -- start start 2024-09-11T02:50:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 librados: setting wanted keys 2024-09-11T02:50:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 librados: calling monclient init 2024-09-11T02:50:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient: init 2024-09-11T02:50:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient: _add_conns ranks=[5,6,0,7,4,2,8,1,3] 2024-09-11T02:50:08.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004240 0x7fe8601d1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): picked mon.noname-h con 0x7fe84c004240 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe860122c60 0x7fe860138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): picked mon.noname-d con 0x7fe860122c60 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860139000 0x7fe860120700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): picked mon.noname-a con 0x7fe860139000 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe860142b50 con 0x7fe860139000 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fe8600517e0 con 0x7fe84c004240 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe860142cd0 con 0x7fe860122c60 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient(hunting): authenticate will time out at 962.016746s 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe860122c60 0x7fe860138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe860122c60 0x7fe860138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39506/0 (socket says 172.21.15.98:39506) 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 -- 172.21.15.98:0/3842587695 learned_addr learned my addr 172.21.15.98:0/3842587695 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): get_auth_request con 0x7fe860122c60 auth_method 0 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004240 0x7fe8601d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient(hunting): get_auth_request con 0x7fe84c004240 auth_method 0 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860139000 0x7fe860120700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 10 monclient(hunting): get_auth_request con 0x7fe860139000 auth_method 0 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe865c0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 -- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004240 msgr2=0x7fe8601d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004240 0x7fe8601d1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 -- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860139000 msgr2=0x7fe860120700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860139000 0x7fe860120700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 monclient: found mon.noname-d 2024-09-11T02:50:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 -- 172.21.15.98:0/3842587695 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8600a4b80 con 0x7fe860122c60 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe864c09640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe860122c60 0x7fe860138ac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe85000d7d0 tx=0x7fe85000dbd0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 client.?.objecter ms_handle_connect 0x7fe860122c60 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 1 -- 172.21.15.98:0/3842587695 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe850007540 con 0x7fe860122c60 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: dump: 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.167 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.168 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 1 -- 172.21.15.98:0/3842587695 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8500047f0 con 0x7fe860122c60 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 5 monclient: authenticate success, global_id 4308 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient: _renew_subs 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe860123340 con 0x7fe860122c60 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe8577fe640 4 set_mon_vals no callback set 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 1 -- 172.21.15.98:0/3842587695 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe850004e10 con 0x7fe860122c60 2024-09-11T02:50:08.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient: _renew_subs 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fe860124460 con 0x7fe860122c60 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86766e640 1 librados: init done 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 monclient: dump: 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.170 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 1 -- 172.21.15.98:0/3842587695 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe8500070d0 con 0x7fe860122c60 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe84403ded0 0x7fe844040390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 1 -- 172.21.15.98:0/3842587695 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fe850054060 con 0x7fe860122c60 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 10 client.4308.objecter ms_dispatch 0x7fe860000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 3 client.4308.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 3 client.4308.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:08.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe855ffb640 20 client.4308.objecter dump_active .. 0 homeless 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe84403ded0 0x7fe844040390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.135+0000 7fe86540a640 10 monclient: get_auth_request con 0x7fe84403ded0 auth_method 0 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.139+0000 7fe86766e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.139+0000 7fe86766e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.139+0000 7fe86766e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.139+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe860139ba0 con 0x7fe860122c60 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.139+0000 7fe86540a640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe84403ded0 0x7fe844040390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe8580ac000 tx=0x7fe8580011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.139+0000 7fe855ffb640 10 client.4308.objecter ms_handle_connect 0x7fe84403ded0 2024-09-11T02:50:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.143+0000 7fe855ffb640 1 -- 172.21.15.98:0/3842587695 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe850014430 con 0x7fe860122c60 2024-09-11T02:50:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.143+0000 7fe855ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.143+0000 7fe855ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:08.185 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T02:50:08.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient: get_monmap_and_config 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient: monmap: 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:08.194878+0000 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:08.194878+0000 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.192 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 1 Processor -- start 2024-09-11T02:50:08.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 1 -- start start 2024-09-11T02:50:08.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient: init 2024-09-11T02:50:08.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient: _add_conns ranks=[9,0,6,4,5,8,1,10,3,2,7] 2024-09-11T02:50:08.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f014409b8c0 0x7f014409bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient(hunting): picked mon.noname-a con 0x7f014409b8c0 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:08.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f014409df80 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:08.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01440a44f0 0x7f01441415a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient(hunting): picked mon.noname-d con 0x7f01440a44f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f014409d2c0 con 0x7f014409df80 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f014409d5a0 con 0x7f014409b8c0 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f014409d880 con 0x7f01440a44f0 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014c41c640 10 monclient(hunting): authenticate will time out at 962.074815s 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f014409b8c0 0x7f014409bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f014409b8c0 0x7f014409bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36988/0 (socket says 172.21.15.98:36988) 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 -- 172.21.15.98:0/2788264732 learned_addr learned my addr 172.21.15.98:0/2788264732 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): get_auth_request con 0x7f014409b8c0 auth_method 0 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a9b9640 1 --2- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01440a44f0 0x7f01441415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a9b9640 10 monclient(hunting): get_auth_request con 0x7f01440a44f0 auth_method 0 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a9b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a9b9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a9b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f01491b6640 1 -- 172.21.15.98:0/2788264732 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 625809104 0 0) 0x7f0134001120 con 0x7f014409df80 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f01491b6640 10 monclient(hunting): _negotiate 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f01491b6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f01491b6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f01491b6640 10 monclient(hunting): my global_id is 4419 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f01491b6640 1 -- 172.21.15.98:0/2788264732 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0130003530 con 0x7f014409df80 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient(hunting): handle_auth_done global_id 4410 payload 274 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 -- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01440a44f0 msgr2=0x7f01441415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 --2- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01440a44f0 0x7f01441415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 -- 172.21.15.98:0/2788264732 >> v1:172.21.15.98:6789/0 conn(0x7f014409df80 legacy=0x7f01440a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 monclient: found mon.noname-a 2024-09-11T02:50:08.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.191+0000 7f014a1b8640 1 -- 172.21.15.98:0/2788264732 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01440a5d10 con 0x7f014409b8c0 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014a1b8640 1 --2- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014409b8c0 0x7f014409bca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0140009870 tx=0x7f014001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a826abe266923b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 1 -- 172.21.15.98:0/2788264732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01400b0070 con 0x7f014409b8c0 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: dump: 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.201 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 1 -- 172.21.15.98:0/2788264732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01400044a0 con 0x7f014409b8c0 2024-09-11T02:50:08.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 1 -- 172.21.15.98:0/2788264732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0140004030 con 0x7f014409b8c0 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f01491b6640 10 monclient: dump: 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 5 monclient: authenticate success, global_id 4410 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 4 set_mon_vals no callback set 2024-09-11T02:50:08.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 10 monclient: shutdown 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 -- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014409b8c0 msgr2=0x7f014409bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 --2- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014409b8c0 0x7f014409bca0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0140009870 tx=0x7f014001c3c0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 -- 172.21.15.98:0/2788264732 shutdown_connections 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 --2- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01440a44f0 0x7f01441415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 --2- 172.21.15.98:0/2788264732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f014409b8c0 0x7f014409bca0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0140009870 tx=0x7f014001c3c0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 -- 172.21.15.98:0/2788264732 >> 172.21.15.98:0/2788264732 conn(0x7f01440a4ed0 msgr2=0x7f01440a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 -- 172.21.15.98:0/2788264732 shutdown_connections 2024-09-11T02:50:08.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.195+0000 7f014c41c640 1 -- 172.21.15.98:0/2788264732 wait complete. 2024-09-11T02:50:08.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: monmap: 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.211 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 librados: starting msgr at 2024-09-11T02:50:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 librados: starting objecter 2024-09-11T02:50:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 Processor -- start 2024-09-11T02:50:08.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 -- start start 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 librados: setting wanted keys 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 librados: calling monclient init 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: init 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: _add_conns ranks=[4,2,6,8,0,1,5,3,7] 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f014409df80 0x7f0144071a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): picked mon.noname-c con 0x7f014409df80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440a44f0 0x7f0144071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): picked mon.noname-b con 0x7f01440a44f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01441d1580 0x7f01441d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): picked mon.noname-d con 0x7f01441d1580 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0144144480 con 0x7f01440a44f0 2024-09-11T02:50:08.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0144144300 con 0x7f014409df80 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0144144600 con 0x7f01441d1580 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient(hunting): authenticate will time out at 962.081303s 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01441d1580 0x7f01441d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01441d1580 0x7f01441d19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39520/0 (socket says 172.21.15.98:39520) 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 -- 172.21.15.98:0/4029214969 learned_addr learned my addr 172.21.15.98:0/4029214969 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): get_auth_request con 0x7f01441d1580 auth_method 0 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f014409df80 0x7f0144071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 10 monclient(hunting): get_auth_request con 0x7f014409df80 auth_method 0 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440a44f0 0x7f0144071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 10 monclient(hunting): get_auth_request con 0x7f01440a44f0 auth_method 0 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a1b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01499b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 -- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f014409df80 msgr2=0x7f0144071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f014409df80 0x7f0144071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 -- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440a44f0 msgr2=0x7f0144071fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440a44f0 0x7f0144071fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 monclient: found mon.noname-d 2024-09-11T02:50:08.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 -- 172.21.15.98:0/4029214969 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014409ad80 con 0x7f01441d1580 2024-09-11T02:50:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014a9b9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01441d1580 0x7f01441d19c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f013c007e60 tx=0x7f013c00e4b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 client.?.objecter ms_handle_connect 0x7f01441d1580 2024-09-11T02:50:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 1 -- 172.21.15.98:0/4029214969 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f013c00edd0 con 0x7f01441d1580 2024-09-11T02:50:08.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: dump: 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.218 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 5 monclient: authenticate success, global_id 4326 2024-09-11T02:50:08.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: _renew_subs 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 -- 172.21.15.98:0/4029214969 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0144092990 con 0x7f01441d1580 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: _renew_subs 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 -- 172.21.15.98:0/4029214969 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f01440a1e90 con 0x7f01441d1580 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f014c41c640 1 librados: init done 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 1 -- 172.21.15.98:0/4029214969 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f013c019070 con 0x7f01441d1580 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 1 -- 172.21.15.98:0/4029214969 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f013c014890 con 0x7f01441d1580 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:08.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f01489b5640 4 set_mon_vals no callback set 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f013affd640 10 monclient: dump: 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.221 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f0138ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f0138ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f0138ff9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.199+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0120005090 con 0x7f01441d1580 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 1 -- 172.21.15.98:0/4029214969 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f013c014420 con 0x7f01441d1580 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f012803df40 0x7f0128040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 1 -- 172.21.15.98:0/4029214969 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f013c04f8d0 con 0x7f01441d1580 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 10 client.4326.objecter ms_dispatch 0x7f0144000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:08.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 3 client.4326.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 3 client.4326.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 20 client.4326.objecter dump_active .. 0 homeless 2024-09-11T02:50:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f014a1b8640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f012803df40 0x7f0128040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f014a1b8640 10 monclient: get_auth_request con 0x7f012803df40 auth_method 0 2024-09-11T02:50:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f014a1b8640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f012803df40 0x7f0128040400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01400ad000 tx=0x7f01400a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.203+0000 7f013affd640 10 client.4326.objecter ms_handle_connect 0x7f012803df40 2024-09-11T02:50:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.211+0000 7f013affd640 1 -- 172.21.15.98:0/4029214969 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f013c01c360 con 0x7f01441d1580 2024-09-11T02:50:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.211+0000 7f013affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.211+0000 7f013affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient: get_monmap_and_config 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient: monmap: 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:08.242336+0000 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:08.242336+0000 2024-09-11T02:50:08.239 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:08.240 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 1 Processor -- start 2024-09-11T02:50:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 1 -- start start 2024-09-11T02:50:08.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient: init 2024-09-11T02:50:08.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient: _add_conns ranks=[0,2,6,4,8,3,7,1,5,9,10] 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3510139b80 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f351006fe50 0x7f351013ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient(hunting): picked mon.noname-b con 0x7f351006fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510072640 0x7f3510140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient(hunting): picked mon.noname-d con 0x7f3510072640 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3510071370 con 0x7f3510139b80 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510072640 0x7f3510140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510072640 0x7f3510140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39526/0 (socket says 172.21.15.98:39526) 2024-09-11T02:50:08.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 1 -- 172.21.15.98:0/356364007 learned_addr learned my addr 172.21.15.98:0/356364007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f35100719c0 con 0x7f351006fe50 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 1 -- 172.21.15.98:0/356364007 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3510071c10 con 0x7f3510072640 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3516c94640 10 monclient(hunting): authenticate will time out at 962.122649s 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 10 monclient(hunting): get_auth_request con 0x7f3510072640 auth_method 0 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 1 --2- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f351006fe50 0x7f351013ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): get_auth_request con 0x7f351006fe50 auth_method 0 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 1 -- 172.21.15.98:0/356364007 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121949457 0 0) 0x7f3510071370 con 0x7f3510139b80 2024-09-11T02:50:08.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 10 monclient(hunting): _negotiate 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 10 monclient(hunting): my global_id is 4428 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 1 -- 172.21.15.98:0/356364007 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3500003530 con 0x7f3510139b80 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 1 -- 172.21.15.98:0/356364007 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4144464430 0 0) 0x7f3500003530 con 0x7f3510139b80 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 1 -- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510072640 msgr2=0x7f3510140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 1 --2- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510072640 0x7f3510140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 1 -- 172.21.15.98:0/356364007 >> v1:172.21.15.98:6789/0 conn(0x7f3510139b80 legacy=0x7f3510139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 1 monclient: found mon.noname-b 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350ffff640 1 -- 172.21.15.98:0/356364007 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35100a9600 con 0x7f351006fe50 2024-09-11T02:50:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f3515231640 1 --2- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510072640 0x7f3510140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:08.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.239+0000 7f350f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:50:08.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350ffff640 1 --2- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f351006fe50 0x7f351013ea00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0097f0 tx=0x7f34fc01c2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8adc7792f5254c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 1 -- 172.21.15.98:0/356364007 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34fc0b0070 con 0x7f351006fe50 2024-09-11T02:50:08.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:08.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: dump: 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.251 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 1 -- 172.21.15.98:0/356364007 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34fc0aa810 con 0x7f351006fe50 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 1 -- 172.21.15.98:0/356364007 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3504001970 con 0x7f351006fe50 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f350f7fe640 10 monclient: dump: 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.252 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.253 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f3516c94640 5 monclient: authenticate success, global_id 4277 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f3516c94640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.243+0000 7f3516c94640 4 set_mon_vals no callback set 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient: shutdown 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f351006fe50 msgr2=0x7f351013ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 --2- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f351006fe50 0x7f351013ea00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0097f0 tx=0x7f34fc01c2c0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/356364007 shutdown_connections 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 --2- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510072640 0x7f3510140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 --2- 172.21.15.98:0/356364007 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f351006fe50 0x7f351013ea00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/356364007 >> 172.21.15.98:0/356364007 conn(0x7f35100ac730 msgr2=0x7f35100a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/356364007 shutdown_connections 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/356364007 wait complete. 2024-09-11T02:50:08.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient: monmap: 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.255 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 librados: starting msgr at 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 librados: starting objecter 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 Processor -- start 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- start start 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 librados: setting wanted keys 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 librados: calling monclient init 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient: init 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient: _add_conns ranks=[8,5,6,1,7,0,4,3,2] 2024-09-11T02:50:08.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f351006fe50 0x7f35101d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): picked mon.noname-e con 0x7f351006fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3510072640 0x7f35101d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f351006fe50 0x7f35101d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f351006fe50 0x7f35101d10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34966/0 (socket says 172.21.15.98:34966) 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 1 -- 172.21.15.98:0/1657631588 learned_addr learned my addr 172.21.15.98:0/1657631588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): picked mon.noname-h con 0x7f3510072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510139b80 0x7f351013d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): picked mon.noname-d con 0x7f3510139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:08.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3510143ec0 con 0x7f3510072640 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f35100702d0 con 0x7f3510139b80 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3510144040 con 0x7f351006fe50 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3516c94640 10 monclient(hunting): authenticate will time out at 962.131255s 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): get_auth_request con 0x7f351006fe50 auth_method 0 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510139b80 0x7f351013d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 10 monclient(hunting): get_auth_request con 0x7f3510139b80 auth_method 0 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3515231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 1 -- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510139b80 msgr2=0x7f351013d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510139b80 0x7f351013d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f350ffff640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3510072640 0x7f35101d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 1 -- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3510072640 msgr2=0x7f35101d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.247+0000 7f3514a30640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3510072640 0x7f35101d17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3514a30640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3514a30640 1 monclient: found mon.noname-e 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3514a30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3514a30640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3514a30640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35101395e0 con 0x7f351006fe50 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3515231640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510139b80 0x7f351013d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:08.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3514a30640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f351006fe50 0x7f35101d10c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f350400c6b0 tx=0x7f350400cb70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 client.?.objecter ms_handle_connect 0x7f351006fe50 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350ffff640 10 monclient: get_auth_request con 0x7f3510072640 auth_method 0 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350ffff640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3510072640 0x7f35101d17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 1 -- 172.21.15.98:0/1657631588 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3504016020 con 0x7f351006fe50 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: dump: 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.261 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 1 -- 172.21.15.98:0/1657631588 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f350400aab0 con 0x7f351006fe50 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 1 -- 172.21.15.98:0/1657631588 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f350400ace0 con 0x7f351006fe50 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 10 monclient: dump: 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.263 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350effd640 4 set_mon_vals no callback set 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 5 monclient: authenticate success, global_id 4319 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 monclient: _renew_subs 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35100a6e90 con 0x7f351006fe50 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 monclient: _renew_subs 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3510122a30 con 0x7f351006fe50 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 1 librados: init done 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3510070dc0 con 0x7f351006fe50 2024-09-11T02:50:08.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 1 -- 172.21.15.98:0/1657631588 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f350400a5e0 con 0x7f351006fe50 2024-09-11T02:50:08.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.251+0000 7f350d7fa640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f403df40 0x7f34f4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.255+0000 7f350ffff640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f403df40 0x7f34f4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.255+0000 7f350ffff640 10 monclient: get_auth_request con 0x7f34f403df40 auth_method 0 2024-09-11T02:50:08.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.255+0000 7f350ffff640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f403df40 0x7f34f4040400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0ad000 tx=0x7f34fc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7f350d7fa640 10 client.4319.objecter ms_handle_connect 0x7f34f403df40 2024-09-11T02:50:08.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7f350d7fa640 1 -- 172.21.15.98:0/1657631588 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f3504012c70 con 0x7f351006fe50 2024-09-11T02:50:08.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7f350d7fa640 10 client.4319.objecter ms_dispatch 0x7f3510000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:08.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7f350d7fa640 3 client.4319.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:08.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7f350d7fa640 3 client.4319.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:08.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7f350d7fa640 20 client.4319.objecter dump_active .. 0 homeless 2024-09-11T02:50:08.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7fbac9969640 10 monclient: get_monmap_and_config 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7fbac9969640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7fbac9969640 10 monclient: monmap: 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:08.264233+0000 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:08.264233+0000 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.270 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.259+0000 7fbac9969640 1 Processor -- start 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 1 -- start start 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient: init 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7f350d7fa640 1 -- 172.21.15.98:0/1657631588 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3510070dc0 con 0x7f351006fe50 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7f350d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7f350d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient: _add_conns ranks=[9,5,6,7,10,8,0,3,4,2,1] 2024-09-11T02:50:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbabc0250f0 0x7fbabc0254d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): picked mon.noname-a con 0x7fbabc0250f0 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0bcd20 0x7fbabc0bd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): picked mon.noname-h con 0x7fbabc0bcd20 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0262d0 0x7fbabc026740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): picked mon.noname-d con 0x7fbabc0262d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbabc0277a0 con 0x7fbabc0250f0 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbabc027520 con 0x7fbabc0bcd20 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbabc028840 con 0x7fbabc0262d0 2024-09-11T02:50:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac9969640 10 monclient(hunting): authenticate will time out at 962.144958s 2024-09-11T02:50:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbabc0250f0 0x7fbabc0254d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbabc0250f0 0x7fbabc0254d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37002/0 (socket says 172.21.15.98:37002) 2024-09-11T02:50:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 -- 172.21.15.98:0/1518696975 learned_addr learned my addr 172.21.15.98:0/1518696975 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): get_auth_request con 0x7fbabc0250f0 auth_method 0 2024-09-11T02:50:08.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0bcd20 0x7fbabc0bd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 10 monclient(hunting): get_auth_request con 0x7fbabc0bcd20 auth_method 0 2024-09-11T02:50:08.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient(hunting): handle_auth_done global_id 4437 payload 274 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 -- 172.21.15.98:0/1518696975 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0262d0 msgr2=0x7fbabc026740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0262d0 0x7fbabc026740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 -- 172.21.15.98:0/1518696975 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0bcd20 msgr2=0x7fbabc0bd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0bcd20 0x7fbabc0bd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 monclient: found mon.noname-a 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac8967640 1 -- 172.21.15.98:0/1518696975 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc02ab50 con 0x7fbabc0250f0 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.263+0000 7fbac3fff640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0bcd20 0x7fbabc0bd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac8967640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc0250f0 0x7fbabc0254d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbab8006fd0 tx=0x7fbab801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6720d81a9e15b930 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 1 -- 172.21.15.98:0/1518696975 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbab80b2070 con 0x7fbabc0250f0 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:08.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: dump: 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 1 -- 172.21.15.98:0/1518696975 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbab80a9070 con 0x7fbabc0250f0 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 1 -- 172.21.15.98:0/1518696975 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbab80abc40 con 0x7fbabc0250f0 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac37fe640 10 monclient: dump: 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.285 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.286 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 5 monclient: authenticate success, global_id 4437 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 4 set_mon_vals no callback set 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 10 monclient: shutdown 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 -- 172.21.15.98:0/1518696975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc0250f0 msgr2=0x7fbabc0254d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc0250f0 0x7fbabc0254d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbab8006fd0 tx=0x7fbab801c420 comp rx=0 tx=0).stop 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 -- 172.21.15.98:0/1518696975 shutdown_connections 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0262d0 0x7fbabc026740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0bcd20 0x7fbabc0bd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 --2- 172.21.15.98:0/1518696975 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbabc0250f0 0x7fbabc0254d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 -- 172.21.15.98:0/1518696975 >> 172.21.15.98:0/1518696975 conn(0x7fbabc007bd0 msgr2=0x7fbabc007fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 -- 172.21.15.98:0/1518696975 shutdown_connections 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.267+0000 7fbac9969640 1 -- 172.21.15.98:0/1518696975 wait complete. 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient: monmap: 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:08.288 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.289 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 librados: starting msgr at 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 librados: starting objecter 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 Processor -- start 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 -- start start 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 librados: setting wanted keys 2024-09-11T02:50:08.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 librados: calling monclient init 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient: init 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient: _add_conns ranks=[3,5,6,0,8,1,7,2,4] 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0250f0 0x7fbabc154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): picked mon.noname-g con 0x7fbabc0250f0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0262d0 0x7fbabc0c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): picked mon.noname-h con 0x7fbabc0262d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0bcd20 0x7fbabc0bda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): picked mon.noname-d con 0x7fbabc0bcd20 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbabc0c67f0 con 0x7fbabc0250f0 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbabc0c6670 con 0x7fbabc0262d0 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbabc0c6970 con 0x7fbabc0bcd20 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9969640 10 monclient(hunting): authenticate will time out at 962.153737s 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0bcd20 0x7fbabc0bda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0bcd20 0x7fbabc0bda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39560/0 (socket says 172.21.15.98:39560) 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 -- 172.21.15.98:0/2097079236 learned_addr learned my addr 172.21.15.98:0/2097079236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): get_auth_request con 0x7fbabc0bcd20 auth_method 0 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0262d0 0x7fbabc0c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 10 monclient(hunting): get_auth_request con 0x7fbabc0262d0 auth_method 0 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-09-11T02:50:08.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 -- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0262d0 msgr2=0x7fbabc0c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0262d0 0x7fbabc0c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 -- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0250f0 msgr2=0x7fbabc154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac8967640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0250f0 0x7fbabc154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0250f0 0x7fbabc154900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 monclient: found mon.noname-d 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 -- 172.21.15.98:0/2097079236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbabc0bd190 con 0x7fbabc0bcd20 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac9168640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0bcd20 0x7fbabc0bda50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbac406b090 tx=0x7fbac4074f50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 client.?.objecter ms_handle_connect 0x7fbabc0bcd20 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac8967640 10 monclient: get_auth_request con 0x7fbabc0250f0 auth_method 0 2024-09-11T02:50:08.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac8967640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0250f0 0x7fbabc154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 1 -- 172.21.15.98:0/2097079236 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbac4093070 con 0x7fbabc0bcd20 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 monclient: dump: 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 1 -- 172.21.15.98:0/2097079236 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac4050ef0 con 0x7fbabc0bcd20 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.271+0000 7fbac2ffd640 4 set_mon_vals no callback set 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 5 monclient: authenticate success, global_id 4353 2024-09-11T02:50:08.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 monclient: _renew_subs 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 1 -- 172.21.15.98:0/2097079236 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbac406b5d0 con 0x7fbabc0bcd20 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbabc000bd0 con 0x7fbabc0bcd20 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 10 monclient: dump: 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.297 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 monclient: _renew_subs 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fbabc0bf3b0 con 0x7fbabc0bcd20 2024-09-11T02:50:08.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 1 -- 172.21.15.98:0/2097079236 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbac4075080 con 0x7fbabc0bcd20 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbaa803dea0 0x7fbaa8040360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 1 -- 172.21.15.98:0/2097079236 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fbac40cacb0 con 0x7fbabc0bcd20 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 10 client.4353.objecter ms_dispatch 0x7fbac4000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 1 librados: init done 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 3 client.4353.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 3 client.4353.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 20 client.4353.objecter dump_active .. 0 homeless 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac8967640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbaa803dea0 0x7fbaa8040360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac8967640 10 monclient: get_auth_request con 0x7fbaa803dea0 auth_method 0 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbabc026740 con 0x7fbabc0bcd20 2024-09-11T02:50:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac17fa640 10 client.4353.objecter ms_handle_connect 0x7fbaa803dea0 2024-09-11T02:50:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.275+0000 7fbac8967640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbaa803dea0 0x7fbaa8040360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbab8006fd0 tx=0x7fbab80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.283+0000 7fbac17fa640 1 -- 172.21.15.98:0/2097079236 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbac4096d70 con 0x7fbabc0bcd20 2024-09-11T02:50:08.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.287+0000 7fbac17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.287+0000 7fbac17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.323+0000 7fc22653f640 10 monclient: get_monmap_and_config 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.323+0000 7fc22653f640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.323+0000 7fc22653f640 10 monclient: monmap: 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:08.328184+0000 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:08.328184+0000 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.325 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.326 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:08.327 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.323+0000 7fc22653f640 1 Processor -- start 2024-09-11T02:50:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.323+0000 7fc22653f640 1 -- start start 2024-09-11T02:50:08.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.323+0000 7fc22653f640 10 monclient: init 2024-09-11T02:50:08.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient: _add_conns ranks=[0,7,5,2,9,10,4,1,3,8,6] 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc220139b80 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc22006fe50 0x7fc22013ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient(hunting): picked mon.noname-i con 0x7fc22006fe50 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220072640 0x7fc220140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient(hunting): picked mon.noname-h con 0x7fc220072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc220071370 con 0x7fc220139b80 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc2200719c0 con 0x7fc220072640 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc220071c10 con 0x7fc22006fe50 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc22653f640 10 monclient(hunting): authenticate will time out at 962.208449s 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc220139b80 0x7fc220139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50942/0 (socket says 172.21.15.98:50942) 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21ffff640 1 -- 172.21.15.98:0/4116928983 learned_addr learned my addr 172.21.15.98:0/4116928983 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc22006fe50 0x7fc22013ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 10 monclient(hunting): get_auth_request con 0x7fc22006fe50 auth_method 0 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220072640 0x7fc220140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): get_auth_request con 0x7fc220072640 auth_method 0 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 1 -- 172.21.15.98:0/4116928983 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 15009504 0 0) 0x7fc220071370 con 0x7fc220139b80 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 10 monclient(hunting): _negotiate 2024-09-11T02:50:08.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 10 monclient(hunting): my global_id is 4446 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 1 -- 172.21.15.98:0/4116928983 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc208003530 con 0x7fc220139b80 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 1 -- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc22006fe50 msgr2=0x7fc22013ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 1 -- 172.21.15.98:0/4116928983 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3446603475 0 0) 0x7fc208003530 con 0x7fc220139b80 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc22006fe50 0x7fc22013ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 1 -- 172.21.15.98:0/4116928983 >> v1:172.21.15.98:6789/0 conn(0x7fc220139b80 legacy=0x7fc220139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 1 monclient: found mon.noname-h 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc224adc640 1 -- 172.21.15.98:0/4116928983 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2200a9600 con 0x7fc220072640 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21f7fe640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc22006fe50 0x7fc22013ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:08.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.327+0000 7fc21effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:50:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T02:50:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T02:50:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe860069cc0 con 0x7fe860122c60 2024-09-11T02:50:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc224adc640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220072640 0x7fc220140f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc218009dc0 tx=0x7fc21801c770 comp rx=0 tx=0).ready entity=mon.5 client_cookie=50b2b9d0c8c102ca server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe855ffb640 1 -- 172.21.15.98:0/3842587695 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe850010cc0 con 0x7fe860122c60 2024-09-11T02:50:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe855ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-11T02:50:08.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe855ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:08.336 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-09-11T02:50:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 1 -- 172.21.15.98:0/4116928983 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc2180ad070 con 0x7fc220072640 2024-09-11T02:50:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: dump: 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.338 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 1 -- 172.21.15.98:0/4116928983 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2180a9440 con 0x7fc220072640 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 1 -- 172.21.15.98:0/4116928983 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc210001970 con 0x7fc220072640 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc21effd640 10 monclient: dump: 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.339 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.340 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 5 monclient: authenticate success, global_id 4325 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 4 set_mon_vals no callback set 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 10 monclient: shutdown 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 -- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220072640 msgr2=0x7fc220140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220072640 0x7fc220140f50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc218009dc0 tx=0x7fc21801c770 comp rx=0 tx=0).stop 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 -- 172.21.15.98:0/4116928983 shutdown_connections 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220072640 0x7fc220140f50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 --2- 172.21.15.98:0/4116928983 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc22006fe50 0x7fc22013ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 -- 172.21.15.98:0/4116928983 >> 172.21.15.98:0/4116928983 conn(0x7fc2200ac730 msgr2=0x7fc2200a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 -- 172.21.15.98:0/4116928983 shutdown_connections 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 -- 172.21.15.98:0/4116928983 wait complete. 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 10 librados: watch_flush enter 2024-09-11T02:50:08.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 10 librados: watch_flush exit 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 20 client.4308.objecter shutdown clearing up homeless session... 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 10 client.4308.objecter successfully canceled tick 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 10 monclient: monmap: 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:08.343 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.344 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe84403ded0 msgr2=0x7fe844040390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fe86766e640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe84403ded0 0x7fe844040390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe8580ac000 tx=0x7fe8580011a0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 10 monclient: shutdown 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe860122c60 msgr2=0x7fe860138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe860122c60 0x7fe860138ac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe85000d7d0 tx=0x7fe85000dbd0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 librados: starting msgr at 2024-09-11T02:50:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 librados: starting objecter 2024-09-11T02:50:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.331+0000 7fc22653f640 1 Processor -- start 2024-09-11T02:50:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 -- start start 2024-09-11T02:50:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 librados: setting wanted keys 2024-09-11T02:50:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 librados: calling monclient init 2024-09-11T02:50:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: init 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 shutdown_connections 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe84403ded0 0x7fe844040390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe860139000 0x7fe860120700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe860122c60 0x7fe860138ac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 --2- 172.21.15.98:0/3842587695 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe84c004240 0x7fe8601d1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 >> 172.21.15.98:0/3842587695 conn(0x7fe8600aa420 msgr2=0x7fe8600a3690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 shutdown_connections 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 -- 172.21.15.98:0/3842587695 wait complete. 2024-09-11T02:50:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fe86766e640 1 librados: shutdown 2024-09-11T02:50:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _add_conns ranks=[8,4,5,2,7,0,1,6,3] 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc22006fe50 0x7fc220091820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): picked mon.noname-e con 0x7fc22006fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc220072640 0x7fc220090f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): picked mon.noname-c con 0x7fc220072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220139b80 0x7fc22013d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): picked mon.noname-h con 0x7fc220139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc220144040 con 0x7fc220072640 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc220143ec0 con 0x7fc220139b80 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc2201441c0 con 0x7fc22006fe50 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient(hunting): authenticate will time out at 962.216962s 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc22006fe50 0x7fc220091820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc22006fe50 0x7fc220091820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34970/0 (socket says 172.21.15.98:34970) 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 -- 172.21.15.98:0/3788016054 learned_addr learned my addr 172.21.15.98:0/3788016054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): get_auth_request con 0x7fc22006fe50 auth_method 0 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc220072640 0x7fc220090f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 10 monclient(hunting): get_auth_request con 0x7fc220072640 auth_method 0 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc224adc640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220139b80 0x7fc22013d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc224adc640 10 monclient(hunting): get_auth_request con 0x7fc220139b80 auth_method 0 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc224adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc224adc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc224adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 -- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220139b80 msgr2=0x7fc22013d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220139b80 0x7fc22013d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 -- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc220072640 msgr2=0x7fc220090f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc220072640 0x7fc220090f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 monclient: found mon.noname-e 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 -- 172.21.15.98:0/3788016054 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22013f420 con 0x7fc22006fe50 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21f7fe640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc220072640 0x7fc220090f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21ffff640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc22006fe50 0x7fc220091820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc21000c680 tx=0x7fc21000cb40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 client.?.objecter ms_handle_connect 0x7fc22006fe50 2024-09-11T02:50:08.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 1 -- 172.21.15.98:0/3788016054 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc210005d50 con 0x7fc22006fe50 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: dump: 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.356 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 1 -- 172.21.15.98:0/3788016054 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc210007070 con 0x7fc22006fe50 2024-09-11T02:50:08.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 5 monclient: authenticate success, global_id 4328 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _renew_subs 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2200a88b0 con 0x7fc22006fe50 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _renew_subs 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2200a8de0 con 0x7fc22006fe50 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 librados: init done 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21e7fc640 4 set_mon_vals no callback set 2024-09-11T02:50:08.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 1 -- 172.21.15.98:0/3788016054 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc210007690 con 0x7fc22006fe50 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc200005090 con 0x7fc22006fe50 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 10 monclient: dump: 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.359 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.360 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 1 -- 172.21.15.98:0/3788016054 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc2100058e0 con 0x7fc22006fe50 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.335+0000 7fc21cff9640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc20c03ded0 0x7fc20c040390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21cff9640 1 -- 172.21.15.98:0/3788016054 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fc21004f150 con 0x7fc22006fe50 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21cff9640 10 client.4328.objecter ms_dispatch 0x7fc220000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21cff9640 3 client.4328.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21cff9640 3 client.4328.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21cff9640 20 client.4328.objecter dump_active .. 0 homeless 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21f7fe640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc20c03ded0 0x7fc20c040390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21f7fe640 10 monclient: get_auth_request con 0x7fc20c03ded0 auth_method 0 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21cff9640 10 client.4328.objecter ms_handle_connect 0x7fc20c03ded0 2024-09-11T02:50:08.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.339+0000 7fc21f7fe640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc20c03ded0 0x7fc20c040390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc22013e9b0 tx=0x7fc214009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.343+0000 7fbd337ca640 10 monclient: get_monmap_and_config 2024-09-11T02:50:08.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.343+0000 7fbd337ca640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.343+0000 7fbd337ca640 10 monclient: monmap: 2024-09-11T02:50:08.362 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.362 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:08.348643+0000 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:08.348643+0000 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.363 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:08.364 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.343+0000 7fc21cff9640 1 -- 172.21.15.98:0/3788016054 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fc2200a88b0 con 0x7fc22006fe50 2024-09-11T02:50:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fc21cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fc21cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 Processor -- start 2024-09-11T02:50:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 -- start start 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient: init 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient: _add_conns ranks=[3,2,9,6,10,4,8,5,0,7,1] 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c136780 0x7fbd2c073750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): picked mon.noname-g con 0x7fbd2c136780 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136b60 0x7fbd2c136f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): picked mon.noname-b con 0x7fbd2c136b60 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbd2c0a4610 0x7fbd2c0a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): picked mon.noname-a con 0x7fbd2c0a4610 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbd2c072db0 con 0x7fbd2c0a4610 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2c0713e0 con 0x7fbd2c136b60 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2c0700d0 con 0x7fbd2c136780 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd337ca640 10 monclient(hunting): authenticate will time out at 962.229234s 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136b60 0x7fbd2c136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136b60 0x7fbd2c136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48100/0 (socket says 172.21.15.98:48100) 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 -- 172.21.15.98:0/1500775782 learned_addr learned my addr 172.21.15.98:0/1500775782 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): get_auth_request con 0x7fbd2c136b60 auth_method 0 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 -- 172.21.15.98:0/1500775782 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c136780 msgr2=0x7fbd2c073750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 --2- 172.21.15.98:0/1500775782 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c136780 0x7fbd2c073750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 -- 172.21.15.98:0/1500775782 >> v2:172.21.15.98:3300/0 conn(0x7fbd2c0a4610 msgr2=0x7fbd2c0a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 --2- 172.21.15.98:0/1500775782 >> v2:172.21.15.98:3300/0 conn(0x7fbd2c0a4610 0x7fbd2c0a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 monclient: found mon.noname-b 2024-09-11T02:50:08.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.347+0000 7fbd30d65640 1 -- 172.21.15.98:0/1500775782 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2c073c90 con 0x7fbd2c136b60 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd30d65640 1 --2- 172.21.15.98:0/1500775782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136b60 0x7fbd2c136f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0098a0 tx=0x7fbd1c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f46090a40e11965 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 1 -- 172.21.15.98:0/1500775782 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbd1c0b0070 con 0x7fbd2c136b60 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 10 monclient: dump: 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.374 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.375 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 5 monclient: authenticate success, global_id 4286 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 1 -- 172.21.15.98:0/1500775782 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0aa810 con 0x7fbd2c136b60 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd23fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 4 set_mon_vals no callback set 2024-09-11T02:50:08.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 10 monclient: shutdown 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 -- 172.21.15.98:0/1500775782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136b60 msgr2=0x7fbd2c136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1500775782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136b60 0x7fbd2c136f40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0098a0 tx=0x7fbd1c01c440 comp rx=0 tx=0).stop 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 -- 172.21.15.98:0/1500775782 shutdown_connections 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1500775782 >> v2:172.21.15.98:3300/0 conn(0x7fbd2c0a4610 0x7fbd2c0a4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1500775782 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136b60 0x7fbd2c136f40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1500775782 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c136780 0x7fbd2c073750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 -- 172.21.15.98:0/1500775782 >> 172.21.15.98:0/1500775782 conn(0x7fbd2c06ed00 msgr2=0x7fbd2c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 -- 172.21.15.98:0/1500775782 shutdown_connections 2024-09-11T02:50:08.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.351+0000 7fbd337ca640 1 -- 172.21.15.98:0/1500775782 wait complete. 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: monmap: 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.378 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 librados: starting msgr at 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 librados: starting objecter 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 Processor -- start 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 -- start start 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 librados: setting wanted keys 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 librados: calling monclient init 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: init 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: _add_conns ranks=[3,2,7,4,5,0,1,6,8] 2024-09-11T02:50:08.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c0a4610 0x7fbd2c1384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): picked mon.noname-g con 0x7fbd2c0a4610 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136780 0x7fbd2c138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): picked mon.noname-b con 0x7fbd2c136780 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbd2c138f40 0x7fbd2c1393a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): picked mon.noname-i con 0x7fbd2c138f40 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2c140090 con 0x7fbd2c136780 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2c13ff10 con 0x7fbd2c0a4610 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbd2c140210 con 0x7fbd2c138f40 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient(hunting): authenticate will time out at 962.237462s 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136780 0x7fbd2c138a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136780 0x7fbd2c138a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48102/0 (socket says 172.21.15.98:48102) 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 -- 172.21.15.98:0/1598929846 learned_addr learned my addr 172.21.15.98:0/1598929846 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): get_auth_request con 0x7fbd2c136780 auth_method 0 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31566640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c0a4610 0x7fbd2c1384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31566640 10 monclient(hunting): get_auth_request con 0x7fbd2c0a4610 auth_method 0 2024-09-11T02:50:08.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31566640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31566640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31566640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31d67640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbd2c138f40 0x7fbd2c1393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 -- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbd2c138f40 msgr2=0x7fbd2c1393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbd2c138f40 0x7fbd2c1393a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 -- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c0a4610 msgr2=0x7fbd2c1384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c0a4610 0x7fbd2c1384c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 monclient: found mon.noname-b 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:08.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 -- 172.21.15.98:0/1598929846 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2c0a7da0 con 0x7fbd2c136780 2024-09-11T02:50:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31566640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c0a4610 0x7fbd2c1384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:50:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31d67640 10 monclient: get_auth_request con 0x7fbd2c138f40 auth_method 0 2024-09-11T02:50:08.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd31d67640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbd2c138f40 0x7fbd2c1393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:08.384 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.2 2024-09-11T02:50:08.384 DEBUG:teuthology.parallel:result is None 2024-09-11T02:50:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd30d65640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136780 0x7fbd2c138a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c01c270 tx=0x7fbd1c0191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 client.?.objecter ms_handle_connect 0x7fbd2c136780 2024-09-11T02:50:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 1 -- 172.21.15.98:0/1598929846 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbd1c0ba070 con 0x7fbd2c136780 2024-09-11T02:50:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:08.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: dump: 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.387 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 1 -- 172.21.15.98:0/1598929846 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0b1040 con 0x7fbd2c136780 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 1 -- 172.21.15.98:0/1598929846 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbd1c0b0070 con 0x7fbd2c136780 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd21ffb640 10 monclient: dump: 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.388 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.389 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd237fe640 4 set_mon_vals no callback set 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 5 monclient: authenticate success, global_id 4295 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: _renew_subs 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd2c13d370 con 0x7fbd2c136780 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: _renew_subs 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd2c139f80 con 0x7fbd2c136780 2024-09-11T02:50:08.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.355+0000 7fbd337ca640 1 librados: init done 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd337ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd337ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd337ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd2c0a4a90 con 0x7fbd2c136780 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 1 -- 172.21.15.98:0/1598929846 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbd1c004040 con 0x7fbd2c136780 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbd1003df70 0x7fbd10040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 1 -- 172.21.15.98:0/1598929846 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fbd1c0e8680 con 0x7fbd2c136780 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 10 client.4295.objecter ms_dispatch 0x7fbd2c000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 3 client.4295.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 3 client.4295.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 20 client.4295.objecter dump_active .. 0 homeless 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd31566640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbd1003df70 0x7fbd10040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd31566640 10 monclient: get_auth_request con 0x7fbd1003df70 auth_method 0 2024-09-11T02:50:08.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd21ffb640 10 client.4295.objecter ms_handle_connect 0x7fbd1003df70 2024-09-11T02:50:08.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.359+0000 7fbd31566640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbd1003df70 0x7fbd10040430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd28009890 tx=0x7fbd28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.367+0000 7fbd21ffb640 1 -- 172.21.15.98:0/1598929846 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbd1c016150 con 0x7fbd2c136780 2024-09-11T02:50:08.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.367+0000 7fbd21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.367+0000 7fbd21ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.463+0000 7f0138ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T02:50:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.463+0000 7f0138ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T02:50:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.463+0000 7f0138ff9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.463+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0120005380 con 0x7f01441d1580 2024-09-11T02:50:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.463+0000 7f013affd640 1 -- 172.21.15.98:0/4029214969 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f013c01e980 con 0x7f01441d1580 2024-09-11T02:50:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.463+0000 7f013affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-11T02:50:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.463+0000 7f013affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:08.466 INFO:teuthology.orchestra.run.smithi098.stdout:51539607557 2024-09-11T02:50:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 10 librados: watch_flush enter 2024-09-11T02:50:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 10 librados: watch_flush exit 2024-09-11T02:50:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 20 client.4326.objecter shutdown clearing up homeless session... 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 10 client.4326.objecter successfully canceled tick 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f012803df40 msgr2=0x7f0128040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f012803df40 0x7f0128040400 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f01400ad000 tx=0x7f01400a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 10 monclient: shutdown 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01441d1580 msgr2=0x7f01441d19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01441d1580 0x7f01441d19c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f013c007e60 tx=0x7f013c00e4b0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 shutdown_connections 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01441d1580 0x7f01441d19c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f012803df40 0x7f0128040400 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01440a44f0 0x7f0144071fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 --2- 172.21.15.98:0/4029214969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f014409df80 0x7f0144071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 >> 172.21.15.98:0/4029214969 conn(0x7f01440a4ed0 msgr2=0x7f01440a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 shutdown_connections 2024-09-11T02:50:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 -- 172.21.15.98:0/4029214969 wait complete. 2024-09-11T02:50:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.467+0000 7f0138ff9640 1 librados: shutdown 2024-09-11T02:50:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.479+0000 7f3516c94640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T02:50:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.479+0000 7f3516c94640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T02:50:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.479+0000 7f3516c94640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.479+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f35100a8c70 con 0x7f351006fe50 2024-09-11T02:50:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.479+0000 7f350d7fa640 1 -- 172.21.15.98:0/1657631588 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f35100a8c70 con 0x7f351006fe50 2024-09-11T02:50:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.479+0000 7f350d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-11T02:50:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.479+0000 7f350d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:08.481 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-09-11T02:50:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 10 librados: watch_flush enter 2024-09-11T02:50:08.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 10 librados: watch_flush exit 2024-09-11T02:50:08.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 20 client.4319.objecter shutdown clearing up homeless session... 2024-09-11T02:50:08.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 10 client.4319.objecter successfully canceled tick 2024-09-11T02:50:08.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f403df40 msgr2=0x7f34f4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f403df40 0x7f34f4040400 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f34fc0ad000 tx=0x7f34fc0a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:08.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 10 monclient: shutdown 2024-09-11T02:50:08.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f351006fe50 msgr2=0x7f35101d10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f351006fe50 0x7f35101d10c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f350400c6b0 tx=0x7f350400cb70 comp rx=0 tx=0).stop 2024-09-11T02:50:08.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 shutdown_connections 2024-09-11T02:50:08.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f34f403df40 0x7f34f4040400 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3510139b80 0x7f351013d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3510072640 0x7f35101d17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 --2- 172.21.15.98:0/1657631588 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f351006fe50 0x7f35101d10c0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f350400c6b0 tx=0x7f350400cb70 comp rx=0 tx=0).stop 2024-09-11T02:50:08.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 >> 172.21.15.98:0/1657631588 conn(0x7f35100ac730 msgr2=0x7f35100a7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.488 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-11T02:50:08.489 DEBUG:teuthology.parallel:result is None 2024-09-11T02:50:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 shutdown_connections 2024-09-11T02:50:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 -- 172.21.15.98:0/1657631588 wait complete. 2024-09-11T02:50:08.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.483+0000 7f3516c94640 1 librados: shutdown 2024-09-11T02:50:08.498 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.3 2024-09-11T02:50:08.498 DEBUG:teuthology.parallel:result is None 2024-09-11T02:50:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.511+0000 7fbac9969640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T02:50:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.511+0000 7fbac9969640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T02:50:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.511+0000 7fbac9969640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.511+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbabc009a10 con 0x7fbabc0bcd20 2024-09-11T02:50:08.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.511+0000 7fbac17fa640 1 -- 172.21.15.98:0/2097079236 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbac4076ce0 con 0x7fbabc0bcd20 2024-09-11T02:50:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.511+0000 7fbac17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-11T02:50:08.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.511+0000 7fbac17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:08.512 INFO:teuthology.orchestra.run.smithi098.stdout:47244640260 2024-09-11T02:50:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 10 librados: watch_flush enter 2024-09-11T02:50:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 10 librados: watch_flush exit 2024-09-11T02:50:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 20 client.4353.objecter shutdown clearing up homeless session... 2024-09-11T02:50:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 10 client.4353.objecter successfully canceled tick 2024-09-11T02:50:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbaa803dea0 msgr2=0x7fbaa8040360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbaa803dea0 0x7fbaa8040360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbab8006fd0 tx=0x7fbab80a7000 comp rx=0 tx=0).stop 2024-09-11T02:50:08.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 10 monclient: shutdown 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0bcd20 msgr2=0x7fbabc0bda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0bcd20 0x7fbabc0bda50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbac406b090 tx=0x7fbac4074f50 comp rx=0 tx=0).stop 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 shutdown_connections 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbaa803dea0 0x7fbaa8040360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbabc0262d0 0x7fbabc0c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbabc0bcd20 0x7fbabc0bda50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 --2- 172.21.15.98:0/2097079236 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbabc0250f0 0x7fbabc154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 >> 172.21.15.98:0/2097079236 conn(0x7fbabc007bd0 msgr2=0x7fbabc02e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 shutdown_connections 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 -- 172.21.15.98:0/2097079236 wait complete. 2024-09-11T02:50:08.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.515+0000 7fbac9969640 1 librados: shutdown 2024-09-11T02:50:08.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-09-11T02:50:08.534 DEBUG:teuthology.parallel:result is None 2024-09-11T02:50:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.583+0000 7fc22653f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T02:50:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.583+0000 7fc22653f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T02:50:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.583+0000 7fc22653f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.583+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc200005380 con 0x7fc22006fe50 2024-09-11T02:50:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.583+0000 7fc21cff9640 1 -- 172.21.15.98:0/3788016054 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc21000a5e0 con 0x7fc22006fe50 2024-09-11T02:50:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.583+0000 7fc21cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-11T02:50:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.583+0000 7fc21cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:08.584 INFO:teuthology.orchestra.run.smithi098.stdout:51539607558 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 10 librados: watch_flush enter 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 10 librados: watch_flush exit 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 20 client.4328.objecter shutdown clearing up homeless session... 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 10 client.4328.objecter successfully canceled tick 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc20c03ded0 msgr2=0x7fc20c040390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc20c03ded0 0x7fc20c040390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc22013e9b0 tx=0x7fc214009190 comp rx=0 tx=0).stop 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 10 monclient: shutdown 2024-09-11T02:50:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc22006fe50 msgr2=0x7fc220091820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc22006fe50 0x7fc220091820 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc21000c680 tx=0x7fc21000cb40 comp rx=0 tx=0).stop 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 shutdown_connections 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc20c03ded0 0x7fc20c040390 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc220139b80 0x7fc22013d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc220072640 0x7fc220090f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 --2- 172.21.15.98:0/3788016054 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc22006fe50 0x7fc220091820 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 >> 172.21.15.98:0/3788016054 conn(0x7fc2200ac730 msgr2=0x7fc2200aad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 shutdown_connections 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 -- 172.21.15.98:0/3788016054 wait complete. 2024-09-11T02:50:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.587+0000 7fc22653f640 1 librados: shutdown 2024-09-11T02:50:08.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.5 2024-09-11T02:50:08.601 DEBUG:teuthology.parallel:result is None 2024-09-11T02:50:08.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T02:50:08.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T02:50:08.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:08.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fbd2c069dd0 con 0x7fbd2c136780 2024-09-11T02:50:08.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd21ffb640 1 -- 172.21.15.98:0/1598929846 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbd1c0e8020 con 0x7fbd2c136780 2024-09-11T02:50:08.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-11T02:50:08.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd21ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:08.616 INFO:teuthology.orchestra.run.smithi098.stdout:47244640261 2024-09-11T02:50:08.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 10 librados: watch_flush enter 2024-09-11T02:50:08.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 10 librados: watch_flush exit 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 20 client.4295.objecter shutdown clearing up homeless session... 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 10 client.4295.objecter successfully canceled tick 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbd1003df70 msgr2=0x7fbd10040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbd1003df70 0x7fbd10040430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd28009890 tx=0x7fbd28008040 comp rx=0 tx=0).stop 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 10 monclient: shutdown 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136780 msgr2=0x7fbd2c138a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136780 0x7fbd2c138a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c01c270 tx=0x7fbd1c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 shutdown_connections 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbd1003df70 0x7fbd10040430 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbd2c136780 0x7fbd2c138a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbd2c138f40 0x7fbd2c1393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.615+0000 7fbd337ca640 1 --2- 172.21.15.98:0/1598929846 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbd2c0a4610 0x7fbd2c1384c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.619+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 >> 172.21.15.98:0/1598929846 conn(0x7fbd2c06ed00 msgr2=0x7fbd2c0a7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.619+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 shutdown_connections 2024-09-11T02:50:08.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.619+0000 7fbd337ca640 1 -- 172.21.15.98:0/1598929846 wait complete. 2024-09-11T02:50:08.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.619+0000 7fbd337ca640 1 librados: shutdown 2024-09-11T02:50:08.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.0 2024-09-11T02:50:08.632 DEBUG:teuthology.parallel:result is None 2024-09-11T02:50:08.632 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T02:50:08.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T02:50:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.779+0000 7fe584ad1640 10 monclient: get_monmap_and_config 2024-09-11T02:50:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.779+0000 7fe584ad1640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.779+0000 7fe584ad1640 10 monclient: monmap: 2024-09-11T02:50:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:08.783844+0000 2024-09-11T02:50:08.780 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:08.783844+0000 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.781 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:08.782 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:08.782 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.779+0000 7fe584ad1640 1 Processor -- start 2024-09-11T02:50:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.779+0000 7fe584ad1640 1 -- start start 2024-09-11T02:50:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.779+0000 7fe584ad1640 10 monclient: init 2024-09-11T02:50:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.779+0000 7fe584ad1640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient: _add_conns ranks=[9,3,4,5,2,7,0,6,1,10,8] 2024-09-11T02:50:08.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe58006f6c0 0x7fe580072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): picked mon.noname-a con 0x7fe58006f6c0 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe58006faa0 0x7fe58006fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): picked mon.noname-g con 0x7fe58006faa0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe5800703c0 0x7fe5801397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): picked mon.noname-c con 0x7fe5800703c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe580071ff0 con 0x7fe58006f6c0 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe5800a8750 con 0x7fe58006faa0 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe5800716c0 con 0x7fe5800703c0 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient(hunting): authenticate will time out at 962.664101s 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe58006f6c0 0x7fe580072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe58006f6c0 0x7fe580072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37020/0 (socket says 172.21.15.98:37020) 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 1 -- 172.21.15.98:0/2236884390 learned_addr learned my addr 172.21.15.98:0/2236884390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 10 monclient(hunting): get_auth_request con 0x7fe58006f6c0 auth_method 0 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 --2- 172.21.15.98:0/2236884390 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe5800703c0 0x7fe5801397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): get_auth_request con 0x7fe5800703c0 auth_method 0 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57dd9b640 1 --2- 172.21.15.98:0/2236884390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe58006faa0 0x7fe58006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 -- 172.21.15.98:0/2236884390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe58006faa0 msgr2=0x7fe58006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 --2- 172.21.15.98:0/2236884390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe58006faa0 0x7fe58006fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 -- 172.21.15.98:0/2236884390 >> v2:172.21.15.98:3300/0 conn(0x7fe58006f6c0 msgr2=0x7fe580072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 --2- 172.21.15.98:0/2236884390 >> v2:172.21.15.98:3300/0 conn(0x7fe58006f6c0 0x7fe580072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 monclient: found mon.noname-c 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 -- 172.21.15.98:0/2236884390 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5800a64a0 con 0x7fe5800703c0 2024-09-11T02:50:08.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57e59c640 1 --2- 172.21.15.98:0/2236884390 >> v2:172.21.15.98:3300/0 conn(0x7fe58006f6c0 0x7fe580072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57ed9d640 1 --2- 172.21.15.98:0/2236884390 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe5800703c0 0x7fe5801397c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe5680097f0 tx=0x7fe56801c0a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=5865db332c9e18a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 1 -- 172.21.15.98:0/2236884390 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5680b0070 con 0x7fe5800703c0 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: dump: 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.795 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 1 -- 172.21.15.98:0/2236884390 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5680aa810 con 0x7fe5800703c0 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 5 monclient: authenticate success, global_id 4324 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 1 -- 172.21.15.98:0/2236884390 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5680a9300 con 0x7fe5800703c0 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 4 set_mon_vals no callback set 2024-09-11T02:50:08.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe57d59a640 10 monclient: dump: 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.799 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.799 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.799 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.799 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.799 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.799 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 10 monclient: shutdown 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- 172.21.15.98:0/2236884390 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe5800703c0 msgr2=0x7fe5801397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 --2- 172.21.15.98:0/2236884390 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe5800703c0 0x7fe5801397c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe5680097f0 tx=0x7fe56801c0a0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- 172.21.15.98:0/2236884390 shutdown_connections 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 --2- 172.21.15.98:0/2236884390 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe5800703c0 0x7fe5801397c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 --2- 172.21.15.98:0/2236884390 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe58006faa0 0x7fe58006fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 --2- 172.21.15.98:0/2236884390 >> v2:172.21.15.98:3300/0 conn(0x7fe58006f6c0 0x7fe580072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- 172.21.15.98:0/2236884390 >> 172.21.15.98:0/2236884390 conn(0x7fe5800acac0 msgr2=0x7fe5800a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- 172.21.15.98:0/2236884390 shutdown_connections 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.783+0000 7fe584ad1640 1 -- 172.21.15.98:0/2236884390 wait complete. 2024-09-11T02:50:08.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: build_initial_monmap 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: monmap: 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:08.801 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 librados: starting msgr at 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 librados: starting objecter 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 Processor -- start 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 -- start start 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 librados: setting wanted keys 2024-09-11T02:50:08.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 librados: calling monclient init 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: init 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _add_conns ranks=[5,0,3,6,7,1,8,4,2] 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe58006f6c0 0x7fe5800a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): picked mon.noname-h con 0x7fe58006f6c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe58006faa0 0x7fe5800a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): picked mon.noname-a con 0x7fe58006faa0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5800703c0 0x7fe5800a2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): picked mon.noname-g con 0x7fe5800703c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe580143840 con 0x7fe58006faa0 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe5801436c0 con 0x7fe5800703c0 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fe5801439c0 con 0x7fe58006f6c0 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient(hunting): authenticate will time out at 962.668930s 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe58006faa0 0x7fe5800a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe58006faa0 0x7fe5800a1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37030/0 (socket says 172.21.15.98:37030) 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 -- 172.21.15.98:0/1900330895 learned_addr learned my addr 172.21.15.98:0/1900330895 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): get_auth_request con 0x7fe58006faa0 auth_method 0 2024-09-11T02:50:08.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57ed9d640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5800703c0 0x7fe5800a2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57e59c640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe58006f6c0 0x7fe5800a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 -- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe58006f6c0 msgr2=0x7fe5800a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe58006f6c0 0x7fe5800a19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 -- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5800703c0 msgr2=0x7fe5800a2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5800703c0 0x7fe5800a2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 monclient: found mon.noname-a 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 -- 172.21.15.98:0/1900330895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5800a54d0 con 0x7fe58006faa0 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57dd9b640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe58006faa0 0x7fe5800a1f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe57000ce10 tx=0x7fe57000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 client.?.objecter ms_handle_connect 0x7fe58006faa0 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 1 -- 172.21.15.98:0/1900330895 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe57000ae30 con 0x7fe58006faa0 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: dump: 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.807 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:08.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 1 -- 172.21.15.98:0/1900330895 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57000ea70 con 0x7fe58006faa0 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 1 -- 172.21.15.98:0/1900330895 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe57000ec60 con 0x7fe58006faa0 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 monclient: dump: 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:08.809 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57cd99640 4 set_mon_vals no callback set 2024-09-11T02:50:08.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 5 monclient: authenticate success, global_id 4464 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _renew_subs 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5800a57b0 con 0x7fe58006faa0 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _renew_subs 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe580140240 con 0x7fe58006faa0 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 librados: init done 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:50:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe58013fcd0 con 0x7fe58006faa0 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 1 -- 172.21.15.98:0/1900330895 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe57000e600 con 0x7fe58006faa0 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe56003df40 0x7fe560040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 1 -- 172.21.15.98:0/1900330895 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fe57004fd10 con 0x7fe58006faa0 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 10 client.4464.objecter ms_dispatch 0x7fe580000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 3 client.4464.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 3 client.4464.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe576ffd640 20 client.4464.objecter dump_active .. 0 homeless 2024-09-11T02:50:08.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57e59c640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe56003df40 0x7fe560040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:08.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.787+0000 7fe57e59c640 10 monclient: get_auth_request con 0x7fe56003df40 auth_method 0 2024-09-11T02:50:08.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.791+0000 7fe576ffd640 10 client.4464.objecter ms_handle_connect 0x7fe56003df40 2024-09-11T02:50:08.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.791+0000 7fe57e59c640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe56003df40 0x7fe560040400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe578009630 tx=0x7fe578008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:08.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.791+0000 7fe576ffd640 1 -- 172.21.15.98:0/1900330895 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe57000ee00 con 0x7fe58006faa0 2024-09-11T02:50:08.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.791+0000 7fe576ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:08.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.791+0000 7fe576ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:08.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.947+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe5800aa590 con 0x7fe56003df40 2024-09-11T02:50:08.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.947+0000 7fe576ffd640 1 -- 172.21.15.98:0/1900330895 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34102 (secure 0 0 0) 0x7fe5800aa590 con 0x7fe56003df40 2024-09-11T02:50:08.949 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:08.949 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T02:50:08.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 10 librados: watch_flush enter 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 10 librados: watch_flush exit 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 20 client.4464.objecter shutdown clearing up homeless session... 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 10 client.4464.objecter successfully canceled tick 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe56003df40 msgr2=0x7fe560040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe56003df40 0x7fe560040400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe578009630 tx=0x7fe578008040 comp rx=0 tx=0).stop 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 10 monclient: shutdown 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe58006faa0 msgr2=0x7fe5800a1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe58006faa0 0x7fe5800a1f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe57000ce10 tx=0x7fe57000a660 comp rx=0 tx=0).stop 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 shutdown_connections 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe56003df40 0x7fe560040400 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe5800703c0 0x7fe5800a2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe58006faa0 0x7fe5800a1f20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 --2- 172.21.15.98:0/1900330895 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe58006f6c0 0x7fe5800a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 >> 172.21.15.98:0/1900330895 conn(0x7fe5800acac0 msgr2=0x7fe580140450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 shutdown_connections 2024-09-11T02:50:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 -- 172.21.15.98:0/1900330895 wait complete. 2024-09-11T02:50:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:08.951+0000 7fe584ad1640 1 librados: shutdown 2024-09-11T02:50:08.965 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-11T02:50:07.747164+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164368,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066672,"statfs":{"total":579820584960,"available":579652272128,"internally_reserved":0,"allocated":2277376,"data_stored":1539766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.815104"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.028180+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T02:50:05.028180+0000","last_peered":"2024-09-11T02:50:05.028180+0000","last_clean":"2024-09-11T02:50:05.028180+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T02:50:05.028180+0000","last_undegraded":"2024-09-11T02:50:05.028180+0000","last_fullsized":"2024-09-11T02:50:05.028180+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975717+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T02:50:04.975717+0000","last_peered":"2024-09-11T02:50:04.975717+0000","last_clean":"2024-09-11T02:50:04.975717+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T02:50:04.975717+0000","last_undegraded":"2024-09-11T02:50:04.975717+0000","last_fullsized":"2024-09-11T02:50:04.975717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027854+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T02:50:05.027854+0000","last_peered":"2024-09-11T02:50:05.027854+0000","last_clean":"2024-09-11T02:50:05.027854+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T02:50:05.027854+0000","last_undegraded":"2024-09-11T02:50:05.027854+0000","last_fullsized":"2024-09-11T02:50:05.027854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975572+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T02:50:04.975572+0000","last_peered":"2024-09-11T02:50:04.975572+0000","last_clean":"2024-09-11T02:50:04.975572+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T02:50:04.975572+0000","last_undegraded":"2024-09-11T02:50:04.975572+0000","last_fullsized":"2024-09-11T02:50:04.975572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.982831+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T02:50:04.982831+0000","last_peered":"2024-09-11T02:50:04.982831+0000","last_clean":"2024-09-11T02:50:04.982831+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T02:50:04.982831+0000","last_undegraded":"2024-09-11T02:50:04.982831+0000","last_fullsized":"2024-09-11T02:50:04.982831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975112+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T02:50:04.975112+0000","last_peered":"2024-09-11T02:50:04.975112+0000","last_clean":"2024-09-11T02:50:04.975112+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T02:50:04.975112+0000","last_undegraded":"2024-09-11T02:50:04.975112+0000","last_fullsized":"2024-09-11T02:50:04.975112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027630+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T02:50:05.027630+0000","last_peered":"2024-09-11T02:50:05.027630+0000","last_clean":"2024-09-11T02:50:05.027630+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T02:50:05.027630+0000","last_undegraded":"2024-09-11T02:50:05.027630+0000","last_fullsized":"2024-09-11T02:50:05.027630+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.974249+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T02:50:04.974249+0000","last_peered":"2024-09-11T02:50:04.974249+0000","last_clean":"2024-09-11T02:50:04.974249+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T02:50:04.974249+0000","last_undegraded":"2024-09-11T02:50:04.974249+0000","last_fullsized":"2024-09-11T02:50:04.974249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.026460+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T02:50:05.026460+0000","last_peered":"2024-09-11T02:50:05.026460+0000","last_clean":"2024-09-11T02:50:05.026460+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T02:50:05.026460+0000","last_undegraded":"2024-09-11T02:50:05.026460+0000","last_fullsized":"2024-09-11T02:50:05.026460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T02:50:08.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T02:50:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.079+0000 7fa45cae7640 10 monclient: get_monmap_and_config 2024-09-11T02:50:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.079+0000 7fa45cae7640 10 monclient: build_initial_monmap 2024-09-11T02:50:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.079+0000 7fa45cae7640 10 monclient: monmap: 2024-09-11T02:50:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:09.083872+0000 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:09.083872+0000 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:09.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.079+0000 7fa45cae7640 1 Processor -- start 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 -- start start 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient: init 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient: _add_conns ranks=[10,6,2,8,1,4,9,5,7,0,3] 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa45806f6c0 0x7fa458072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): picked mon.noname-f con 0x7fa45806f6c0 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa45806faa0 0x7fa45806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): picked mon.noname-d con 0x7fa45806faa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4580703c0 0x7fa4581397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): picked mon.noname-b con 0x7fa4580703c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa458071ff0 con 0x7fa45806f6c0 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa4580a8750 con 0x7fa4580703c0 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa4580716c0 con 0x7fa45806faa0 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient(hunting): authenticate will time out at 962.964653s 2024-09-11T02:50:09.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4580703c0 0x7fa4581397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4580703c0 0x7fa4581397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48116/0 (socket says 172.21.15.98:48116) 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 -- 172.21.15.98:0/2585745764 learned_addr learned my addr 172.21.15.98:0/2585745764 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): get_auth_request con 0x7fa4580703c0 auth_method 0 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 1 --2- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa45806faa0 0x7fa45806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 10 monclient(hunting): get_auth_request con 0x7fa45806faa0 auth_method 0 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa455d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45659c640 1 --2- 172.21.15.98:0/2585745764 >> v2:172.21.15.190:3300/0 conn(0x7fa45806f6c0 0x7fa458072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45659c640 10 monclient(hunting): get_auth_request con 0x7fa45806f6c0 auth_method 0 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient(hunting): handle_auth_done global_id 4304 payload 274 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 -- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa45806faa0 msgr2=0x7fa45806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:09.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 --2- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa45806faa0 0x7fa45806fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 -- 172.21.15.98:0/2585745764 >> v2:172.21.15.190:3300/0 conn(0x7fa45806f6c0 msgr2=0x7fa458072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 --2- 172.21.15.98:0/2585745764 >> v2:172.21.15.190:3300/0 conn(0x7fa45806f6c0 0x7fa458072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 monclient: found mon.noname-b 2024-09-11T02:50:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 -- 172.21.15.98:0/2585745764 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4580a64a0 con 0x7fa4580703c0 2024-09-11T02:50:09.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa456d9d640 1 --2- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4580703c0 0x7fa4581397c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa4500098a0 tx=0x7fa45001c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=954440e512f2144 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 1 -- 172.21.15.98:0/2585745764 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4500b0070 con 0x7fa4580703c0 2024-09-11T02:50:09.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: dump: 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: _finish_auth 0 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 5 monclient: authenticate success, global_id 4304 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 1 -- 172.21.15.98:0/2585745764 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4500aa810 con 0x7fa4580703c0 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:09.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 4 set_mon_vals no callback set 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 1 -- 172.21.15.98:0/2585745764 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4500a9300 con 0x7fa4580703c0 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45559a640 10 monclient: dump: 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 10 monclient: shutdown 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 -- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4580703c0 msgr2=0x7fa4581397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 --2- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4580703c0 0x7fa4581397c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa4500098a0 tx=0x7fa45001c140 comp rx=0 tx=0).stop 2024-09-11T02:50:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 -- 172.21.15.98:0/2585745764 shutdown_connections 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 --2- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa4580703c0 0x7fa4581397c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.083+0000 7fa45cae7640 1 --2- 172.21.15.98:0/2585745764 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa45806faa0 0x7fa45806fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 --2- 172.21.15.98:0/2585745764 >> v2:172.21.15.190:3300/0 conn(0x7fa45806f6c0 0x7fa458072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- 172.21.15.98:0/2585745764 >> 172.21.15.98:0/2585745764 conn(0x7fa4580acac0 msgr2=0x7fa4580a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- 172.21.15.98:0/2585745764 shutdown_connections 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- 172.21.15.98:0/2585745764 wait complete. 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: build_initial_monmap 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: monmap: 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 librados: starting msgr at 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 librados: starting objecter 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 Processor -- start 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- start start 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 librados: setting wanted keys 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 librados: calling monclient init 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: init 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _add_conns ranks=[5,2,1,3,4,8,6,7,0] 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa45806f6c0 0x7fa4580a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): picked mon.noname-h con 0x7fa45806f6c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa45806faa0 0x7fa4580a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): picked mon.noname-b con 0x7fa45806faa0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4580703c0 0x7fa4581367d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): picked mon.noname-f con 0x7fa4580703c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa458143840 con 0x7fa4580703c0 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa4581436c0 con 0x7fa45806faa0 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa4581439c0 con 0x7fa45806f6c0 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient(hunting): authenticate will time out at 962.969461s 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa45806faa0 0x7fa4580a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa45806faa0 0x7fa4580a2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48122/0 (socket says 172.21.15.98:48122) 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 -- 172.21.15.98:0/1262785715 learned_addr learned my addr 172.21.15.98:0/1262785715 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): get_auth_request con 0x7fa45806faa0 auth_method 0 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 -- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa45806f6c0 msgr2=0x7fa4580a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa45806f6c0 0x7fa4580a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 -- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4580703c0 msgr2=0x7fa4581367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4580703c0 0x7fa4581367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 monclient: found mon.noname-b 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 -- 172.21.15.98:0/1262785715 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4580a3830 con 0x7fa45806faa0 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa455d9b640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa45806faa0 0x7fa4580a2230 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa44000e930 tx=0x7fa44000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 client.?.objecter ms_handle_connect 0x7fa45806faa0 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 1 -- 172.21.15.98:0/1262785715 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa440017070 con 0x7fa45806faa0 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: dump: 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.104 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: _finish_auth 0 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 1 -- 172.21.15.98:0/1262785715 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa440013470 con 0x7fa45806faa0 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 1 -- 172.21.15.98:0/1262785715 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa440013a90 con 0x7fa45806faa0 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 5 monclient: authenticate success, global_id 4313 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _renew_subs 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45813fcd0 con 0x7fa45806faa0 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa454d99640 4 set_mon_vals no callback set 2024-09-11T02:50:09.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 monclient: dump: 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.106 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _renew_subs 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa458136d10 con 0x7fa45806faa0 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 librados: init done 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 1 -- 172.21.15.98:0/1262785715 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa440013c30 con 0x7fa45806faa0 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45813ff70 con 0x7fa45806faa0 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa43803df40 0x7fa438040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 1 -- 172.21.15.98:0/1262785715 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fa44000f040 con 0x7fa45806faa0 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 10 client.4313.objecter ms_dispatch 0x7fa458000ba0 osd_map(19..19 src has 1..19) 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 3 client.4313.objecter handle_osd_map got epochs [19,19] > 0 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 3 client.4313.objecter handle_osd_map decoding full epoch 19 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa44effd640 20 client.4313.objecter dump_active .. 0 homeless 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45659c640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa43803df40 0x7fa438040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.087+0000 7fa45659c640 10 monclient: get_auth_request con 0x7fa43803df40 auth_method 0 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.091+0000 7fa44effd640 10 client.4313.objecter ms_handle_connect 0x7fa43803df40 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.091+0000 7fa45659c640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa43803df40 0x7fa438040400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4480070b0 tx=0x7fa448007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.091+0000 7fa44effd640 1 -- 172.21.15.98:0/1262785715 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fa45813ff70 con 0x7fa45806faa0 2024-09-11T02:50:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.091+0000 7fa44effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.091+0000 7fa44effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:09.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.251+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa4580a2770 con 0x7fa43803df40 2024-09-11T02:50:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.255+0000 7fa44effd640 1 -- 172.21.15.98:0/1262785715 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34066 (secure 0 0 0) 0x7fa448009710 con 0x7fa43803df40 2024-09-11T02:50:09.256 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 10 librados: watch_flush enter 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 10 librados: watch_flush exit 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 20 client.4313.objecter shutdown clearing up homeless session... 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 10 client.4313.objecter successfully canceled tick 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa43803df40 msgr2=0x7fa438040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa43803df40 0x7fa438040400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4480070b0 tx=0x7fa448007040 comp rx=0 tx=0).stop 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 10 monclient: shutdown 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa45806faa0 msgr2=0x7fa4580a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa45806faa0 0x7fa4580a2230 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa44000e930 tx=0x7fa44000edf0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 shutdown_connections 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4580703c0 0x7fa4581367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa43803df40 0x7fa438040400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa45806faa0 0x7fa4580a2230 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 --2- 172.21.15.98:0/1262785715 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa45806f6c0 0x7fa4580a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 >> 172.21.15.98:0/1262785715 conn(0x7fa4580acac0 msgr2=0x7fa4580a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 shutdown_connections 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 -- 172.21.15.98:0/1262785715 wait complete. 2024-09-11T02:50:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.259+0000 7fa45cae7640 1 librados: shutdown 2024-09-11T02:50:09.272 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-11T02:50:09.213257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164368,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066672,"statfs":{"total":579820584960,"available":579652272128,"internally_reserved":0,"allocated":2277376,"data_stored":1539766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.265940"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.028180+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T02:50:05.028180+0000","last_peered":"2024-09-11T02:50:05.028180+0000","last_clean":"2024-09-11T02:50:05.028180+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T02:50:05.028180+0000","last_undegraded":"2024-09-11T02:50:05.028180+0000","last_fullsized":"2024-09-11T02:50:05.028180+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975717+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T02:50:04.975717+0000","last_peered":"2024-09-11T02:50:04.975717+0000","last_clean":"2024-09-11T02:50:04.975717+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T02:50:04.975717+0000","last_undegraded":"2024-09-11T02:50:04.975717+0000","last_fullsized":"2024-09-11T02:50:04.975717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027854+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T02:50:05.027854+0000","last_peered":"2024-09-11T02:50:05.027854+0000","last_clean":"2024-09-11T02:50:05.027854+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T02:50:05.027854+0000","last_undegraded":"2024-09-11T02:50:05.027854+0000","last_fullsized":"2024-09-11T02:50:05.027854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975572+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T02:50:04.975572+0000","last_peered":"2024-09-11T02:50:04.975572+0000","last_clean":"2024-09-11T02:50:04.975572+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T02:50:04.975572+0000","last_undegraded":"2024-09-11T02:50:04.975572+0000","last_fullsized":"2024-09-11T02:50:04.975572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.982831+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T02:50:04.982831+0000","last_peered":"2024-09-11T02:50:04.982831+0000","last_clean":"2024-09-11T02:50:04.982831+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T02:50:04.982831+0000","last_undegraded":"2024-09-11T02:50:04.982831+0000","last_fullsized":"2024-09-11T02:50:04.982831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975112+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T02:50:04.975112+0000","last_peered":"2024-09-11T02:50:04.975112+0000","last_clean":"2024-09-11T02:50:04.975112+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T02:50:04.975112+0000","last_undegraded":"2024-09-11T02:50:04.975112+0000","last_fullsized":"2024-09-11T02:50:04.975112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027630+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T02:50:05.027630+0000","last_peered":"2024-09-11T02:50:05.027630+0000","last_clean":"2024-09-11T02:50:05.027630+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T02:50:05.027630+0000","last_undegraded":"2024-09-11T02:50:05.027630+0000","last_fullsized":"2024-09-11T02:50:05.027630+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.974249+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T02:50:04.974249+0000","last_peered":"2024-09-11T02:50:04.974249+0000","last_clean":"2024-09-11T02:50:04.974249+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T02:50:04.974249+0000","last_undegraded":"2024-09-11T02:50:04.974249+0000","last_fullsized":"2024-09-11T02:50:04.974249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.026460+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T02:50:05.026460+0000","last_peered":"2024-09-11T02:50:05.026460+0000","last_clean":"2024-09-11T02:50:05.026460+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T02:50:05.026460+0000","last_undegraded":"2024-09-11T02:50:05.026460+0000","last_fullsized":"2024-09-11T02:50:05.026460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T02:50:09.273 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T02:50:09.273 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T02:50:09.274 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T02:50:09.274 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T02:50:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.387+0000 7fbcae47b640 10 monclient: get_monmap_and_config 2024-09-11T02:50:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.387+0000 7fbcae47b640 10 monclient: build_initial_monmap 2024-09-11T02:50:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.387+0000 7fbcae47b640 10 monclient: monmap: 2024-09-11T02:50:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.388 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:09.391908+0000 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:09.391908+0000 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:09.389 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:09.390 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.387+0000 7fbcae47b640 1 Processor -- start 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.387+0000 7fbcae47b640 1 -- start start 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.387+0000 7fbcae47b640 10 monclient: init 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient: _add_conns ranks=[2,7,8,0,4,6,10,5,3,9,1] 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): picked mon.noname-b con 0x7fbca806f6c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbca806faa0 0x7fbca806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): picked mon.noname-i con 0x7fbca806faa0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbca80703c0 0x7fbca81397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): picked mon.noname-e con 0x7fbca80703c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbca8071ff0 con 0x7fbca806f6c0 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbca80a8750 con 0x7fbca806faa0 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbca80716c0 con 0x7fbca80703c0 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient(hunting): authenticate will time out at 963.272127s 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbca80703c0 0x7fbca81397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbca80703c0 0x7fbca81397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34974/0 (socket says 172.21.15.98:34974) 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48134/0 (socket says 172.21.15.98:48134) 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 -- 172.21.15.98:0/3488311702 learned_addr learned my addr 172.21.15.98:0/3488311702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:09.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): get_auth_request con 0x7fbca80703c0 auth_method 0 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 10 monclient(hunting): get_auth_request con 0x7fbca806f6c0 auth_method 0 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient(hunting): handle_auth_done global_id 4337 payload 274 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 -- 172.21.15.98:0/3488311702 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbca806faa0 msgr2=0x7fbca806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca77fe640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbca806faa0 0x7fbca806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbca806faa0 0x7fbca806fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 -- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 msgr2=0x7fbca8072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient: _finish_hunting 0 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 monclient: found mon.noname-e 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 -- 172.21.15.98:0/3488311702 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca80a64a0 con 0x7fbca80703c0 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca7fff640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:09.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcaca18640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbca80703c0 0x7fbca81397c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009e80 tx=0x7fbca001c650 comp rx=0 tx=0).ready entity=mon.8 client_cookie=38c1ff932ddd18a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 1 -- 172.21.15.98:0/3488311702 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbca00ac020 con 0x7fbca80703c0 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: dump: 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 1 -- 172.21.15.98:0/3488311702 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca0009110 con 0x7fbca80703c0 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 5 monclient: authenticate success, global_id 4337 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:09.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 4 set_mon_vals no callback set 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 1 -- 172.21.15.98:0/3488311702 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbca00a6d40 con 0x7fbca80703c0 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbca6ffd640 10 monclient: dump: 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.400 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 10 monclient: shutdown 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbca80703c0 msgr2=0x7fbca81397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbca80703c0 0x7fbca81397c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009e80 tx=0x7fbca001c650 comp rx=0 tx=0).stop 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- 172.21.15.98:0/3488311702 shutdown_connections 2024-09-11T02:50:09.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbca80703c0 0x7fbca81397c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbca806faa0 0x7fbca806fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 --2- 172.21.15.98:0/3488311702 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- 172.21.15.98:0/3488311702 >> 172.21.15.98:0/3488311702 conn(0x7fbca80acac0 msgr2=0x7fbca80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- 172.21.15.98:0/3488311702 shutdown_connections 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.391+0000 7fbcae47b640 1 -- 172.21.15.98:0/3488311702 wait complete. 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: build_initial_monmap 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: monmap: 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:09.402 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 librados: starting msgr at 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 librados: starting objecter 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 Processor -- start 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 -- start start 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 librados: setting wanted keys 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 librados: calling monclient init 2024-09-11T02:50:09.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: init 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _add_conns ranks=[2,0,3,5,1,4,8,7,6] 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): picked mon.noname-b con 0x7fbca806f6c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbca806faa0 0x7fbca81368d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): picked mon.noname-a con 0x7fbca806faa0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbca80703c0 0x7fbca813bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): picked mon.noname-g con 0x7fbca80703c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8143840 con 0x7fbca806faa0 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbca81436c0 con 0x7fbca806f6c0 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbca81439c0 con 0x7fbca80703c0 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient(hunting): authenticate will time out at 963.277096s 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8136390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48148/0 (socket says 172.21.15.98:48148) 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 -- 172.21.15.98:0/2565719380 learned_addr learned my addr 172.21.15.98:0/2565719380 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): get_auth_request con 0x7fbca806f6c0 auth_method 0 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbca806faa0 0x7fbca81368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient(hunting): get_auth_request con 0x7fbca806faa0 auth_method 0 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 -- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbca80703c0 msgr2=0x7fbca813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbca80703c0 0x7fbca813bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 -- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbca806faa0 msgr2=0x7fbca81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbca806faa0 0x7fbca81368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 monclient: found mon.noname-b 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 -- 172.21.15.98:0/2565719380 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca813f490 con 0x7fbca806f6c0 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca7fff640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8136390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800ebe0 tx=0x7fbc9800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 client.?.objecter ms_handle_connect 0x7fbca806f6c0 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 1 -- 172.21.15.98:0/2565719380 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc98017070 con 0x7fbca806f6c0 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: dump: 2024-09-11T02:50:09.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.412 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: _finish_auth 0 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 1 -- 172.21.15.98:0/2565719380 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9800cc30 con 0x7fbca806f6c0 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 1 -- 172.21.15.98:0/2565719380 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc9800cde0 con 0x7fbca806f6c0 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 monclient: dump: 2024-09-11T02:50:09.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca67fc640 4 set_mon_vals no callback set 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 5 monclient: authenticate success, global_id 4331 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _renew_subs 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca813f770 con 0x7fbca806f6c0 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _renew_subs 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca80aa540 con 0x7fbca806f6c0 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 librados: init done 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 1 -- 172.21.15.98:0/2565719380 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbc98013470 con 0x7fbca806f6c0 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca806fe80 con 0x7fbca806f6c0 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc8803df40 0x7fbc88040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 1 -- 172.21.15.98:0/2565719380 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fbc9800f040 con 0x7fbca806f6c0 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 10 client.4331.objecter ms_dispatch 0x7fbca8000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 3 client.4331.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 3 client.4331.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca4ff9640 20 client.4331.objecter dump_active .. 0 homeless 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc8803df40 0x7fbc88040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.395+0000 7fbca77fe640 10 monclient: get_auth_request con 0x7fbc8803df40 auth_method 0 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.399+0000 7fbca4ff9640 10 client.4331.objecter ms_handle_connect 0x7fbc8803df40 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.399+0000 7fbca77fe640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc8803df40 0x7fbc88040400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbca813d210 tx=0x7fbc90008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.399+0000 7fbca4ff9640 1 -- 172.21.15.98:0/2565719380 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbc98021070 con 0x7fbca806f6c0 2024-09-11T02:50:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.399+0000 7fbca4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.399+0000 7fbca4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.595+0000 7fbcae47b640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-11T02:50:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.595+0000 7fbcae47b640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-11T02:50:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.595+0000 7fbcae47b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.595+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fbca813dcf0 con 0x7fbca806f6c0 2024-09-11T02:50:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.595+0000 7fbca4ff9640 1 -- 172.21.15.98:0/2565719380 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fbc9804f090 con 0x7fbca806f6c0 2024-09-11T02:50:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.595+0000 7fbca4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-11T02:50:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.595+0000 7fbca4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:09.598 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:09.598 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T02:50:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 10 librados: watch_flush enter 2024-09-11T02:50:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 10 librados: watch_flush exit 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 20 client.4331.objecter shutdown clearing up homeless session... 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 10 client.4331.objecter successfully canceled tick 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc8803df40 msgr2=0x7fbc88040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc8803df40 0x7fbc88040400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbca813d210 tx=0x7fbc90008040 comp rx=0 tx=0).stop 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 10 monclient: shutdown 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 msgr2=0x7fbca8136390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8136390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800ebe0 tx=0x7fbc9800c520 comp rx=0 tx=0).stop 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 shutdown_connections 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbca80703c0 0x7fbca813bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbca806faa0 0x7fbca81368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc8803df40 0x7fbc88040400 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 --2- 172.21.15.98:0/2565719380 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbca806f6c0 0x7fbca8136390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 >> 172.21.15.98:0/2565719380 conn(0x7fbca80acac0 msgr2=0x7fbca80aa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 shutdown_connections 2024-09-11T02:50:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 -- 172.21.15.98:0/2565719380 wait complete. 2024-09-11T02:50:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.599+0000 7fbcae47b640 1 librados: shutdown 2024-09-11T02:50:09.614 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T02:50:09.614 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-11T02:50:09.628 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-11T02:50:09.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T02:50:09.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.743+0000 7fa56522d640 10 monclient: get_monmap_and_config 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.743+0000 7fa56522d640 10 monclient: build_initial_monmap 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.743+0000 7fa56522d640 10 monclient: monmap: 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:09.747998+0000 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:09.747998+0000 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:09.745 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:09.746 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.743+0000 7fa56522d640 1 Processor -- start 2024-09-11T02:50:09.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.743+0000 7fa56522d640 1 -- start start 2024-09-11T02:50:09.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.743+0000 7fa56522d640 10 monclient: init 2024-09-11T02:50:09.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:09.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient: _add_conns ranks=[4,9,1,3,10,8,6,5,7,2,0] 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5600a5140 0x7fa5601399d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5600a5140 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5600a1920 0x7fa560139f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient(hunting): picked mon.noname-a con 0x7fa5600a1920 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa56013a450 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5600a0c60 con 0x7fa56013a450 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa5600a5520 con 0x7fa5600a1920 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa5600a1e00 con 0x7fa5600a5140 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient(hunting): authenticate will time out at 963.628219s 2024-09-11T02:50:09.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5600a5140 0x7fa5601399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa5600a1920 0x7fa560139f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5600a5140 0x7fa5601399d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39140/0 (socket says 172.21.15.98:39140) 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 -- 172.21.15.98:0/4177455410 learned_addr learned my addr 172.21.15.98:0/4177455410 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 10 monclient(hunting): get_auth_request con 0x7fa5600a1920 auth_method 0 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): get_auth_request con 0x7fa5600a5140 auth_method 0 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 -- 172.21.15.98:0/4177455410 >> v2:172.21.15.98:3300/0 conn(0x7fa5600a1920 msgr2=0x7fa560139f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 --2- 172.21.15.98:0/4177455410 >> v2:172.21.15.98:3300/0 conn(0x7fa5600a1920 0x7fa560139f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 -- 172.21.15.98:0/4177455410 >> v1:172.21.15.190:6789/0 conn(0x7fa56013a450 legacy=0x7fa5600a3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 monclient: found mon.noname-c 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 -- 172.21.15.98:0/4177455410 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5600a1500 con 0x7fa5600a5140 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55e59c640 1 --2- 172.21.15.98:0/4177455410 >> v2:172.21.15.98:3300/0 conn(0x7fa5600a1920 0x7fa560139f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55ed9d640 1 --2- 172.21.15.98:0/4177455410 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5600a5140 0x7fa5601399d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa550009870 tx=0x7fa55001c2d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=49132241ea130be server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 1 -- 172.21.15.98:0/4177455410 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5500b0070 con 0x7fa5600a5140 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: dump: 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: _finish_auth 0 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: _check_auth_tickets 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 1 -- 172.21.15.98:0/4177455410 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5500aa810 con 0x7fa5600a5140 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 1 -- 172.21.15.98:0/4177455410 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5500a9300 con 0x7fa5600a5140 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 5 monclient: authenticate success, global_id 4333 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 4 set_mon_vals no callback set 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa55dd9b640 10 monclient: dump: 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.760 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 10 monclient: shutdown 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- 172.21.15.98:0/4177455410 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5600a5140 msgr2=0x7fa5601399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 --2- 172.21.15.98:0/4177455410 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5600a5140 0x7fa5601399d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa550009870 tx=0x7fa55001c2d0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- 172.21.15.98:0/4177455410 shutdown_connections 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 --2- 172.21.15.98:0/4177455410 >> v2:172.21.15.98:3300/0 conn(0x7fa5600a1920 0x7fa560139f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 --2- 172.21.15.98:0/4177455410 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5600a5140 0x7fa5601399d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- 172.21.15.98:0/4177455410 >> 172.21.15.98:0/4177455410 conn(0x7fa5600a64a0 msgr2=0x7fa56006ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- 172.21.15.98:0/4177455410 shutdown_connections 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.747+0000 7fa56522d640 1 -- 172.21.15.98:0/4177455410 wait complete. 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: build_initial_monmap 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: monmap: 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:09.762 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 librados: starting msgr at 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 librados: starting objecter 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 Processor -- start 2024-09-11T02:50:09.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 -- start start 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 librados: setting wanted keys 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 librados: calling monclient init 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: init 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _add_conns ranks=[7,3,5,8,0,6,2,1,4] 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa5600a1920 0x7fa560138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): picked mon.noname-i con 0x7fa5600a1920 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5600a5140 0x7fa560139150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): picked mon.noname-g con 0x7fa5600a5140 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa56013a450 0x7fa56013d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): picked mon.noname-h con 0x7fa56013a450 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa560143e50 con 0x7fa5600a5140 2024-09-11T02:50:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa5600a4e20 con 0x7fa56013a450 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa560143fd0 con 0x7fa5600a1920 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient(hunting): authenticate will time out at 963.632940s 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa5600a1920 0x7fa560138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa5600a1920 0x7fa560138c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:43858/0 (socket says 172.21.15.98:43858) 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 -- 172.21.15.98:0/1491115575 learned_addr learned my addr 172.21.15.98:0/1491115575 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): get_auth_request con 0x7fa5600a1920 auth_method 0 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5600a5140 0x7fa560139150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa56013a450 0x7fa56013d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 10 monclient(hunting): get_auth_request con 0x7fa5600a5140 auth_method 0 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 10 monclient(hunting): get_auth_request con 0x7fa56013a450 auth_method 0 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 -- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa56013a450 msgr2=0x7fa56013d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa56013a450 0x7fa56013d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 -- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5600a5140 msgr2=0x7fa560139150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5600a5140 0x7fa560139150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 monclient: found mon.noname-i 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:50:09.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 -- 172.21.15.98:0/1491115575 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa560123bf0 con 0x7fa5600a1920 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55ed9d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa5600a1920 0x7fa560138c10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa5500ae040 tx=0x7fa550019070 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 client.?.objecter ms_handle_connect 0x7fa5600a1920 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 1 -- 172.21.15.98:0/1491115575 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5500b9070 con 0x7fa5600a1920 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: dump: 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 1 -- 172.21.15.98:0/1491115575 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5500044b0 con 0x7fa5600a1920 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 1 -- 172.21.15.98:0/1491115575 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5500b0040 con 0x7fa5600a1920 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 5 monclient: authenticate success, global_id 4264 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _renew_subs 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa560070210 con 0x7fa5600a1920 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa55d59a640 4 set_mon_vals no callback set 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa5477fe640 10 monclient: dump: 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _renew_subs 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa56006fc90 con 0x7fa5600a1920 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 librados: init done 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.751+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5600a3cb0 con 0x7fa5600a1920 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 1 -- 172.21.15.98:0/1491115575 <== mon.7 v2:172.21.15.190:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa550004040 con 0x7fa5600a1920 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa54c03de70 0x7fa54c040330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 1 -- 172.21.15.98:0/1491115575 <== mon.7 v2:172.21.15.190:3303/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fa5500e8dc0 con 0x7fa5600a1920 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 10 client.4264.objecter ms_dispatch 0x7fa560000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 3 client.4264.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 3 client.4264.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-09-11T02:50:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa55e59c640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa54c03de70 0x7fa54c040330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa55e59c640 10 monclient: get_auth_request con 0x7fa54c03de70 auth_method 0 2024-09-11T02:50:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa5477fe640 10 client.4264.objecter ms_handle_connect 0x7fa54c03de70 2024-09-11T02:50:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.755+0000 7fa55e59c640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa54c03de70 0x7fa54c040330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa554009580 tx=0x7fa554009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.767+0000 7fa5477fe640 1 -- 172.21.15.98:0/1491115575 <== mon.7 v2:172.21.15.190:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fa5600a0c60 con 0x7fa5600a1920 2024-09-11T02:50:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.767+0000 7fa5477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.767+0000 7fa5477fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:09.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.935+0000 7fa56522d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:09.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.935+0000 7fa56522d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:09.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.935+0000 7fa56522d640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:09.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.935+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa56013a1e0 con 0x7fa5600a1920 2024-09-11T02:50:09.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa5477fe640 1 -- 172.21.15.98:0/1491115575 <== mon.7 v2:172.21.15.190:3303/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+12358 (secure 0 0 0) 0x7fa5500aa7e0 con 0x7fa5600a1920 2024-09-11T02:50:09.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa5477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T02:50:09.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa5477fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:09.938 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:09.938 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":20,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T02:50:09.206528+0000","last_up_change":"2024-09-11T02:49:54.844374+0000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T02:49:55.787515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T02:50:01.768815+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6801","nonce":1456395826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6803","nonce":1456395826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6807","nonce":1456395826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6805","nonce":1456395826}]},"public_addr":"172.21.15.98:6801/1456395826","cluster_addr":"172.21.15.98:6803/1456395826","heartbeat_back_addr":"172.21.15.98:6807/1456395826","heartbeat_front_addr":"172.21.15.98:6805/1456395826","state":["exists","up"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6809","nonce":2313933591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6811","nonce":2313933591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6815","nonce":2313933591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6813","nonce":2313933591}]},"public_addr":"172.21.15.98:6809/2313933591","cluster_addr":"172.21.15.98:6811/2313933591","heartbeat_back_addr":"172.21.15.98:6815/2313933591","heartbeat_front_addr":"172.21.15.98:6813/2313933591","state":["exists","up"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6817","nonce":3987433753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3987433753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3987433753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3987433753}]},"public_addr":"172.21.15.98:6817/3987433753","cluster_addr":"172.21.15.98:6819/3987433753","heartbeat_back_addr":"172.21.15.98:6823/3987433753","heartbeat_front_addr":"172.21.15.98:6821/3987433753","state":["exists","up"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6801","nonce":373809828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6803","nonce":373809828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6807","nonce":373809828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6805","nonce":373809828}]},"public_addr":"172.21.15.190:6801/373809828","cluster_addr":"172.21.15.190:6803/373809828","heartbeat_back_addr":"172.21.15.190:6807/373809828","heartbeat_front_addr":"172.21.15.190:6805/373809828","state":["exists","up"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6817","nonce":1271446291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6819","nonce":1271446291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6823","nonce":1271446291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6821","nonce":1271446291}]},"public_addr":"172.21.15.190:6817/1271446291","cluster_addr":"172.21.15.190:6819/1271446291","heartbeat_back_addr":"172.21.15.190:6823/1271446291","heartbeat_front_addr":"172.21.15.190:6821/1271446291","state":["exists","up"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6809","nonce":1497028198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6811","nonce":1497028198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6815","nonce":1497028198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6813","nonce":1497028198}]},"public_addr":"172.21.15.190:6809/1497028198","cluster_addr":"172.21.15.190:6811/1497028198","heartbeat_back_addr":"172.21.15.190:6815/1497028198","heartbeat_front_addr":"172.21.15.190:6813/1497028198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.670036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.679318+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.766598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.015321+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.338711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.969084+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 10 librados: watch_flush enter 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 10 librados: watch_flush exit 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 20 client.4264.objecter shutdown clearing up homeless session... 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 10 client.4264.objecter successfully canceled tick 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa54c03de70 msgr2=0x7fa54c040330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa54c03de70 0x7fa54c040330 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa554009580 tx=0x7fa554009190 comp rx=0 tx=0).stop 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 10 monclient: shutdown 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa5600a1920 msgr2=0x7fa560138c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa5600a1920 0x7fa560138c10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa5500ae040 tx=0x7fa550019070 comp rx=0 tx=0).stop 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 shutdown_connections 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa54c03de70 0x7fa54c040330 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa56013a450 0x7fa56013d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa5600a5140 0x7fa560139150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 --2- 172.21.15.98:0/1491115575 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa5600a1920 0x7fa560138c10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 >> 172.21.15.98:0/1491115575 conn(0x7fa5600a64a0 msgr2=0x7fa5601414c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 shutdown_connections 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 -- 172.21.15.98:0/1491115575 wait complete. 2024-09-11T02:50:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:09.939+0000 7fa56522d640 1 librados: shutdown 2024-09-11T02:50:09.956 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T02:49:55.787515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T02:50:01.768815+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-11T02:50:09.957 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-11T02:50:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.075+0000 7f5f41ade640 10 monclient: get_monmap_and_config 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.075+0000 7f5f41ade640 10 monclient: build_initial_monmap 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.075+0000 7f5f41ade640 10 monclient: monmap: 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:10.080377+0000 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:10.080377+0000 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:10.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.075+0000 7f5f41ade640 1 Processor -- start 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.075+0000 7f5f41ade640 1 -- start start 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.075+0000 7f5f41ade640 10 monclient: init 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient: _add_conns ranks=[10,8,4,1,6,2,5,9,0,7,3] 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5f3c06f670 0x7f5f3c06fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): picked mon.noname-f con 0x7f5f3c06f670 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f3c0727b0 0x7f5f3c06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): picked mon.noname-e con 0x7f5f3c0727b0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c139c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): picked mon.noname-c con 0x7f5f3c1397c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5f3c0704d0 con 0x7f5f3c06f670 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5f3c0a64a0 con 0x7f5f3c1397c0 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5f3c071500 con 0x7f5f3c0727b0 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient(hunting): authenticate will time out at 963.960612s 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f3c0727b0 0x7f5f3c06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c139c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f3c0727b0 0x7f5f3c06ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34986/0 (socket says 172.21.15.98:34986) 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 -- 172.21.15.98:0/2055258010 learned_addr learned my addr 172.21.15.98:0/2055258010 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): get_auth_request con 0x7f5f3c0727b0 auth_method 0 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c139c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39150/0 (socket says 172.21.15.98:39150) 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request con 0x7f5f3c1397c0 auth_method 0 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3b7fe640 1 --2- 172.21.15.98:0/2055258010 >> v2:172.21.15.190:3300/0 conn(0x7f5f3c06f670 0x7f5f3c06fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 -- 172.21.15.98:0/2055258010 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 msgr2=0x7f5f3c139c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 --2- 172.21.15.98:0/2055258010 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c139c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 -- 172.21.15.98:0/2055258010 >> v2:172.21.15.190:3300/0 conn(0x7f5f3c06f670 msgr2=0x7f5f3c06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 --2- 172.21.15.98:0/2055258010 >> v2:172.21.15.190:3300/0 conn(0x7f5f3c06f670 0x7f5f3c06fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 monclient: found mon.noname-e 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 -- 172.21.15.98:0/2055258010 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c0717e0 con 0x7f5f3c0727b0 2024-09-11T02:50:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3affd640 1 --2- 172.21.15.98:0/2055258010 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f3c0727b0 0x7f5f3c06ff90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009870 tx=0x7f5f2401c420 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3659bbfc212305e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 1 -- 172.21.15.98:0/2055258010 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f240b2070 con 0x7f5f3c0727b0 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: dump: 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 1 -- 172.21.15.98:0/2055258010 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f240a9070 con 0x7f5f3c0727b0 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 5 monclient: authenticate success, global_id 4346 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 1 -- 172.21.15.98:0/2055258010 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f240abc40 con 0x7f5f3c0727b0 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 4 set_mon_vals no callback set 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f3a7fc640 10 monclient: dump: 2024-09-11T02:50:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 10 monclient: shutdown 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 -- 172.21.15.98:0/2055258010 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f3c0727b0 msgr2=0x7f5f3c06ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 --2- 172.21.15.98:0/2055258010 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f3c0727b0 0x7f5f3c06ff90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5f24009870 tx=0x7f5f2401c420 comp rx=0 tx=0).stop 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 -- 172.21.15.98:0/2055258010 shutdown_connections 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 --2- 172.21.15.98:0/2055258010 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c139c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 --2- 172.21.15.98:0/2055258010 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5f3c0727b0 0x7f5f3c06ff90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 --2- 172.21.15.98:0/2055258010 >> v2:172.21.15.190:3300/0 conn(0x7f5f3c06f670 0x7f5f3c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 -- 172.21.15.98:0/2055258010 >> 172.21.15.98:0/2055258010 conn(0x7f5f3c0ac950 msgr2=0x7f5f3c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.079+0000 7f5f41ade640 1 -- 172.21.15.98:0/2055258010 shutdown_connections 2024-09-11T02:50:10.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- 172.21.15.98:0/2055258010 wait complete. 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: build_initial_monmap 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: monmap: 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 librados: starting msgr at 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 librados: starting objecter 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 Processor -- start 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- start start 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 librados: setting wanted keys 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 librados: calling monclient init 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: init 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _add_conns ranks=[0,7,4,6,5,3,2,1,8] 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c06f670 0x7f5f3c1d1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): picked mon.noname-a con 0x7f5f3c06f670 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5f3c0727b0 0x7f5f3c138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): picked mon.noname-i con 0x7f5f3c0727b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c0a3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): picked mon.noname-c con 0x7f5f3c1397c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3c143440 con 0x7f5f3c06f670 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5f3c0a7d00 con 0x7f5f3c1397c0 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f5f3c1435c0 con 0x7f5f3c0727b0 2024-09-11T02:50:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient(hunting): authenticate will time out at 963.965331s 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c0a3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c06f670 0x7f5f3c1d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c0a3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39162/0 (socket says 172.21.15.98:39162) 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c06f670 0x7f5f3c1d1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37064/0 (socket says 172.21.15.98:37064) 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 -- 172.21.15.98:0/3065665636 learned_addr learned my addr 172.21.15.98:0/3065665636 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request con 0x7f5f3c1397c0 auth_method 0 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f3c06f670 auth_method 0 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3affd640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5f3c0727b0 0x7f5f3c138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3affd640 10 monclient(hunting): get_auth_request con 0x7f5f3c0727b0 auth_method 0 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 -- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5f3c0727b0 msgr2=0x7f5f3c138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5f3c0727b0 0x7f5f3c138910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 -- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c06f670 msgr2=0x7f5f3c1d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c06f670 0x7f5f3c1d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 monclient: found mon.noname-c 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 -- 172.21.15.98:0/3065665636 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3c0aba70 con 0x7f5f3c1397c0 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c06f670 0x7f5f3c1d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3bfff640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c0a3ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00e7f0 tx=0x7f5f2c00ecb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 client.?.objecter ms_handle_connect 0x7f5f3c1397c0 2024-09-11T02:50:10.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 1 -- 172.21.15.98:0/3065665636 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f2c017070 con 0x7f5f3c1397c0 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: dump: 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.100 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 1 -- 172.21.15.98:0/3065665636 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c00cdb0 con 0x7f5f3c1397c0 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 5 monclient: authenticate success, global_id 4351 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _renew_subs 2024-09-11T02:50:10.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f3c0a9f00 con 0x7f5f3c1397c0 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 1 -- 172.21.15.98:0/3065665636 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5f2c00eea0 con 0x7f5f3c1397c0 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f397fa640 4 set_mon_vals no callback set 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 monclient: dump: 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.102 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _renew_subs 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:10.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f3c123fd0 con 0x7f5f3c1397c0 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 librados: init done 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 1 -- 172.21.15.98:0/3065665636 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5f2c024060 con 0x7f5f3c1397c0 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f3c139c70 con 0x7f5f3c1397c0 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5f1c03df40 0x7f5f1c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 1 -- 172.21.15.98:0/3065665636 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f5f2c00f040 con 0x7f5f3c1397c0 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 10 client.4351.objecter ms_dispatch 0x7f5f3c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 3 client.4351.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 3 client.4351.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:10.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f33fff640 20 client.4351.objecter dump_active .. 0 homeless 2024-09-11T02:50:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5f1c03df40 0x7f5f1c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.083+0000 7f5f3b7fe640 10 monclient: get_auth_request con 0x7f5f1c03df40 auth_method 0 2024-09-11T02:50:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.087+0000 7f5f33fff640 10 client.4351.objecter ms_handle_connect 0x7f5f1c03df40 2024-09-11T02:50:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.087+0000 7f5f3b7fe640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5f1c03df40 0x7f5f1c040400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400d7f0 tx=0x7f5f340077b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.087+0000 7f5f33fff640 1 -- 172.21.15.98:0/3065665636 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f5f2c01a360 con 0x7f5f3c1397c0 2024-09-11T02:50:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.087+0000 7f5f33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.087+0000 7f5f33fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T02:50:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T02:50:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5f3c0a77d0 con 0x7f5f3c1397c0 2024-09-11T02:50:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f33fff640 1 -- 172.21.15.98:0/3065665636 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f5f2c04dbe0 con 0x7f5f3c1397c0 2024-09-11T02:50:10.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-11T02:50:10.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f33fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:10.251 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 10 librados: watch_flush enter 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 10 librados: watch_flush exit 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 20 client.4351.objecter shutdown clearing up homeless session... 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 10 client.4351.objecter successfully canceled tick 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5f1c03df40 msgr2=0x7f5f1c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5f1c03df40 0x7f5f1c040400 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5f3400d7f0 tx=0x7f5f340077b0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 10 monclient: shutdown 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 msgr2=0x7f5f3c0a3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c0a3ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c00e7f0 tx=0x7f5f2c00ecb0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 shutdown_connections 2024-09-11T02:50:10.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5f1c03df40 0x7f5f1c040400 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5f3c1397c0 0x7f5f3c0a3ed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5f3c0727b0 0x7f5f3c138910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 --2- 172.21.15.98:0/3065665636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f3c06f670 0x7f5f3c1d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 >> 172.21.15.98:0/3065665636 conn(0x7f5f3c0ac950 msgr2=0x7f5f3c0a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 shutdown_connections 2024-09-11T02:50:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 -- 172.21.15.98:0/3065665636 wait complete. 2024-09-11T02:50:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.251+0000 7f5f41ade640 1 librados: shutdown 2024-09-11T02:50:10.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-11T02:50:10.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient: get_monmap_and_config 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient: build_initial_monmap 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient: monmap: 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:10.390560+0000 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:10.390560+0000 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.387 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:10.388 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:10.388 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:10.388 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:10.388 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:10.389 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:10.389 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 Processor -- start 2024-09-11T02:50:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 -- start start 2024-09-11T02:50:10.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient: init 2024-09-11T02:50:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient: _add_conns ranks=[8,6,10,9,7,1,0,2,3,4,5] 2024-09-11T02:50:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f342c139b10 0x7f342c0725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): picked mon.noname-e con 0x7f342c139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:10.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f342c06fe30 0x7f342c139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): picked mon.noname-d con 0x7f342c06fe30 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f342c0a7a10 0x7f342c0a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): picked mon.noname-f con 0x7f342c0a7a10 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f342c0a9650 con 0x7f342c0a7a10 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f342c071350 con 0x7f342c06fe30 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f342c071630 con 0x7f342c139b10 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f34313fb640 10 monclient(hunting): authenticate will time out at 964.271076s 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f342c06fe30 0x7f342c139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f342c139b10 0x7f342c0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f342c06fe30 0x7f342c139ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39582/0 (socket says 172.21.15.98:39582) 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 -- 172.21.15.98:0/1993500632 learned_addr learned my addr 172.21.15.98:0/1993500632 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): get_auth_request con 0x7f342c06fe30 auth_method 0 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 10 monclient(hunting): get_auth_request con 0x7f342c139b10 auth_method 0 2024-09-11T02:50:10.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342b7fe640 1 --2- 172.21.15.98:0/1993500632 >> v2:172.21.15.190:3300/0 conn(0x7f342c0a7a10 0x7f342c0a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 -- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f342c139b10 msgr2=0x7f342c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:10.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 --2- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f342c139b10 0x7f342c0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 -- 172.21.15.98:0/1993500632 >> v2:172.21.15.190:3300/0 conn(0x7f342c0a7a10 msgr2=0x7f342c0a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 --2- 172.21.15.98:0/1993500632 >> v2:172.21.15.190:3300/0 conn(0x7f342c0a7a10 0x7f342c0a7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 monclient: found mon.noname-d 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342a7fc640 1 -- 172.21.15.98:0/1993500632 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c0702b0 con 0x7f342c06fe30 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.387+0000 7f342affd640 1 --2- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f342c139b10 0x7f342c0725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f342a7fc640 1 --2- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f342c06fe30 0x7f342c139ef0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3414009870 tx=0x7f341401c420 comp rx=0 tx=0).ready entity=mon.6 client_cookie=182a3b619898691f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 1 -- 172.21.15.98:0/1993500632 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34140b0070 con 0x7f342c06fe30 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:10.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: dump: 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.405 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:10.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 1 -- 172.21.15.98:0/1993500632 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34140aa810 con 0x7f342c06fe30 2024-09-11T02:50:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 1 -- 172.21.15.98:0/1993500632 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34140a9300 con 0x7f342c06fe30 2024-09-11T02:50:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:10.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f3429ffb640 10 monclient: dump: 2024-09-11T02:50:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.408 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.409 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.410 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 5 monclient: authenticate success, global_id 4371 2024-09-11T02:50:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 4 set_mon_vals no callback set 2024-09-11T02:50:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 10 monclient: shutdown 2024-09-11T02:50:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 -- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f342c06fe30 msgr2=0x7f342c139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 --2- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f342c06fe30 0x7f342c139ef0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3414009870 tx=0x7f341401c420 comp rx=0 tx=0).stop 2024-09-11T02:50:10.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 -- 172.21.15.98:0/1993500632 shutdown_connections 2024-09-11T02:50:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 --2- 172.21.15.98:0/1993500632 >> v2:172.21.15.190:3300/0 conn(0x7f342c0a7a10 0x7f342c0a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 --2- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f342c06fe30 0x7f342c139ef0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 --2- 172.21.15.98:0/1993500632 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f342c139b10 0x7f342c0725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 -- 172.21.15.98:0/1993500632 >> 172.21.15.98:0/1993500632 conn(0x7f342c0ac770 msgr2=0x7f342c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 -- 172.21.15.98:0/1993500632 shutdown_connections 2024-09-11T02:50:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 -- 172.21.15.98:0/1993500632 wait complete. 2024-09-11T02:50:10.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 10 monclient: build_initial_monmap 2024-09-11T02:50:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 10 monclient: monmap: 2024-09-11T02:50:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:10.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:10.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:10.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 librados: starting msgr at 2024-09-11T02:50:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 librados: starting objecter 2024-09-11T02:50:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 Processor -- start 2024-09-11T02:50:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 -- start start 2024-09-11T02:50:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 librados: setting wanted keys 2024-09-11T02:50:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 1 librados: calling monclient init 2024-09-11T02:50:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.391+0000 7f34313fb640 10 monclient: init 2024-09-11T02:50:10.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _add_conns ranks=[1,5,7,8,0,6,4,2,3] 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f342c0a7a10 0x7f342c1d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): picked mon.noname-f con 0x7f342c0a7a10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f342c139b10 0x7f342c13fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): picked mon.noname-h con 0x7f342c139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f342c140250 0x7f342c0a3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): picked mon.noname-i con 0x7f342c140250 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:10.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f342c1435c0 con 0x7f342c0a7a10 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f342c143440 con 0x7f342c139b10 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f342c143740 con 0x7f342c140250 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient(hunting): authenticate will time out at 964.276127s 2024-09-11T02:50:10.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f342c0a7a10 0x7f342c1d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f342c0a7a10 0x7f342c1d1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:35666/0 (socket says 172.21.15.98:35666) 2024-09-11T02:50:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 1 -- 172.21.15.98:0/57346034 learned_addr learned my addr 172.21.15.98:0/57346034 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient(hunting): get_auth_request con 0x7f342c0a7a10 auth_method 0 2024-09-11T02:50:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f342c140250 0x7f342c0a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f342c139b10 0x7f342c13fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 10 monclient(hunting): get_auth_request con 0x7f342c140250 auth_method 0 2024-09-11T02:50:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): get_auth_request con 0x7f342c139b10 auth_method 0 2024-09-11T02:50:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-11T02:50:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 -- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f342c140250 msgr2=0x7f342c0a3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f342c140250 0x7f342c0a3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 -- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f342c0a7a10 msgr2=0x7f342c1d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f342c0a7a10 0x7f342c1d1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 monclient: found mon.noname-h 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 -- 172.21.15.98:0/57346034 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c0733d0 con 0x7f342c139b10 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f342c0a7a10 0x7f342c1d1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:10.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342a7fc640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f342c139b10 0x7f342c13fd10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f342c0714d0 tx=0x7f34140191c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 client.?.objecter ms_handle_connect 0x7f342c139b10 2024-09-11T02:50:10.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:10.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 1 -- 172.21.15.98:0/57346034 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34140b0040 con 0x7f342c139b10 2024-09-11T02:50:10.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:50:10.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: dump: 2024-09-11T02:50:10.425 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.426 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.426 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.426 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.426 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.426 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.427 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.427 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.427 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.427 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.427 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 5 monclient: authenticate success, global_id 4352 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _renew_subs 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342c0a4b00 con 0x7f342c139b10 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 1 -- 172.21.15.98:0/57346034 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34140aacd0 con 0x7f342c139b10 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:10.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _renew_subs 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f342c0a5660 con 0x7f342c139b10 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 librados: init done 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34297fa640 4 set_mon_vals no callback set 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 1 -- 172.21.15.98:0/57346034 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34140aae80 con 0x7f342c139b10 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 monclient: dump: 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.429 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342c0a7ea0 con 0x7f342c139b10 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 1 -- 172.21.15.98:0/57346034 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3414004040 con 0x7f342c139b10 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f340c03e010 0x7f340c0404d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 1 -- 172.21.15.98:0/57346034 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f34140e8a50 con 0x7f342c139b10 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 10 client.4352.objecter ms_dispatch 0x7f342c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 3 client.4352.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 3 client.4352.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f34237fe640 20 client.4352.objecter dump_active .. 0 homeless 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f340c03e010 0x7f340c0404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.395+0000 7f342affd640 10 monclient: get_auth_request con 0x7f340c03e010 auth_method 0 2024-09-11T02:50:10.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.399+0000 7f342affd640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f340c03e010 0x7f340c0404d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f341c009600 tx=0x7f341c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.399+0000 7f34237fe640 10 client.4352.objecter ms_handle_connect 0x7f340c03e010 2024-09-11T02:50:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.411+0000 7f34237fe640 1 -- 172.21.15.98:0/57346034 <== mon.5 v2:172.21.15.190:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f34140c2080 con 0x7f342c139b10 2024-09-11T02:50:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.411+0000 7f34237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.411+0000 7f34237fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:10.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T02:50:10.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T02:50:10.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:50:10.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f342c06d210 con 0x7f342c139b10 2024-09-11T02:50:10.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34237fe640 1 -- 172.21.15.98:0/57346034 <== mon.5 v2:172.21.15.190:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (secure 0 0 0) 0x7f3414016130 con 0x7f342c139b10 2024-09-11T02:50:10.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-11T02:50:10.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34237fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:10.572 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 8 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 10 librados: watch_flush enter 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 10 librados: watch_flush exit 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 20 client.4352.objecter shutdown clearing up homeless session... 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.571+0000 7f34313fb640 10 client.4352.objecter successfully canceled tick 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f340c03e010 msgr2=0x7f340c0404d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f340c03e010 0x7f340c0404d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f341c009600 tx=0x7f341c009190 comp rx=0 tx=0).stop 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 10 monclient: shutdown 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:10.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f342c139b10 msgr2=0x7f342c13fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f342c139b10 0x7f342c13fd10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f342c0714d0 tx=0x7f34140191c0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 shutdown_connections 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f340c03e010 0x7f340c0404d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f342c140250 0x7f342c0a3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f342c139b10 0x7f342c13fd10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 --2- 172.21.15.98:0/57346034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f342c0a7a10 0x7f342c1d1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 >> 172.21.15.98:0/57346034 conn(0x7f342c0ac770 msgr2=0x7f342c0a9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 shutdown_connections 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 -- 172.21.15.98:0/57346034 wait complete. 2024-09-11T02:50:10.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.575+0000 7f34313fb640 1 librados: shutdown 2024-09-11T02:50:10.589 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-11T02:50:10.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T02:50:10.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.707+0000 7f720ffdc640 10 monclient: get_monmap_and_config 2024-09-11T02:50:10.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.707+0000 7f720ffdc640 10 monclient: build_initial_monmap 2024-09-11T02:50:10.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.707+0000 7f720ffdc640 10 monclient: monmap: 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:10.711806+0000 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:10.711806+0000 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:10.709 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:10.710 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.707+0000 7f720ffdc640 1 Processor -- start 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 1 -- start start 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient: init 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient: _add_conns ranks=[1,8,6,7,3,2,0,4,9,5,10] 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f72080a5930 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72080a1870 0x7f720813c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient(hunting): picked mon.noname-e con 0x7f72080a1870 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:10.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72080a6420 0x7f720813ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient(hunting): picked mon.noname-d con 0x7f72080a6420 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7208071d90 con 0x7f72080a5930 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f72080a4170 con 0x7f72080a6420 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7208073b50 con 0x7f72080a1870 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient(hunting): authenticate will time out at 964.592711s 2024-09-11T02:50:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72080a6420 0x7f720813ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72080a1870 0x7f720813c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72080a6420 0x7f720813ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39590/0 (socket says 172.21.15.98:39590) 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72080a1870 0x7f720813c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34998/0 (socket says 172.21.15.98:34998) 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 1 -- 172.21.15.98:0/2565048108 learned_addr learned my addr 172.21.15.98:0/2565048108 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 10 monclient(hunting): get_auth_request con 0x7f72080a6420 auth_method 0 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): get_auth_request con 0x7f72080a1870 auth_method 0 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720e579640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-09-11T02:50:10.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 -- 172.21.15.98:0/2565048108 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72080a6420 msgr2=0x7f720813ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 --2- 172.21.15.98:0/2565048108 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72080a6420 0x7f720813ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 -- 172.21.15.98:0/2565048108 >> v1:172.21.15.190:6789/0 conn(0x7f72080a5930 legacy=0x7f72080a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient: _finish_hunting 0 2024-09-11T02:50:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 monclient: found mon.noname-e 2024-09-11T02:50:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:10.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 -- 172.21.15.98:0/2565048108 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7208073870 con 0x7f72080a1870 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720d577640 1 --2- 172.21.15.98:0/2565048108 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72080a1870 0x7f720813c920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71f80097f0 tx=0x7f71f801c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=77c9c503602580a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 1 -- 172.21.15.98:0/2565048108 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71f80b0070 con 0x7f72080a1870 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: dump: 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.724 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: _finish_auth 0 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: _check_auth_tickets 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 1 -- 172.21.15.98:0/2565048108 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71f80aa810 con 0x7f72080a1870 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 5 monclient: authenticate success, global_id 4364 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 1 -- 172.21.15.98:0/2565048108 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71f80a9300 con 0x7f72080a1870 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720ffdc640 4 set_mon_vals no callback set 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.711+0000 7f720cd76640 10 monclient: dump: 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.727 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: shutdown 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:10.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/2565048108 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72080a1870 msgr2=0x7f720813c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 --2- 172.21.15.98:0/2565048108 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72080a1870 0x7f720813c920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f71f80097f0 tx=0x7f71f801c3a0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/2565048108 shutdown_connections 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 --2- 172.21.15.98:0/2565048108 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72080a6420 0x7f720813ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 --2- 172.21.15.98:0/2565048108 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72080a1870 0x7f720813c920 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/2565048108 >> 172.21.15.98:0/2565048108 conn(0x7f72080aa520 msgr2=0x7f72080a6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/2565048108 shutdown_connections 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/2565048108 wait complete. 2024-09-11T02:50:10.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: build_initial_monmap 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: monmap: 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:10.730 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:10.731 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 librados: starting msgr at 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 librados: starting objecter 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 Processor -- start 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- start start 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 librados: setting wanted keys 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 librados: calling monclient init 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: init 2024-09-11T02:50:10.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _add_conns ranks=[1,7,2,6,4,8,3,5,0] 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72080a1870 0x7f72081d1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): picked mon.noname-f con 0x7f72080a1870 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f72080a5930 0x7f720813da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): picked mon.noname-i con 0x7f72080a5930 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72080a6420 0x7f720813dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): picked mon.noname-b con 0x7f72080a6420 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:10.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7208141f60 con 0x7f72080a1870 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7208141de0 con 0x7f72080a6420 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f72081420e0 con 0x7f72080a5930 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient(hunting): authenticate will time out at 964.597676s 2024-09-11T02:50:10.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72080a6420 0x7f720813dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72080a6420 0x7f720813dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48158/0 (socket says 172.21.15.98:48158) 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 -- 172.21.15.98:0/3296488686 learned_addr learned my addr 172.21.15.98:0/3296488686 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): get_auth_request con 0x7f72080a6420 auth_method 0 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:10.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:10.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-09-11T02:50:10.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 -- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f72080a5930 msgr2=0x7f720813da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f72080a5930 0x7f720813da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 -- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72080a1870 msgr2=0x7f72081d1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72080a1870 0x7f72081d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient: _finish_hunting 0 2024-09-11T02:50:10.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 monclient: found mon.noname-b 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 -- 172.21.15.98:0/3296488686 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720813eed0 con 0x7f72080a6420 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720e579640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72080a6420 0x7f720813dfc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7200009cf0 tx=0x7f720000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 client.?.objecter ms_handle_connect 0x7f72080a6420 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 1 -- 172.21.15.98:0/3296488686 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72000039a0 con 0x7f72080a6420 2024-09-11T02:50:10.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: dump: 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.739 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.739 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.739 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.739 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.739 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.739 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.739 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 1 -- 172.21.15.98:0/3296488686 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7200003b60 con 0x7f72080a6420 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:10.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 1 -- 172.21.15.98:0/3296488686 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7200003d10 con 0x7f72080a6420 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 monclient: dump: 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:10.741 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fffff640 4 set_mon_vals no callback set 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 5 monclient: authenticate success, global_id 4340 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _renew_subs 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72080a9a40 con 0x7f72080a6420 2024-09-11T02:50:10.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _renew_subs 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7208128eb0 con 0x7f72080a6420 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 librados: init done 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72080a9760 con 0x7f72080a6420 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 1 -- 172.21.15.98:0/3296488686 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f720000cb20 con 0x7f72080a6420 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71ec03df70 0x7f71ec040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 1 -- 172.21.15.98:0/3296488686 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f720004f880 con 0x7f72080a6420 2024-09-11T02:50:10.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 10 client.4340.objecter ms_dispatch 0x7f7208000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:10.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 3 client.4340.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:10.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 3 client.4340.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:10.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.715+0000 7f71fe7fc640 20 client.4340.objecter dump_active .. 0 homeless 2024-09-11T02:50:10.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.719+0000 7f720dd78640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71ec03df70 0x7f71ec040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:10.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.719+0000 7f720dd78640 10 monclient: get_auth_request con 0x7f71ec03df70 auth_method 0 2024-09-11T02:50:10.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.719+0000 7f71fe7fc640 10 client.4340.objecter ms_handle_connect 0x7f71ec03df70 2024-09-11T02:50:10.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.719+0000 7f720dd78640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71ec03df70 0x7f71ec040430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7204003cf0 tx=0x7f72040079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:10.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.723+0000 7f71fe7fc640 1 -- 172.21.15.98:0/3296488686 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f72080a9760 con 0x7f72080a6420 2024-09-11T02:50:10.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.723+0000 7f71fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:10.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.723+0000 7f71fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.875+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f72080a5d10 con 0x7f71ec03df70 2024-09-11T02:50:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f71fe7fc640 1 -- 172.21.15.98:0/3296488686 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34066 (secure 0 0 0) 0x7f72080a5d10 con 0x7f71ec03df70 2024-09-11T02:50:10.878 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T02:50:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 10 librados: watch_flush enter 2024-09-11T02:50:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 10 librados: watch_flush exit 2024-09-11T02:50:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 20 client.4340.objecter shutdown clearing up homeless session... 2024-09-11T02:50:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 10 client.4340.objecter successfully canceled tick 2024-09-11T02:50:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71ec03df70 msgr2=0x7f71ec040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71ec03df70 0x7f71ec040430 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7204003cf0 tx=0x7f72040079c0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 10 monclient: shutdown 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72080a6420 msgr2=0x7f720813dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72080a6420 0x7f720813dfc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7200009cf0 tx=0x7f720000c490 comp rx=0 tx=0).stop 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 shutdown_connections 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71ec03df70 0x7f71ec040430 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72080a6420 0x7f720813dfc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f72080a5930 0x7f720813da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 --2- 172.21.15.98:0/3296488686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72080a1870 0x7f72081d1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 >> 172.21.15.98:0/3296488686 conn(0x7f72080aa520 msgr2=0x7f7208072770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 shutdown_connections 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 -- 172.21.15.98:0/3296488686 wait complete. 2024-09-11T02:50:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:10.879+0000 7f720ffdc640 1 librados: shutdown 2024-09-11T02:50:10.895 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-11T02:50:09.747788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164368,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066672,"statfs":{"total":579820584960,"available":579652272128,"internally_reserved":0,"allocated":2277376,"data_stored":1539766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001712"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.028180+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T02:50:05.028180+0000","last_peered":"2024-09-11T02:50:05.028180+0000","last_clean":"2024-09-11T02:50:05.028180+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T02:50:05.028180+0000","last_undegraded":"2024-09-11T02:50:05.028180+0000","last_fullsized":"2024-09-11T02:50:05.028180+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975717+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T02:50:04.975717+0000","last_peered":"2024-09-11T02:50:04.975717+0000","last_clean":"2024-09-11T02:50:04.975717+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T02:50:04.975717+0000","last_undegraded":"2024-09-11T02:50:04.975717+0000","last_fullsized":"2024-09-11T02:50:04.975717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027854+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T02:50:05.027854+0000","last_peered":"2024-09-11T02:50:05.027854+0000","last_clean":"2024-09-11T02:50:05.027854+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T02:50:05.027854+0000","last_undegraded":"2024-09-11T02:50:05.027854+0000","last_fullsized":"2024-09-11T02:50:05.027854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975572+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T02:50:04.975572+0000","last_peered":"2024-09-11T02:50:04.975572+0000","last_clean":"2024-09-11T02:50:04.975572+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T02:50:04.975572+0000","last_undegraded":"2024-09-11T02:50:04.975572+0000","last_fullsized":"2024-09-11T02:50:04.975572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.982831+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T02:50:04.982831+0000","last_peered":"2024-09-11T02:50:04.982831+0000","last_clean":"2024-09-11T02:50:04.982831+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T02:50:04.982831+0000","last_undegraded":"2024-09-11T02:50:04.982831+0000","last_fullsized":"2024-09-11T02:50:04.982831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975112+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T02:50:04.975112+0000","last_peered":"2024-09-11T02:50:04.975112+0000","last_clean":"2024-09-11T02:50:04.975112+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T02:50:04.975112+0000","last_undegraded":"2024-09-11T02:50:04.975112+0000","last_fullsized":"2024-09-11T02:50:04.975112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027630+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T02:50:05.027630+0000","last_peered":"2024-09-11T02:50:05.027630+0000","last_clean":"2024-09-11T02:50:05.027630+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T02:50:05.027630+0000","last_undegraded":"2024-09-11T02:50:05.027630+0000","last_fullsized":"2024-09-11T02:50:05.027630+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.974249+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T02:50:04.974249+0000","last_peered":"2024-09-11T02:50:04.974249+0000","last_clean":"2024-09-11T02:50:04.974249+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T02:50:04.974249+0000","last_undegraded":"2024-09-11T02:50:04.974249+0000","last_fullsized":"2024-09-11T02:50:04.974249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.026460+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T02:50:05.026460+0000","last_peered":"2024-09-11T02:50:05.026460+0000","last_clean":"2024-09-11T02:50:05.026460+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T02:50:05.026460+0000","last_undegraded":"2024-09-11T02:50:05.026460+0000","last_fullsized":"2024-09-11T02:50:05.026460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T02:50:10.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T02:50:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.011+0000 7fe4b5831640 10 monclient: get_monmap_and_config 2024-09-11T02:50:11.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.011+0000 7fe4b5831640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.011+0000 7fe4b5831640 10 monclient: monmap: 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:11.016374+0000 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:11.016374+0000 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.013 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:11.014 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:11.014 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 1 Processor -- start 2024-09-11T02:50:11.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 1 -- start start 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient: init 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient: _add_conns ranks=[8,4,1,6,5,0,3,2,10,9,7] 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b0071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient(hunting): picked mon.noname-e con 0x7fe4b00a5830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b00a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b00a5c10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe4b0139600 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b00707d0 con 0x7fe4b0139600 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe4b006fc70 con 0x7fe4b00a5c10 2024-09-11T02:50:11.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe4b00704f0 con 0x7fe4b00a5830 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4b5831640 10 monclient(hunting): authenticate will time out at 964.897155s 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b00a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b0071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b00a5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39174/0 (socket says 172.21.15.98:39174) 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 -- 172.21.15.98:0/2892631183 learned_addr learned my addr 172.21.15.98:0/2892631183 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): get_auth_request con 0x7fe4b00a5c10 auth_method 0 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 10 monclient(hunting): get_auth_request con 0x7fe4b00a5830 auth_method 0 2024-09-11T02:50:11.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 -- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 msgr2=0x7fe4b0071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 --2- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b0071ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 -- 172.21.15.98:0/2892631183 >> v1:172.21.15.190:6789/0 conn(0x7fe4b0139600 legacy=0x7fe4b0139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 monclient: found mon.noname-c 2024-09-11T02:50:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4ae7fc640 1 -- 172.21.15.98:0/2892631183 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b00a7350 con 0x7fe4b00a5c10 2024-09-11T02:50:11.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.015+0000 7fe4aeffd640 1 --2- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b0071ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:11.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 1 --2- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b00a5ff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098a0 tx=0x7fe4a401c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2e7711165c2a26f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 1 -- 172.21.15.98:0/2892631183 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a40b0070 con 0x7fe4b00a5c10 2024-09-11T02:50:11.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: dump: 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.026 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 1 -- 172.21.15.98:0/2892631183 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a40aa810 con 0x7fe4b00a5c10 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 5 monclient: authenticate success, global_id 4360 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 4 set_mon_vals no callback set 2024-09-11T02:50:11.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 1 -- 172.21.15.98:0/2892631183 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a40a9300 con 0x7fe4b00a5c10 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4adffb640 10 monclient: dump: 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.028 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient: shutdown 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 msgr2=0x7fe4b00a5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b00a5ff0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098a0 tx=0x7fe4a401c440 comp rx=0 tx=0).stop 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- 172.21.15.98:0/2892631183 shutdown_connections 2024-09-11T02:50:11.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b00a5ff0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2892631183 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b0071ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- 172.21.15.98:0/2892631183 >> 172.21.15.98:0/2892631183 conn(0x7fe4b00aa420 msgr2=0x7fe4b00a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- 172.21.15.98:0/2892631183 shutdown_connections 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- 172.21.15.98:0/2892631183 wait complete. 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient: monmap: 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:11.030 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 librados: starting msgr at 2024-09-11T02:50:11.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 librados: starting objecter 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 Processor -- start 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- start start 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 librados: setting wanted keys 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 librados: calling monclient init 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient: init 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient: _add_conns ranks=[8,4,5,0,7,1,2,6,3] 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b01214c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): picked mon.noname-e con 0x7fe4b00a5830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b0138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b00a5c10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe4b0139600 0x7fe4b0120470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): picked mon.noname-h con 0x7fe4b0139600 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe4b0144480 con 0x7fe4b00a5c10 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fe4b0144300 con 0x7fe4b0139600 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe4b0144600 con 0x7fe4b00a5830 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4b5831640 10 monclient(hunting): authenticate will time out at 964.903232s 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b01214c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b01214c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35018/0 (socket says 172.21.15.98:35018) 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 -- 172.21.15.98:0/2917926272 learned_addr learned my addr 172.21.15.98:0/2917926272 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:11.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): get_auth_request con 0x7fe4b00a5830 auth_method 0 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b0138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 10 monclient(hunting): get_auth_request con 0x7fe4b00a5c10 auth_method 0 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 -- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe4b0139600 msgr2=0x7fe4b0120470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe4b0139600 0x7fe4b0120470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 -- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 msgr2=0x7fe4b0138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b0138310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 monclient: found mon.noname-e 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 -- 172.21.15.98:0/2917926272 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b0140690 con 0x7fe4b00a5830 2024-09-11T02:50:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe4aeffd640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b01214c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000ec10 tx=0x7fe4a000c520 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe497fff640 10 client.?.objecter ms_handle_connect 0x7fe4b00a5830 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe497fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe497fff640 1 -- 172.21.15.98:0/2917926272 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a0017070 con 0x7fe4b00a5830 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe497fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.019+0000 7fe497fff640 10 monclient: dump: 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 1 -- 172.21.15.98:0/2917926272 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a000cc30 con 0x7fe4b00a5830 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 1 -- 172.21.15.98:0/2917926272 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4a000cde0 con 0x7fe4b00a5830 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 monclient: dump: 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.038 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4ad7fa640 4 set_mon_vals no callback set 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 5 monclient: authenticate success, global_id 4382 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 monclient: _renew_subs 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b00a7db0 con 0x7fe4b00a5830 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 monclient: _renew_subs 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b00a6080 con 0x7fe4b00a5830 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 1 librados: init done 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b0071ae0 con 0x7fe4b00a5830 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 1 -- 172.21.15.98:0/2917926272 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe4a0013470 con 0x7fe4b00a5830 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe49c03df40 0x7fe49c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 1 -- 172.21.15.98:0/2917926272 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fe4a000f040 con 0x7fe4b00a5830 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 client.4382.objecter ms_dispatch 0x7fe4b0000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 3 client.4382.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 3 client.4382.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 20 client.4382.objecter dump_active .. 0 homeless 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4ae7fc640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe49c03df40 0x7fe49c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4ae7fc640 10 monclient: get_auth_request con 0x7fe49c03df40 auth_method 0 2024-09-11T02:50:11.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe497fff640 10 client.4382.objecter ms_handle_connect 0x7fe49c03df40 2024-09-11T02:50:11.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.023+0000 7fe4ae7fc640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe49c03df40 0x7fe49c040400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40ad000 tx=0x7fe4a40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.027+0000 7fe497fff640 1 -- 172.21.15.98:0/2917926272 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe4a0021070 con 0x7fe4b00a5830 2024-09-11T02:50:11.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.027+0000 7fe497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.027+0000 7fe497fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:11.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.179+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4b0074100 con 0x7fe49c03df40 2024-09-11T02:50:11.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.179+0000 7fe497fff640 1 -- 172.21.15.98:0/2917926272 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34066 (secure 0 0 0) 0x7fe4b0074100 con 0x7fe49c03df40 2024-09-11T02:50:11.181 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T02:50:11.182 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T02:50:11.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 10 librados: watch_flush enter 2024-09-11T02:50:11.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 10 librados: watch_flush exit 2024-09-11T02:50:11.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 20 client.4382.objecter shutdown clearing up homeless session... 2024-09-11T02:50:11.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 10 client.4382.objecter successfully canceled tick 2024-09-11T02:50:11.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe49c03df40 msgr2=0x7fe49c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe49c03df40 0x7fe49c040400 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40ad000 tx=0x7fe4a40a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 10 monclient: shutdown 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 msgr2=0x7fe4b01214c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b01214c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000ec10 tx=0x7fe4a000c520 comp rx=0 tx=0).stop 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 shutdown_connections 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe49c03df40 0x7fe49c040400 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe4b0139600 0x7fe4b0120470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe4b00a5c10 0x7fe4b0138310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 --2- 172.21.15.98:0/2917926272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe4b00a5830 0x7fe4b01214c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 >> 172.21.15.98:0/2917926272 conn(0x7fe4b00aa420 msgr2=0x7fe4b0140d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 shutdown_connections 2024-09-11T02:50:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 -- 172.21.15.98:0/2917926272 wait complete. 2024-09-11T02:50:11.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.183+0000 7fe4b5831640 1 librados: shutdown 2024-09-11T02:50:11.198 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-11T02:50:09.747788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164368,"kb_used_data":2224,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066672,"statfs":{"total":579820584960,"available":579652272128,"internally_reserved":0,"allocated":2277376,"data_stored":1539766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":165927586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001712"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.028180+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T02:50:05.028180+0000","last_peered":"2024-09-11T02:50:05.028180+0000","last_clean":"2024-09-11T02:50:05.028180+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T02:50:05.028180+0000","last_undegraded":"2024-09-11T02:50:05.028180+0000","last_fullsized":"2024-09-11T02:50:05.028180+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975717+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T02:50:04.975717+0000","last_peered":"2024-09-11T02:50:04.975717+0000","last_clean":"2024-09-11T02:50:04.975717+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T02:50:04.975717+0000","last_undegraded":"2024-09-11T02:50:04.975717+0000","last_fullsized":"2024-09-11T02:50:04.975717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027854+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T02:50:05.027854+0000","last_peered":"2024-09-11T02:50:05.027854+0000","last_clean":"2024-09-11T02:50:05.027854+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T02:50:05.027854+0000","last_undegraded":"2024-09-11T02:50:05.027854+0000","last_fullsized":"2024-09-11T02:50:05.027854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975572+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T02:50:04.975572+0000","last_peered":"2024-09-11T02:50:04.975572+0000","last_clean":"2024-09-11T02:50:04.975572+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T02:50:04.975572+0000","last_undegraded":"2024-09-11T02:50:04.975572+0000","last_fullsized":"2024-09-11T02:50:04.975572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.982831+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T02:50:04.982831+0000","last_peered":"2024-09-11T02:50:04.982831+0000","last_clean":"2024-09-11T02:50:04.982831+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T02:50:04.982831+0000","last_undegraded":"2024-09-11T02:50:04.982831+0000","last_fullsized":"2024-09-11T02:50:04.982831+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.975112+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T02:50:04.975112+0000","last_peered":"2024-09-11T02:50:04.975112+0000","last_clean":"2024-09-11T02:50:04.975112+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T02:50:04.975112+0000","last_undegraded":"2024-09-11T02:50:04.975112+0000","last_fullsized":"2024-09-11T02:50:04.975112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.027630+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T02:50:05.027630+0000","last_peered":"2024-09-11T02:50:05.027630+0000","last_clean":"2024-09-11T02:50:05.027630+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T02:50:05.027630+0000","last_undegraded":"2024-09-11T02:50:05.027630+0000","last_fullsized":"2024-09-11T02:50:05.027630+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:04.974249+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T02:50:04.974249+0000","last_peered":"2024-09-11T02:50:04.974249+0000","last_clean":"2024-09-11T02:50:04.974249+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T02:50:04.974249+0000","last_undegraded":"2024-09-11T02:50:04.974249+0000","last_fullsized":"2024-09-11T02:50:04.974249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-11T02:50:05.026460+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T02:50:05.026460+0000","last_peered":"2024-09-11T02:50:05.026460+0000","last_clean":"2024-09-11T02:50:05.026460+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T02:50:05.026460+0000","last_undegraded":"2024-09-11T02:50:05.026460+0000","last_fullsized":"2024-09-11T02:50:05.026460+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":59832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":184320,"data_stored":59851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640262,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":774144,"data_stored":650200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T02:50:11.199 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-11T02:50:11.199 DEBUG:tasks.mon_thrash:Yielding 2024-09-11T02:50:11.199 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T02:50:11.212 INFO:tasks.workunit:Pulling workunits from ref 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T02:50:11.213 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T02:50:11.213 DEBUG:teuthology.orchestra.run.smithi190:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T02:50:11.213 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-11T02:50:11.214 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726023011, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-11T02:50:11.214 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:50:11.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:50:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T02:50:11.220 INFO:teuthology.orchestra.run.smithi190.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T02:50:11.221 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T02:50:11.270 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T02:50:11.270 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T02:50:11.318 INFO:tasks.workunit:timeout=3h 2024-09-11T02:50:11.318 INFO:tasks.workunit:cleanup=True 2024-09-11T02:50:11.318 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8f219c476cb4f26fc5719c661c0323719a45cb30 2024-09-11T02:50:11.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient: get_monmap_and_config 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient: monmap: 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:11.334697+0000 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:11.334697+0000 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.331 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:11.332 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 Processor -- start 2024-09-11T02:50:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 -- start start 2024-09-11T02:50:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient: init 2024-09-11T02:50:11.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient: _add_conns ranks=[4,5,8,6,2,9,10,1,0,7,3] 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf806f650 0x7fbaf8072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf806f650 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf80a8380 0x7fbaf806fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): picked mon.noname-h con 0x7fbaf80a8380 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbaf806ff70 0x7fbaf81397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): picked mon.noname-e con 0x7fbaf806ff70 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbaf80a8800 con 0x7fbaf806f650 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbaf8071a80 con 0x7fbaf80a8380 2024-09-11T02:50:11.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbaf8071d60 con 0x7fbaf806ff70 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbb00598640 10 monclient(hunting): authenticate will time out at 965.214972s 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf806f650 0x7fbaf8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf806f650 0x7fbaf8072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39190/0 (socket says 172.21.15.98:39190) 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 -- 172.21.15.98:0/2934348510 learned_addr learned my addr 172.21.15.98:0/2934348510 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): get_auth_request con 0x7fbaf806f650 auth_method 0 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbaf806ff70 0x7fbaf81397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 10 monclient(hunting): get_auth_request con 0x7fbaf806ff70 auth_method 0 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient(hunting): handle_auth_done global_id 4378 payload 274 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 -- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbaf806ff70 msgr2=0x7fbaf81397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbaf806ff70 0x7fbaf81397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafdb33640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf80a8380 0x7fbaf806fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 -- 172.21.15.98:0/2934348510 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf80a8380 msgr2=0x7fbaf806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:11.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf80a8380 0x7fbaf806fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 monclient: found mon.noname-c 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafe334640 1 -- 172.21.15.98:0/2934348510 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf80714c0 con 0x7fbaf806f650 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafeb35640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbaf806ff70 0x7fbaf81397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafdb33640 10 monclient: get_auth_request con 0x7fbaf80a8380 auth_method 0 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.331+0000 7fbafdb33640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf80a8380 0x7fbaf806fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafe334640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf806f650 0x7fbaf8072640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0098d0 tx=0x7fbaec01c180 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a88789be12acc4fb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 1 -- 172.21.15.98:0/2934348510 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbaec0b0070 con 0x7fbaf806f650 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:11.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: dump: 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 1 -- 172.21.15.98:0/2934348510 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec0aa810 con 0x7fbaf806f650 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 5 monclient: authenticate success, global_id 4378 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 4 set_mon_vals no callback set 2024-09-11T02:50:11.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 1 -- 172.21.15.98:0/2934348510 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbaec0a9300 con 0x7fbaf806f650 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbafd332640 10 monclient: dump: 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.343 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient: shutdown 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 -- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf806f650 msgr2=0x7fbaf8072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf806f650 0x7fbaf8072640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0098d0 tx=0x7fbaec01c180 comp rx=0 tx=0).stop 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 -- 172.21.15.98:0/2934348510 shutdown_connections 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbaf806ff70 0x7fbaf81397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf80a8380 0x7fbaf806fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 --2- 172.21.15.98:0/2934348510 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf806f650 0x7fbaf8072640 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 -- 172.21.15.98:0/2934348510 >> 172.21.15.98:0/2934348510 conn(0x7fbaf80ac8e0 msgr2=0x7fbaf80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 -- 172.21.15.98:0/2934348510 shutdown_connections 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 -- 172.21.15.98:0/2934348510 wait complete. 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient: monmap: 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 librados: starting msgr at 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 librados: starting objecter 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 Processor -- start 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 -- start start 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 librados: setting wanted keys 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 librados: calling monclient init 2024-09-11T02:50:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient: init 2024-09-11T02:50:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient: _add_conns ranks=[6,5,4,2,3,8,0,7,1] 2024-09-11T02:50:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbaf806f650 0x7fbaf81d1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient(hunting): picked mon.noname-d con 0x7fbaf806f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf806ff70 0x7fbaf8138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 10 monclient(hunting): picked mon.noname-h con 0x7fbaf806ff70 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.335+0000 7fbb00598640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf80a8380 0x7fbaf813dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient(hunting): picked mon.noname-c con 0x7fbaf80a8380 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbaf8142fc0 con 0x7fbaf80a8380 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbaf8142e40 con 0x7fbaf806ff70 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbaf8143140 con 0x7fbaf806f650 2024-09-11T02:50:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient(hunting): authenticate will time out at 965.220028s 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf80a8380 0x7fbaf813dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbaf806f650 0x7fbaf81d1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf80a8380 0x7fbaf813dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39196/0 (socket says 172.21.15.98:39196) 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbaf806f650 0x7fbaf81d1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39592/0 (socket says 172.21.15.98:39592) 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 -- 172.21.15.98:0/4288494465 learned_addr learned my addr 172.21.15.98:0/4288494465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): get_auth_request con 0x7fbaf80a8380 auth_method 0 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient(hunting): get_auth_request con 0x7fbaf806f650 auth_method 0 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafdb33640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf806ff70 0x7fbaf8138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 -- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbaf806f650 msgr2=0x7fbaf81d1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbaf806f650 0x7fbaf81d1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 -- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf806ff70 msgr2=0x7fbaf8138b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf806ff70 0x7fbaf8138b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 monclient: found mon.noname-c 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 -- 172.21.15.98:0/4288494465 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf81244b0 con 0x7fbaf80a8380 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafdb33640 10 monclient: get_auth_request con 0x7fbaf806ff70 auth_method 0 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafdb33640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf806ff70 0x7fbaf8138b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbaf806f650 0x7fbaf81d1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafeb35640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf80a8380 0x7fbaf813dd80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000e920 tx=0x7fbaf000ede0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 client.?.objecter ms_handle_connect 0x7fbaf80a8380 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 1 -- 172.21.15.98:0/4288494465 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbaf0017070 con 0x7fbaf80a8380 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: dump: 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 1 -- 172.21.15.98:0/4288494465 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf000ce10 con 0x7fbaf80a8380 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 5 monclient: authenticate success, global_id 4387 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient: _renew_subs 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:11.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf813ff80 con 0x7fbaf80a8380 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 1 -- 172.21.15.98:0/4288494465 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbaf00138e0 con 0x7fbaf80a8380 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafcb31640 4 set_mon_vals no callback set 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 monclient: dump: 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.357 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient: _renew_subs 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf8072920 con 0x7fbaf80a8380 2024-09-11T02:50:11.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 1 librados: init done 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf81397c0 con 0x7fbaf80a8380 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 1 -- 172.21.15.98:0/4288494465 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbaf0024060 con 0x7fbaf80a8380 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbadc03df40 0x7fbadc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 1 -- 172.21.15.98:0/4288494465 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fbaf000f040 con 0x7fbaf80a8380 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 client.4387.objecter ms_dispatch 0x7fbaf8000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 3 client.4387.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 3 client.4387.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 20 client.4387.objecter dump_active .. 0 homeless 2024-09-11T02:50:11.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbadc03df40 0x7fbadc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 10 monclient: get_auth_request con 0x7fbadc03df40 auth_method 0 2024-09-11T02:50:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbaeaffd640 10 client.4387.objecter ms_handle_connect 0x7fbadc03df40 2024-09-11T02:50:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.339+0000 7fbafe334640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbadc03df40 0x7fbadc040400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0ad000 tx=0x7fbaec004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.343+0000 7fbaeaffd640 1 -- 172.21.15.98:0/4288494465 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbaf0009380 con 0x7fbaf80a8380 2024-09-11T02:50:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.343+0000 7fbaeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.343+0000 7fbaeaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:11.371 INFO:tasks.workunit.client.0.smithi190.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T02:50:11.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.571+0000 7fbb00598640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:50:11.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.571+0000 7fbb00598640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:11.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.571+0000 7fbb00598640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:11.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.571+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbaf813f840 con 0x7fbaf80a8380 2024-09-11T02:50:11.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.571+0000 7fbaeaffd640 1 -- 172.21.15.98:0/4288494465 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fbaf001ccd0 con 0x7fbaf80a8380 2024-09-11T02:50:11.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.571+0000 7fbaeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:11.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.571+0000 7fbaeaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:11.572 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 10 librados: watch_flush enter 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 10 librados: watch_flush exit 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 20 client.4387.objecter shutdown clearing up homeless session... 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 10 client.4387.objecter successfully canceled tick 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbadc03df40 msgr2=0x7fbadc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbadc03df40 0x7fbadc040400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbaec0ad000 tx=0x7fbaec004740 comp rx=0 tx=0).stop 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 10 monclient: shutdown 2024-09-11T02:50:11.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf80a8380 msgr2=0x7fbaf813dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf80a8380 0x7fbaf813dd80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000e920 tx=0x7fbaf000ede0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 shutdown_connections 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbadc03df40 0x7fbadc040400 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbaf80a8380 0x7fbaf813dd80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbaf806ff70 0x7fbaf8138b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 --2- 172.21.15.98:0/4288494465 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbaf806f650 0x7fbaf81d1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 >> 172.21.15.98:0/4288494465 conn(0x7fbaf80ac8e0 msgr2=0x7fbaf80a9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 shutdown_connections 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 -- 172.21.15.98:0/4288494465 wait complete. 2024-09-11T02:50:11.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.575+0000 7fbb00598640 1 librados: shutdown 2024-09-11T02:50:11.590 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:50:11.590 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T02:50:11.590 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:50:11.590 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:50:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.695+0000 7fafd8905640 10 monclient: get_monmap_and_config 2024-09-11T02:50:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.695+0000 7fafd8905640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.695+0000 7fafd8905640 10 monclient: monmap: 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:11.700348+0000 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:11.700348+0000 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.697 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:11.698 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:11.698 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.695+0000 7fafd8905640 1 Processor -- start 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.695+0000 7fafd8905640 1 -- start start 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.695+0000 7fafd8905640 10 monclient: init 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient: _add_conns ranks=[1,10,2,6,4,7,5,8,0,9,3] 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fafd40a7030 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fafd406f720 0x7fafd413ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient(hunting): picked mon.noname-f con 0x7fafd406f720 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd413a2b0 0x7fafd4140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient(hunting): picked mon.noname-b con 0x7fafd413a2b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd406ee20 con 0x7fafd40a7030 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fafd40a7410 con 0x7fafd406f720 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fafd406f4e0 con 0x7fafd413a2b0 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient(hunting): authenticate will time out at 965.580718s 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd413a2b0 0x7fafd4140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd413a2b0 0x7fafd4140f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48172/0 (socket says 172.21.15.98:48172) 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1676893276 learned_addr learned my addr 172.21.15.98:0/1676893276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): get_auth_request con 0x7fafd413a2b0 auth_method 0 2024-09-11T02:50:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd1d9b640 1 --2- 172.21.15.98:0/1676893276 >> v2:172.21.15.190:3300/0 conn(0x7fafd406f720 0x7fafd413ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1676893276 >> v2:172.21.15.190:3300/0 conn(0x7fafd406f720 msgr2=0x7fafd413ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 --2- 172.21.15.98:0/1676893276 >> v2:172.21.15.190:3300/0 conn(0x7fafd406f720 0x7fafd413ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1676893276 >> v1:172.21.15.190:6789/0 conn(0x7fafd40a7030 legacy=0x7fafd4139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 monclient: found mon.noname-b 2024-09-11T02:50:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1676893276 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd4070580 con 0x7fafd413a2b0 2024-09-11T02:50:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd1d9b640 10 monclient: get_auth_request con 0x7fafd406f720 auth_method 0 2024-09-11T02:50:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd1d9b640 1 --2- 172.21.15.98:0/1676893276 >> v2:172.21.15.190:3300/0 conn(0x7fafd406f720 0x7fafd413ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd2d9d640 1 --2- 172.21.15.98:0/1676893276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd413a2b0 0x7fafd4140f50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafd40a0ef0 tx=0x7fafcc01c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cb1b48afac31954 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 1 -- 172.21.15.98:0/1676893276 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafcc0090d0 con 0x7fafd413a2b0 2024-09-11T02:50:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: dump: 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 5 monclient: authenticate success, global_id 4349 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 1 -- 172.21.15.98:0/1676893276 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafcc0a5a90 con 0x7fafd413a2b0 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 4 set_mon_vals no callback set 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 1 -- 172.21.15.98:0/1676893276 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafcc0a5c80 con 0x7fafd413a2b0 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd159a640 10 monclient: dump: 2024-09-11T02:50:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.718 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 10 monclient: shutdown 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 -- 172.21.15.98:0/1676893276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd413a2b0 msgr2=0x7fafd4140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 --2- 172.21.15.98:0/1676893276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd413a2b0 0x7fafd4140f50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fafd40a0ef0 tx=0x7fafcc01c880 comp rx=0 tx=0).stop 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 -- 172.21.15.98:0/1676893276 shutdown_connections 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 --2- 172.21.15.98:0/1676893276 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fafd413a2b0 0x7fafd4140f50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 --2- 172.21.15.98:0/1676893276 >> v2:172.21.15.190:3300/0 conn(0x7fafd406f720 0x7fafd413ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.699+0000 7fafd8905640 1 -- 172.21.15.98:0/1676893276 >> 172.21.15.98:0/1676893276 conn(0x7fafd40aa420 msgr2=0x7fafd40a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- 172.21.15.98:0/1676893276 shutdown_connections 2024-09-11T02:50:11.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- 172.21.15.98:0/1676893276 wait complete. 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: monmap: 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.720 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 librados: starting msgr at 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 librados: starting objecter 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 Processor -- start 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- start start 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 librados: setting wanted keys 2024-09-11T02:50:11.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 librados: calling monclient init 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: init 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: _add_conns ranks=[3,5,6,1,8,4,0,7,2] 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafd406f720 0x7fafd4120470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): picked mon.noname-g con 0x7fafd406f720 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fafd40a7030 0x7fafd41209b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): picked mon.noname-h con 0x7fafd40a7030 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fafd413a2b0 0x7fafd4120ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): picked mon.noname-d con 0x7fafd413a2b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fafd4144040 con 0x7fafd406f720 2024-09-11T02:50:11.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fafd4143ec0 con 0x7fafd40a7030 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fafd41441c0 con 0x7fafd413a2b0 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): authenticate will time out at 965.585511s 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fafd413a2b0 0x7fafd4120ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fafd413a2b0 0x7fafd4120ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39608/0 (socket says 172.21.15.98:39608) 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1917614385 learned_addr learned my addr 172.21.15.98:0/1917614385 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): get_auth_request con 0x7fafd413a2b0 auth_method 0 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafd406f720 0x7fafd4120470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd1d9b640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fafd40a7030 0x7fafd41209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd1d9b640 10 monclient(hunting): get_auth_request con 0x7fafd40a7030 auth_method 0 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): get_auth_request con 0x7fafd406f720 auth_method 0 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fafd40a7030 msgr2=0x7fafd41209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fafd40a7030 0x7fafd41209b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafd406f720 msgr2=0x7fafd4120470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fafd406f720 0x7fafd4120470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 monclient: found mon.noname-d 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1917614385 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd4122c40 con 0x7fafd413a2b0 2024-09-11T02:50:11.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd2d9d640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fafd413a2b0 0x7fafd4120ef0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fafcc01c6b0 tx=0x7fafcc019620 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 client.?.objecter ms_handle_connect 0x7fafd413a2b0 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafcc0bc020 con 0x7fafd413a2b0 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: dump: 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.726 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 5 monclient: authenticate success, global_id 4398 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: _renew_subs 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafcc003bb0 con 0x7fafd413a2b0 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd40a6700 con 0x7fafd413a2b0 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafcc003da0 con 0x7fafd413a2b0 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd0d99640 4 set_mon_vals no callback set 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:11.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: dump: 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.729 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: _renew_subs 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd413fc90 con 0x7fafd413a2b0 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 librados: init done 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd41230d0 0x7fafd41d1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafd406ca10 con 0x7fafd41230d0 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd41230d0 0x7fafd41d1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient: get_auth_request con 0x7fafd41230d0 auth_method 0 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fafcc004040 con 0x7fafd413a2b0 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 10 monclient(hunting): handle_auth_done global_id 4500 payload 824 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 1 -- 172.21.15.98:0/1917614385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fafd4139ba0 con 0x7fafd41230d0 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fafb403df40 0x7fafb4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fafcc0b3040 con 0x7fafd413a2b0 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 client.4398.objecter ms_dispatch 0x7fafd4000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 3 client.4398.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 3 client.4398.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 20 client.4398.objecter dump_active .. 0 homeless 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 client.4398.objecter ms_handle_connect 0x7fafd41230d0 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd259c640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd41230d0 0x7fafd41d1890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e450 tx=0x7fafc400e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 client.4398.objecter resend_mon_ops 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd1d9b640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fafb403df40 0x7fafb4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafc40076b0 con 0x7fafd41230d0 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafc2ffd640 10 monclient: dump: 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.733 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.703+0000 7fafd1d9b640 10 monclient: get_auth_request con 0x7fafb403df40 auth_method 0 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fafc4007240 con 0x7fafd41230d0 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd41230d0 msgr2=0x7fafd41d1890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd41230d0 0x7fafd41d1890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e450 tx=0x7fafc400e910 comp rx=0 tx=0).stop 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd41230d0 msgr2=0x7fafd41d1890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd41230d0 0x7fafd41d1890 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e450 tx=0x7fafc400e910 comp rx=0 tx=0).stop 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafc2ffd640 10 client.4398.objecter ms_handle_connect 0x7fafb403df40 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.707+0000 7fafd1d9b640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fafb403df40 0x7fafb4040400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafd413f340 tx=0x7fafc8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd402d450 0x7fafd402d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafd406e870 con 0x7fafd402d450 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd402d450 0x7fafd402d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient: get_auth_request con 0x7fafd402d450 auth_method 0 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 10 monclient(hunting): handle_auth_done global_id 4509 payload 824 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 1 -- 172.21.15.98:0/1917614385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fafd413bee0 con 0x7fafd402d450 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 10 client.4398.objecter ms_handle_connect 0x7fafd402d450 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 10 client.4398.objecter resend_mon_ops 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd259c640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd402d450 0x7fafd402d860 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e450 tx=0x7fafc4003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafc4006c80 con 0x7fafd402d450 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 10 monclient: dump: 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.737 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7fafc4006c80 con 0x7fafd402d450 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd402d450 msgr2=0x7fafd402d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd402d450 0x7fafd402d860 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e450 tx=0x7fafc4003770 comp rx=0 tx=0).stop 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd402d450 msgr2=0x7fafd402d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafc2ffd640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafd402d450 0x7fafd402d860 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e450 tx=0x7fafc4003770 comp rx=0 tx=0).stop 2024-09-11T02:50:11.740 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:50:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:11.740 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 27515, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:11.741 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 27, 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:11.742 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:11.743 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:11.744 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:11.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:11.746 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.747 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:11.748 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.751 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:11.753 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.754 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:11.757 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:11.764 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.765 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.766 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:11.767 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:11.767 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:11.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.767 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.767 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:11.767 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 10 librados: watch_flush enter 2024-09-11T02:50:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 10 librados: watch_flush exit 2024-09-11T02:50:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 20 client.4398.objecter shutdown clearing up homeless session... 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 10 client.4398.objecter successfully canceled tick 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fafb403df40 msgr2=0x7fafb4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fafb403df40 0x7fafb4040400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafd413f340 tx=0x7fafc8008040 comp rx=0 tx=0).stop 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 10 monclient: shutdown 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fafd413a2b0 msgr2=0x7fafd4120ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd2d9d640 1 -- 172.21.15.98:0/1917614385 reap_dead start 2024-09-11T02:50:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fafd413a2b0 0x7fafd4120ef0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fafcc01c6b0 tx=0x7fafcc019620 comp rx=0 tx=0).stop 2024-09-11T02:50:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 shutdown_connections 2024-09-11T02:50:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 --2- 172.21.15.98:0/1917614385 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fafd413a2b0 0x7fafd4120ef0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 >> 172.21.15.98:0/1917614385 conn(0x7fafd40aa420 msgr2=0x7fafd40a3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 shutdown_connections 2024-09-11T02:50:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 -- 172.21.15.98:0/1917614385 wait complete. 2024-09-11T02:50:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.711+0000 7fafd8905640 1 librados: shutdown 2024-09-11T02:50:11.771 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:11.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:50:11.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:50:11.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.879+0000 7f0e67d0a640 10 monclient: get_monmap_and_config 2024-09-11T02:50:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.879+0000 7f0e67d0a640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.879+0000 7f0e67d0a640 10 monclient: monmap: 2024-09-11T02:50:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.882 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:11.885078+0000 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:11.885078+0000 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.883 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:11.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 1 Processor -- start 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 1 -- start start 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient: init 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient: _add_conns ranks=[7,8,0,4,2,6,1,10,3,9,5] 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a3de0 0x7f0e600a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient(hunting): picked mon.noname-i con 0x7f0e600a3de0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0e600a4700 0x7f0e6009b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient(hunting): picked mon.noname-e con 0x7f0e600a4700 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e6009b680 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e60068fa0 con 0x7f0e6009b680 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0e600a2200 con 0x7f0e600a3de0 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0e600a18d0 con 0x7f0e600a4700 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e67d0a640 10 monclient(hunting): authenticate will time out at 965.765469s 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e662a7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0e6009b680 0x7f0e6013d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50944/0 (socket says 172.21.15.98:50944) 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0e600a4700 0x7f0e6009b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e662a7640 1 -- 172.21.15.98:0/4240244009 learned_addr learned my addr 172.21.15.98:0/4240244009 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): get_auth_request con 0x7f0e600a4700 auth_method 0 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e65aa6640 1 --2- 172.21.15.98:0/4240244009 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a3de0 0x7f0e600a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e65aa6640 10 monclient(hunting): get_auth_request con 0x7f0e600a3de0 auth_method 0 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e65aa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e65aa6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e65aa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient(hunting): handle_auth_done global_id 4400 payload 274 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 1 -- 172.21.15.98:0/4240244009 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a3de0 msgr2=0x7f0e600a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 1 --2- 172.21.15.98:0/4240244009 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a3de0 0x7f0e600a41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e64aa4640 1 -- 172.21.15.98:0/4240244009 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3089093225 0 0) 0x7f0e60068fa0 con 0x7f0e6009b680 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 1 -- 172.21.15.98:0/4240244009 >> v1:172.21.15.98:6789/0 conn(0x7f0e6009b680 legacy=0x7f0e6013d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 1 monclient: found mon.noname-e 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 1 -- 172.21.15.98:0/4240244009 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e600a1bb0 con 0x7f0e600a4700 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e64aa4640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.883+0000 7f0e652a5640 1 --2- 172.21.15.98:0/4240244009 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0e600a4700 0x7f0e6009b140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0e500097f0 tx=0x7f0e5001c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c411d943cf59b19e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 1 -- 172.21.15.98:0/4240244009 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0e500b0070 con 0x7f0e600a4700 2024-09-11T02:50:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: dump: 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 5 monclient: authenticate success, global_id 4400 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 1 -- 172.21.15.98:0/4240244009 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e500aa810 con 0x7f0e600a4700 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 4 set_mon_vals no callback set 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 1 -- 172.21.15.98:0/4240244009 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0e500a9300 con 0x7f0e600a4700 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e64aa4640 10 monclient: dump: 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.905 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: shutdown 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/4240244009 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0e600a4700 msgr2=0x7f0e6009b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/4240244009 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0e600a4700 0x7f0e6009b140 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0e500097f0 tx=0x7f0e5001c3a0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/4240244009 shutdown_connections 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/4240244009 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0e600a4700 0x7f0e6009b140 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/4240244009 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a3de0 0x7f0e600a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/4240244009 >> 172.21.15.98:0/4240244009 conn(0x7f0e600a4ed0 msgr2=0x7f0e600a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/4240244009 shutdown_connections 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/4240244009 wait complete. 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: build_initial_monmap 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: monmap: 2024-09-11T02:50:11.906 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:11.907 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 librados: starting msgr at 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 librados: starting objecter 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 Processor -- start 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- start start 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 librados: setting wanted keys 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 librados: calling monclient init 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: init 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: _add_conns ranks=[2,3,7,8,6,1,0,5,4] 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009b680 0x7f0e60136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6009b680 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e600a3de0 0x7f0e600707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): picked mon.noname-g con 0x7f0e600a3de0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a4700 0x7f0e60070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): picked mon.noname-i con 0x7f0e600a4700 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0e600a1020 con 0x7f0e6009b680 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0e600a0ea0 con 0x7f0e600a3de0 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0e600a2670 con 0x7f0e600a4700 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): authenticate will time out at 965.770257s 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009b680 0x7f0e60136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009b680 0x7f0e60136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48174/0 (socket says 172.21.15.98:48174) 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 -- 172.21.15.98:0/1253235137 learned_addr learned my addr 172.21.15.98:0/1253235137 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): get_auth_request con 0x7f0e6009b680 auth_method 0 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e662a7640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a4700 0x7f0e60070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e600a3de0 0x7f0e600707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a4700 msgr2=0x7f0e60070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0e600a4700 0x7f0e60070cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e600a3de0 msgr2=0x7f0e600707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e600a3de0 0x7f0e600707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient: _finish_hunting 0 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 monclient: found mon.noname-b 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 -- 172.21.15.98:0/1253235137 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6013b850 con 0x7f0e6009b680 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 10 monclient: get_auth_request con 0x7f0e600a3de0 auth_method 0 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0e600a3de0 0x7f0e600707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e65aa6640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009b680 0x7f0e60136de0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00d800 tx=0x7f0e5c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 client.?.objecter ms_handle_connect 0x7f0e6009b680 2024-09-11T02:50:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0e5c00bb30 con 0x7f0e6009b680 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: dump: 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 5 monclient: authenticate success, global_id 4358 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e5c019070 con 0x7f0e6009b680 2024-09-11T02:50:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: _renew_subs 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6013e940 con 0x7f0e6009b680 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0e5c00bcd0 con 0x7f0e6009b680 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e57fff640 4 set_mon_vals no callback set 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 10 monclient: dump: 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.916 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: _renew_subs 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6013a8f0 con 0x7f0e6009b680 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 librados: init done 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e60071230 0x7f0e60071610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6013c050 con 0x7f0e60071230 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e60071230 0x7f0e60071610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 10 monclient: get_auth_request con 0x7f0e60071230 auth_method 0 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e652a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.887+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0e5c013070 con 0x7f0e6009b680 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e652a5640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e652a5640 1 -- 172.21.15.98:0/1253235137 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e60074080 con 0x7f0e60071230 2024-09-11T02:50:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0e4403df70 0x7f0e44040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f0e5c00f040 con 0x7f0e6009b680 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 client.4358.objecter ms_dispatch 0x7f0e60000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 3 client.4358.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 3 client.4358.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e652a5640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e60071230 0x7f0e60071610 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e6006f4e0 tx=0x7f0e50019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 20 client.4358.objecter dump_active .. 0 homeless 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 client.4358.objecter ms_handle_connect 0x7f0e60071230 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 client.4358.objecter resend_mon_ops 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0e500bb030 con 0x7f0e60071230 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 monclient: dump: 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e662a7640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0e4403df70 0x7f0e44040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e662a7640 10 monclient: get_auth_request con 0x7f0e4403df70 auth_method 0 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0e50003620 con 0x7f0e60071230 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e60071230 msgr2=0x7f0e60071610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e60071230 0x7f0e60071610 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e6006f4e0 tx=0x7f0e50019140 comp rx=0 tx=0).stop 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e60071230 msgr2=0x7f0e60071610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e60071230 0x7f0e60071610 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e662a7640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0e4403df70 0x7f0e44040430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0e60071df0 tx=0x7f0e58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.891+0000 7f0e567fc640 10 client.4358.objecter ms_handle_connect 0x7f0e4403df70 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e67d0a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009a2b0 0x7f0e6009a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e67d0a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6006ca10 con 0x7f0e6009a2b0 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009a2b0 0x7f0e6009a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient: get_auth_request con 0x7f0e6009a2b0 auth_method 0 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-09-11T02:50:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 1 -- 172.21.15.98:0/1253235137 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0e600a41c0 con 0x7f0e6009a2b0 2024-09-11T02:50:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e652a5640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009a2b0 0x7f0e6009a6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e5001aa50 tx=0x7f0e500a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 10 client.4358.objecter ms_handle_connect 0x7f0e6009a2b0 2024-09-11T02:50:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 10 client.4358.objecter resend_mon_ops 2024-09-11T02:50:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0e500bb070 con 0x7f0e6009a2b0 2024-09-11T02:50:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 10 monclient: dump: 2024-09-11T02:50:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f0e500ca480 con 0x7f0e6009a2b0 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009a2b0 msgr2=0x7f0e6009a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009a2b0 0x7f0e6009a6c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e5001aa50 tx=0x7f0e500a8000 comp rx=0 tx=0).stop 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009a2b0 msgr2=0x7f0e6009a6c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e567fc640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009a2b0 0x7f0e6009a6c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.926 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 27679, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:11.927 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 27, 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:11.928 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:11.929 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:11.930 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:11.931 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:11.932 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:11.933 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:11.933 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:11.933 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:11.933 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:11.933 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.934 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:11.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.935 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:11.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.935 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:11.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:11.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.939 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:11.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:11.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:11.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.943 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:11.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:11.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:11.952 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:11.953 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:11.954 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:11.954 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:11.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:11.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:11.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:11.954 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:11.954 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.895+0000 7f0e67d0a640 10 librados: watch_flush enter 2024-09-11T02:50:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 10 librados: watch_flush exit 2024-09-11T02:50:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 20 client.4358.objecter shutdown clearing up homeless session... 2024-09-11T02:50:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 10 client.4358.objecter successfully canceled tick 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0e4403df70 msgr2=0x7f0e44040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0e4403df70 0x7f0e44040430 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0e60071df0 tx=0x7f0e58009040 comp rx=0 tx=0).stop 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 10 monclient: shutdown 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009b680 msgr2=0x7f0e60136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e662a7640 1 -- 172.21.15.98:0/1253235137 reap_dead start 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009b680 0x7f0e60136de0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c00d800 tx=0x7f0e5c00dcc0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 shutdown_connections 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 --2- 172.21.15.98:0/1253235137 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0e6009b680 0x7f0e60136de0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 >> 172.21.15.98:0/1253235137 conn(0x7f0e600a4ed0 msgr2=0x7f0e6009f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 shutdown_connections 2024-09-11T02:50:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 -- 172.21.15.98:0/1253235137 wait complete. 2024-09-11T02:50:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:11.899+0000 7f0e67d0a640 1 librados: shutdown 2024-09-11T02:50:11.958 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:11.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:50:11.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:50:12.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient: get_monmap_and_config 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient: monmap: 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:12.077483+0000 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:12.077483+0000 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.074 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.075 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.075 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.075 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.075 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.075 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:12.075 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:12.075 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 Processor -- start 2024-09-11T02:50:12.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 -- start start 2024-09-11T02:50:12.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient: init 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient: _add_conns ranks=[6,3,10,2,9,4,1,0,7,8,5] 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c88072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): picked mon.noname-d con 0x7f1c88139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c88139ef0 0x7f1c880a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): picked mon.noname-g con 0x7f1c88139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1c8813a2d0 0x7f1c880a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): picked mon.noname-f con 0x7f1c8813a2d0 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1c880702b0 con 0x7f1c8813a2d0 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1c88071bf0 con 0x7f1c88139ef0 2024-09-11T02:50:12.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1c88071350 con 0x7f1c88139b10 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c8d2b4640 10 monclient(hunting): authenticate will time out at 965.957889s 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c88072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c88072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39610/0 (socket says 172.21.15.98:39610) 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 -- 172.21.15.98:0/1401256871 learned_addr learned my addr 172.21.15.98:0/1401256871 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): get_auth_request con 0x7f1c88139b10 auth_method 0 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c877fe640 1 --2- 172.21.15.98:0/1401256871 >> v2:172.21.15.190:3300/0 conn(0x7f1c8813a2d0 0x7f1c880a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c7ffff640 1 --2- 172.21.15.98:0/1401256871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c88139ef0 0x7f1c880a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c877fe640 10 monclient(hunting): get_auth_request con 0x7f1c8813a2d0 auth_method 0 2024-09-11T02:50:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c7ffff640 10 monclient(hunting): get_auth_request con 0x7f1c88139ef0 auth_method 0 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 -- 172.21.15.98:0/1401256871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c88139ef0 msgr2=0x7f1c880a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 --2- 172.21.15.98:0/1401256871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c88139ef0 0x7f1c880a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 -- 172.21.15.98:0/1401256871 >> v2:172.21.15.190:3300/0 conn(0x7f1c8813a2d0 msgr2=0x7f1c880a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 --2- 172.21.15.98:0/1401256871 >> v2:172.21.15.190:3300/0 conn(0x7f1c8813a2d0 0x7f1c880a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 monclient: found mon.noname-d 2024-09-11T02:50:12.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:12.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.075+0000 7f1c86ffd640 1 -- 172.21.15.98:0/1401256871 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c880a9620 con 0x7f1c88139b10 2024-09-11T02:50:12.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 --2- 172.21.15.98:0/1401256871 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c88072600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009870 tx=0x7f1c7801c420 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c1686f41d922044f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 1 -- 172.21.15.98:0/1401256871 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1c780b0070 con 0x7f1c88139b10 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: dump: 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.089 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 1 -- 172.21.15.98:0/1401256871 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c780aa810 con 0x7f1c88139b10 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 5 monclient: authenticate success, global_id 4407 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:12.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 1 -- 172.21.15.98:0/1401256871 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1c780a9300 con 0x7f1c88139b10 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c867fc640 10 monclient: dump: 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.091 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 4 set_mon_vals no callback set 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient: shutdown 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/1401256871 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 msgr2=0x7f1c88072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/1401256871 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c88072600 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c78009870 tx=0x7f1c7801c420 comp rx=0 tx=0).stop 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/1401256871 shutdown_connections 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/1401256871 >> v2:172.21.15.190:3300/0 conn(0x7f1c8813a2d0 0x7f1c880a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/1401256871 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1c88139ef0 0x7f1c880a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/1401256871 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c88072600 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/1401256871 >> 172.21.15.98:0/1401256871 conn(0x7f1c880ac790 msgr2=0x7f1c880a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/1401256871 shutdown_connections 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/1401256871 wait complete. 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient: monmap: 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:12.093 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.094 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 librados: starting msgr at 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 librados: starting objecter 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 Processor -- start 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- start start 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 librados: setting wanted keys 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 librados: calling monclient init 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient: init 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient: _add_conns ranks=[6,5,1,8,7,3,2,0,4] 2024-09-11T02:50:12.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c8813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): picked mon.noname-d con 0x7f1c88139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1c88139ef0 0x7f1c8813ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): picked mon.noname-h con 0x7f1c88139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c8813a2d0 0x7f1c8813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): picked mon.noname-f con 0x7f1c8813a2d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1c88143460 con 0x7f1c8813a2d0 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1c880a88a0 con 0x7f1c88139ef0 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1c881435e0 con 0x7f1c88139b10 2024-09-11T02:50:12.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c8d2b4640 10 monclient(hunting): authenticate will time out at 965.963435s 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c8813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c8813a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39624/0 (socket says 172.21.15.98:39624) 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 -- 172.21.15.98:0/4249329986 learned_addr learned my addr 172.21.15.98:0/4249329986 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): get_auth_request con 0x7f1c88139b10 auth_method 0 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c877fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c8813a2d0 0x7f1c8813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c877fe640 10 monclient(hunting): get_auth_request con 0x7f1c8813a2d0 auth_method 0 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c7ffff640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1c88139ef0 0x7f1c8813ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1c88139ef0 msgr2=0x7f1c8813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1c88139ef0 0x7f1c8813ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c8813a2d0 msgr2=0x7f1c8813b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c8813a2d0 0x7f1c8813b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 monclient: found mon.noname-d 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.079+0000 7f1c86ffd640 1 -- 172.21.15.98:0/4249329986 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8813cbb0 con 0x7f1c88139b10 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c86ffd640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c8813a7d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1c780ae040 tx=0x7f1c780191c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 client.?.objecter ms_handle_connect 0x7f1c88139b10 2024-09-11T02:50:12.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1c780b9070 con 0x7f1c88139b10 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: dump: 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.100 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c78003620 con 0x7f1c88139b10 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 5 monclient: authenticate success, global_id 4416 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 10 monclient: _renew_subs 2024-09-11T02:50:12.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c880aa3e0 con 0x7f1c88139b10 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1c780b0040 con 0x7f1c88139b10 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: dump: 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 10 monclient: _renew_subs 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c85ffb640 4 set_mon_vals no callback set 2024-09-11T02:50:12.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c880a9f00 con 0x7f1c88139b10 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 1 librados: init done 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88133760 0x7f1c88133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1c8806ca10 con 0x7f1c88133760 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88133760 0x7f1c88133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient: get_auth_request con 0x7f1c88133760 auth_method 0 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1c78004100 con 0x7f1c88139b10 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 1 -- 172.21.15.98:0/4249329986 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c8806e680 con 0x7f1c88133760 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1c6c03df70 0x7f1c6c040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f1c780e84f0 con 0x7f1c88139b10 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 client.4416.objecter ms_dispatch 0x7f1c88000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 3 client.4416.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 3 client.4416.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7ffff640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88133760 0x7f1c88133b40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c880a4c10 tx=0x7f1c7000e990 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 20 client.4416.objecter dump_active .. 0 homeless 2024-09-11T02:50:12.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 client.4416.objecter ms_handle_connect 0x7f1c88133760 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 client.4416.objecter resend_mon_ops 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1c70017070 con 0x7f1c88133760 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: dump: 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.106 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c877fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1c6c03df70 0x7f1c6c040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c877fe640 10 monclient: get_auth_request con 0x7f1c6c03df70 auth_method 0 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1c7001c450 con 0x7f1c88133760 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:12.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88133760 msgr2=0x7f1c88133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88133760 0x7f1c88133b40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c880a4c10 tx=0x7f1c7000e990 comp rx=0 tx=0).stop 2024-09-11T02:50:12.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88133760 msgr2=0x7f1c88133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88133760 0x7f1c88133b40 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1c880a4c10 tx=0x7f1c7000e990 comp rx=0 tx=0).stop 2024-09-11T02:50:12.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c7f7fe640 10 client.4416.objecter ms_handle_connect 0x7f1c6c03df70 2024-09-11T02:50:12.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.083+0000 7f1c877fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1c6c03df70 0x7f1c6c040430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009e60 tx=0x7f1c8000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88134080 0x7f1c880a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1c8806e870 con 0x7f1c88134080 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88134080 0x7f1c880a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient: get_auth_request con 0x7f1c88134080 auth_method 0 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 10 monclient(hunting): handle_auth_done global_id 4405 payload 824 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 1 -- 172.21.15.98:0/4249329986 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c88072600 con 0x7f1c88134080 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7ffff640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88134080 0x7f1c880a0e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c70008010 tx=0x7f1c70004740 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 10 client.4416.objecter ms_handle_connect 0x7f1c88134080 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 10 client.4416.objecter resend_mon_ops 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1c7000f040 con 0x7f1c88134080 2024-09-11T02:50:12.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 10 monclient: dump: 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.112 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f1c70013b80 con 0x7f1c88134080 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88134080 msgr2=0x7f1c880a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88134080 0x7f1c880a0e60 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c70008010 tx=0x7f1c70004740 comp rx=0 tx=0).stop 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88134080 msgr2=0x7f1c880a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c7f7fe640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1c88134080 0x7f1c880a0e60 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c70008010 tx=0x7f1c70004740 comp rx=0 tx=0).stop 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 27875, 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:12.114 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 27, 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:12.115 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.116 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.117 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:12.118 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.119 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:12.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:12.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.125 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:12.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:12.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:12.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.132 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:12.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:12.136 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.138 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:12.138 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 10 librados: watch_flush enter 2024-09-11T02:50:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 10 librados: watch_flush exit 2024-09-11T02:50:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 20 client.4416.objecter shutdown clearing up homeless session... 2024-09-11T02:50:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 10 client.4416.objecter successfully canceled tick 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1c6c03df70 msgr2=0x7f1c6c040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1c6c03df70 0x7f1c6c040430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009e60 tx=0x7f1c8000b040 comp rx=0 tx=0).stop 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 10 monclient: shutdown 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 msgr2=0x7f1c8813a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 --2- 172.21.15.98:0/4249329986 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1c88139b10 0x7f1c8813a7d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1c780ae040 tx=0x7f1c780191c0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c877fe640 1 -- 172.21.15.98:0/4249329986 reap_dead start 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c877fe640 1 -- 172.21.15.98:0/4249329986 reap_dead start 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 shutdown_connections 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 >> 172.21.15.98:0/4249329986 conn(0x7f1c880ac790 msgr2=0x7f1c880a6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 shutdown_connections 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 -- 172.21.15.98:0/4249329986 wait complete. 2024-09-11T02:50:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.091+0000 7f1c8d2b4640 1 librados: shutdown 2024-09-11T02:50:12.141 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:12.141 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:50:12.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:50:12.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient: get_monmap_and_config 2024-09-11T02:50:12.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient: monmap: 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:12.263320+0000 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:12.263320+0000 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:12.260 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:12.261 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 Processor -- start 2024-09-11T02:50:12.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 -- start start 2024-09-11T02:50:12.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient: init 2024-09-11T02:50:12.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient: _add_conns ranks=[2,10,8,9,6,4,5,3,1,7,0] 2024-09-11T02:50:12.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b40139b10 0x7f2b40072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): picked mon.noname-b con 0x7f2b40139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:12.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2b40139ef0 0x7f2b400a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): picked mon.noname-f con 0x7f2b40139ef0 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:12.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b4013a2d0 0x7f2b400a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): picked mon.noname-e con 0x7f2b4013a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:12.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2b400702b0 con 0x7f2b40139ef0 2024-09-11T02:50:12.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2b40071bf0 con 0x7f2b40139b10 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2b40071350 con 0x7f2b4013a2d0 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b46000640 10 monclient(hunting): authenticate will time out at 966.143637s 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b3f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b40139b10 0x7f2b40072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b3ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b4013a2d0 0x7f2b400a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b3f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b40139b10 0x7f2b40072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48212/0 (socket says 172.21.15.98:48212) 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.259+0000 7f2b3ffff640 1 --2- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b4013a2d0 0x7f2b400a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35048/0 (socket says 172.21.15.98:35048) 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 -- 172.21.15.98:0/2834325795 learned_addr learned my addr 172.21.15.98:0/2834325795 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b40139b10 auth_method 0 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 10 monclient(hunting): get_auth_request con 0x7f2b4013a2d0 auth_method 0 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3effd640 1 --2- 172.21.15.98:0/2834325795 >> v2:172.21.15.190:3300/0 conn(0x7f2b40139ef0 0x7f2b400a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3effd640 10 monclient(hunting): get_auth_request con 0x7f2b40139ef0 auth_method 0 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_done global_id 4385 payload 274 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 -- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b4013a2d0 msgr2=0x7f2b400a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 --2- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b4013a2d0 0x7f2b400a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 -- 172.21.15.98:0/2834325795 >> v2:172.21.15.190:3300/0 conn(0x7f2b40139ef0 msgr2=0x7f2b400a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 --2- 172.21.15.98:0/2834325795 >> v2:172.21.15.190:3300/0 conn(0x7f2b40139ef0 0x7f2b400a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 monclient: found mon.noname-b 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 -- 172.21.15.98:0/2834325795 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b400a9620 con 0x7f2b40139b10 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3ffff640 1 --2- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b4013a2d0 0x7f2b400a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3f7fe640 1 --2- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b40139b10 0x7f2b40072600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009e80 tx=0x7f2b3801c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48a785c2b9ff08 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 1 -- 172.21.15.98:0/2834325795 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b380090d0 con 0x7f2b40139b10 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: dump: 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 1 -- 172.21.15.98:0/2834325795 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b380a5a10 con 0x7f2b40139b10 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 1 -- 172.21.15.98:0/2834325795 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b380a5c00 con 0x7f2b40139b10 2024-09-11T02:50:12.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 5 monclient: authenticate success, global_id 4385 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 4 set_mon_vals no callback set 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b3e7fc640 10 monclient: dump: 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 10 monclient: shutdown 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 -- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b40139b10 msgr2=0x7f2b40072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 --2- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b40139b10 0x7f2b40072600 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2b38009e80 tx=0x7f2b3801c960 comp rx=0 tx=0).stop 2024-09-11T02:50:12.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 -- 172.21.15.98:0/2834325795 shutdown_connections 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 --2- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b4013a2d0 0x7f2b400a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 --2- 172.21.15.98:0/2834325795 >> v2:172.21.15.190:3300/0 conn(0x7f2b40139ef0 0x7f2b400a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 --2- 172.21.15.98:0/2834325795 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b40139b10 0x7f2b40072600 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 -- 172.21.15.98:0/2834325795 >> 172.21.15.98:0/2834325795 conn(0x7f2b400ac790 msgr2=0x7f2b400a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 -- 172.21.15.98:0/2834325795 shutdown_connections 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 -- 172.21.15.98:0/2834325795 wait complete. 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 10 monclient: monmap: 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:12.284 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 librados: starting msgr at 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.263+0000 7f2b46000640 1 librados: starting objecter 2024-09-11T02:50:12.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 Processor -- start 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 -- start start 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 librados: setting wanted keys 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 librados: calling monclient init 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient: init 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient: _add_conns ranks=[1,3,7,5,0,6,8,2,4] 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b40139b10 0x7f2b4013a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): picked mon.noname-f con 0x7f2b40139b10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b40139ef0 0x7f2b4013acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): picked mon.noname-g con 0x7f2b40139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b4013a2d0 0x7f2b4013b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): picked mon.noname-i con 0x7f2b4013a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b40143460 con 0x7f2b40139b10 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2b400a88a0 con 0x7f2b40139ef0 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f2b401435e0 con 0x7f2b4013a2d0 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): authenticate will time out at 966.148537s 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b40139b10 0x7f2b4013a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b4013a2d0 0x7f2b4013b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b40139ef0 0x7f2b4013acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b40139b10 0x7f2b4013a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:35736/0 (socket says 172.21.15.98:35736) 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 1 -- 172.21.15.98:0/2105855702 learned_addr learned my addr 172.21.15.98:0/2105855702 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): get_auth_request con 0x7f2b40139b10 auth_method 0 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 10 monclient(hunting): get_auth_request con 0x7f2b4013a2d0 auth_method 0 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): get_auth_request con 0x7f2b40139ef0 auth_method 0 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-09-11T02:50:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b4013a2d0 msgr2=0x7f2b4013b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b4013a2d0 0x7f2b4013b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b40139b10 msgr2=0x7f2b4013a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b40139b10 0x7f2b4013a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 monclient: found mon.noname-g 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 -- 172.21.15.98:0/2105855702 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4013ccc0 con 0x7f2b40139ef0 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3ffff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b4013a2d0 0x7f2b4013b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3effd640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b40139ef0 0x7f2b4013acc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d5f0 tx=0x7f2b2800d9f0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 client.?.objecter ms_handle_connect 0x7f2b40139ef0 2024-09-11T02:50:12.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b28012a60 con 0x7f2b40139ef0 2024-09-11T02:50:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:50:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: dump: 2024-09-11T02:50:12.292 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.293 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 5 monclient: authenticate success, global_id 4386 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient: _renew_subs 2024-09-11T02:50:12.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4013cfa0 con 0x7f2b40139ef0 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b28019070 con 0x7f2b40139ef0 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3dffb640 4 set_mon_vals no callback set 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient: _renew_subs 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b4013c9e0 con 0x7f2b40139ef0 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 librados: init done 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b28012c00 con 0x7f2b40139ef0 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: dump: 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40133760 0x7f2b40133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2b4006b760 con 0x7f2b40133760 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40133760 0x7f2b40133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient: get_auth_request con 0x7f2b40133760 auth_method 0 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 1 -- 172.21.15.98:0/2105855702 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b4013c740 con 0x7f2b40133760 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 client.4386.objecter ms_handle_connect 0x7f2b40133760 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 client.4386.objecter resend_mon_ops 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b3f7fe640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40133760 0x7f2b40133b40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801d200 tx=0x7f2b38019700 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b38009510 con 0x7f2b40133760 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 monclient: dump: 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.299 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2b28013070 con 0x7f2b40139ef0 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b2003e1e0 0x7f2b200406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f2b2800f040 con 0x7f2b40139ef0 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 10 client.4386.objecter ms_dispatch 0x7f2b40000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 3 client.4386.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.267+0000 7f2b37fff640 3 client.4386.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 20 client.4386.objecter dump_active .. 0 homeless 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b38004250 con 0x7f2b40133760 2024-09-11T02:50:12.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40133760 msgr2=0x7f2b40133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40133760 0x7f2b40133b40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801d200 tx=0x7f2b38019700 comp rx=0 tx=0).stop 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40133760 msgr2=0x7f2b40133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40133760 0x7f2b40133b40 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801d200 tx=0x7f2b38019700 comp rx=0 tx=0).stop 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b3ffff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b2003e1e0 0x7f2b200406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b3ffff640 10 monclient: get_auth_request con 0x7f2b2003e1e0 auth_method 0 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b37fff640 10 client.4386.objecter ms_handle_connect 0x7f2b2003e1e0 2024-09-11T02:50:12.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.271+0000 7f2b3ffff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b2003e1e0 0x7f2b200406a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b300095a0 tx=0x7f2b30009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:12.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40134080 0x7f2b40137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2b4013ba80 con 0x7f2b40134080 2024-09-11T02:50:12.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40134080 0x7f2b40137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient: get_auth_request con 0x7f2b40134080 auth_method 0 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 1 -- 172.21.15.98:0/2105855702 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b4006e680 con 0x7f2b40134080 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3f7fe640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40134080 0x7f2b40137ba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801c790 tx=0x7f2b380a7000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 10 client.4386.objecter ms_handle_connect 0x7f2b40134080 2024-09-11T02:50:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 10 client.4386.objecter resend_mon_ops 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b380cb880 con 0x7f2b40134080 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 10 monclient: dump: 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f2b380cb410 con 0x7f2b40134080 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40134080 msgr2=0x7f2b40137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40134080 0x7f2b40137ba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801c790 tx=0x7f2b380a7000 comp rx=0 tx=0).stop 2024-09-11T02:50:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40134080 msgr2=0x7f2b40137ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b37fff640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b40134080 0x7f2b40137ba0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2b3801c790 tx=0x7f2b380a7000 comp rx=0 tx=0).stop 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 28071, 2024-09-11T02:50:12.307 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 27, 2024-09-11T02:50:12.308 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.309 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.313 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.314 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:12.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:12.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.323 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:12.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:12.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.326 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.327 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.327 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.328 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:12.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:12.328 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:12.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:12.330 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.330 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.331 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.331 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.331 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.332 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:12.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:12.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:12.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:12.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.334 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.335 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.335 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.336 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.336 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:12.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:12.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:12.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:12.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.339 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:12.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:12.341 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:12.342 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.342 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:12.343 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.343 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.343 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.343 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.343 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:12.354 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:12.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 10 librados: watch_flush enter 2024-09-11T02:50:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 10 librados: watch_flush exit 2024-09-11T02:50:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 20 client.4386.objecter shutdown clearing up homeless session... 2024-09-11T02:50:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 10 client.4386.objecter successfully canceled tick 2024-09-11T02:50:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b2003e1e0 msgr2=0x7f2b200406a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b2003e1e0 0x7f2b200406a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b300095a0 tx=0x7f2b30009190 comp rx=0 tx=0).stop 2024-09-11T02:50:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 10 monclient: shutdown 2024-09-11T02:50:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b40139ef0 msgr2=0x7f2b4013acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b3ffff640 1 -- 172.21.15.98:0/2105855702 reap_dead start 2024-09-11T02:50:12.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b40139ef0 0x7f2b4013acc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800d5f0 tx=0x7f2b2800d9f0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 shutdown_connections 2024-09-11T02:50:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 --2- 172.21.15.98:0/2105855702 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2b40139ef0 0x7f2b4013acc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 >> 172.21.15.98:0/2105855702 conn(0x7f2b400ac790 msgr2=0x7f2b400a6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 shutdown_connections 2024-09-11T02:50:12.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 -- 172.21.15.98:0/2105855702 wait complete. 2024-09-11T02:50:12.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.275+0000 7f2b46000640 1 librados: shutdown 2024-09-11T02:50:12.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:12.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:50:12.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient: get_monmap_and_config 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient: monmap: 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:12.489932+0000 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:12.489932+0000 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 Processor -- start 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 -- start start 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient: init 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient: _add_conns ranks=[5,8,3,4,7,0,6,9,1,2,10] 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f53e0139b10 0x7f53e0072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): picked mon.noname-h con 0x7f53e0139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): picked mon.noname-e con 0x7f53e0139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e00a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): picked mon.noname-g con 0x7f53e013a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f53e00702b0 con 0x7f53e013a2d0 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f53e0071bf0 con 0x7f53e0139b10 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f53e0071350 con 0x7f53e0139ef0 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e63e7640 10 monclient(hunting): authenticate will time out at 966.370188s 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e00a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35060/0 (socket says 172.21.15.98:35060) 2024-09-11T02:50:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 -- 172.21.15.98:0/3215538526 learned_addr learned my addr 172.21.15.98:0/3215538526 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): get_auth_request con 0x7f53e0139ef0 auth_method 0 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53dffff640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f53e0139b10 0x7f53e0072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e4984640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e00a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53dffff640 10 monclient(hunting): get_auth_request con 0x7f53e0139b10 auth_method 0 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e4984640 10 monclient(hunting): get_auth_request con 0x7f53e013a2d0 auth_method 0 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e4984640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e4984640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53e4984640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 -- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f53e0139b10 msgr2=0x7f53e0072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f53e0139b10 0x7f53e0072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 -- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 msgr2=0x7f53e00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e00a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 monclient: found mon.noname-e 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.487+0000 7f53df7fe640 1 -- 172.21.15.98:0/3215538526 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e00a9620 con 0x7f53e0139ef0 2024-09-11T02:50:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e00a7a10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53c80098a0 tx=0x7f53c801c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=16621a74abe5dd6f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 1 -- 172.21.15.98:0/3215538526 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c80b0070 con 0x7f53e0139ef0 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: dump: 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 1 -- 172.21.15.98:0/3215538526 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c80aa810 con 0x7f53e0139ef0 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 1 -- 172.21.15.98:0/3215538526 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c80a9300 con 0x7f53e0139ef0 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 5 monclient: authenticate success, global_id 4418 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 4 set_mon_vals no callback set 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53deffd640 10 monclient: dump: 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient: shutdown 2024-09-11T02:50:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- 172.21.15.98:0/3215538526 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 msgr2=0x7f53e00a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e00a7a10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53c80098a0 tx=0x7f53c801c440 comp rx=0 tx=0).stop 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- 172.21.15.98:0/3215538526 shutdown_connections 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e00a7a10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 --2- 172.21.15.98:0/3215538526 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f53e0139b10 0x7f53e0072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- 172.21.15.98:0/3215538526 >> 172.21.15.98:0/3215538526 conn(0x7f53e00ac790 msgr2=0x7f53e00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- 172.21.15.98:0/3215538526 shutdown_connections 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- 172.21.15.98:0/3215538526 wait complete. 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient: monmap: 2024-09-11T02:50:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 librados: starting msgr at 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 librados: starting objecter 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 Processor -- start 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- start start 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 librados: setting wanted keys 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 librados: calling monclient init 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient: init 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient: _add_conns ranks=[7,8,3,0,2,5,1,4,6] 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53e0139b10 0x7f53e0134f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): picked mon.noname-i con 0x7f53e0139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e0135490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): picked mon.noname-e con 0x7f53e0139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): picked mon.noname-g con 0x7f53e013a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f53e0143460 con 0x7f53e013a2d0 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f53e00a88a0 con 0x7f53e0139b10 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f53e01435e0 con 0x7f53e0139ef0 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e63e7640 10 monclient(hunting): authenticate will time out at 966.375521s 2024-09-11T02:50:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e0135490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e0135490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35064/0 (socket says 172.21.15.98:35064) 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 -- 172.21.15.98:0/2787240749 learned_addr learned my addr 172.21.15.98:0/2787240749 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): get_auth_request con 0x7f53e0139ef0 auth_method 0 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e4984640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53dffff640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53e0139b10 0x7f53e0134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient(hunting): handle_auth_done global_id 4427 payload 824 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53e0139b10 msgr2=0x7f53e0134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53e0139b10 0x7f53e0134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 msgr2=0x7f53e013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 monclient: found mon.noname-e 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53df7fe640 1 -- 172.21.15.98:0/2787240749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e013dad0 con 0x7f53e0139ef0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53dffff640 10 monclient: get_auth_request con 0x7f53e0139b10 auth_method 0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e4984640 10 monclient: get_auth_request con 0x7f53e013a2d0 auth_method 0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53dffff640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53e0139b10 0x7f53e0134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.491+0000 7f53e4984640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f53e013a2d0 0x7f53e013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53df7fe640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e0135490 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f53c80ae040 tx=0x7f53c80191e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 client.?.objecter ms_handle_connect 0x7f53e0139ef0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c80b9070 con 0x7f53e0139ef0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: dump: 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c80044b0 con 0x7f53e0139ef0 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 5 monclient: authenticate success, global_id 4427 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 10 monclient: _renew_subs 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e00a9f00 con 0x7f53e0139ef0 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c80b0040 con 0x7f53e0139ef0 2024-09-11T02:50:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: dump: 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 10 monclient: _renew_subs 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53de7fc640 4 set_mon_vals no callback set 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e013ed80 con 0x7f53e0139ef0 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 1 librados: init done 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013cb70 0x7f53e013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f53e013e2d0 con 0x7f53e013cb70 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013cb70 0x7f53e013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient: get_auth_request con 0x7f53e013cb70 auth_method 0 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 1 -- 172.21.15.98:0/2787240749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53e006e680 con 0x7f53e013cb70 2024-09-11T02:50:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f53c8004040 con 0x7f53e0139ef0 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dffff640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013cb70 0x7f53e013ad10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e740 tx=0x7f53d800ec00 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53c003df30 0x7f53c00403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 client.4427.objecter ms_handle_connect 0x7f53e013cb70 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 client.4427.objecter resend_mon_ops 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f53c80e8820 con 0x7f53e0139ef0 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 client.4427.objecter ms_dispatch 0x7f53e0000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 3 client.4427.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 3 client.4427.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 20 client.4427.objecter dump_active .. 0 homeless 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53d8017070 con 0x7f53e013cb70 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: dump: 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e4984640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53c003df30 0x7f53c00403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f53d800ed30 con 0x7f53e013cb70 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013cb70 msgr2=0x7f53e013ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013cb70 0x7f53e013ad10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e740 tx=0x7f53d800ec00 comp rx=0 tx=0).stop 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013cb70 msgr2=0x7f53e013ad10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013cb70 0x7f53e013ad10 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e740 tx=0x7f53d800ec00 comp rx=0 tx=0).stop 2024-09-11T02:50:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e4984640 10 monclient: get_auth_request con 0x7f53c003df30 auth_method 0 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53dcff9640 10 client.4427.objecter ms_handle_connect 0x7f53c003df30 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.495+0000 7f53e4984640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53c003df30 0x7f53c00403f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53e01359d0 tx=0x7f53d000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.499+0000 7f53e63e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.499+0000 7f53e63e7640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013b250 0x7f53e013b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.499+0000 7f53e63e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.499+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f53e006ca10 con 0x7f53e013b250 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013b250 0x7f53e013b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient: get_auth_request con 0x7f53e013b250 auth_method 0 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 1 -- 172.21.15.98:0/2787240749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53e013bba0 con 0x7f53e013b250 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 10 client.4427.objecter ms_handle_connect 0x7f53e013b250 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dffff640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013b250 0x7f53e013b660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e740 tx=0x7f53d80046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 10 client.4427.objecter resend_mon_ops 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53d800f040 con 0x7f53e013b250 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 10 monclient: dump: 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f53d8003620 con 0x7f53e013b250 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013b250 msgr2=0x7f53e013b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013b250 0x7f53e013b660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e740 tx=0x7f53d80046d0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013b250 msgr2=0x7f53e013b660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53dcff9640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e013b250 0x7f53e013b660 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e740 tx=0x7f53d80046d0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.526 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 28295, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 27, 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:12.550 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 10 librados: watch_flush enter 2024-09-11T02:50:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 10 librados: watch_flush exit 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 20 client.4427.objecter shutdown clearing up homeless session... 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 10 client.4427.objecter successfully canceled tick 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53c003df30 msgr2=0x7f53c00403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53c003df30 0x7f53c00403f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53e01359d0 tx=0x7f53d000b040 comp rx=0 tx=0).stop 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 10 monclient: shutdown 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 msgr2=0x7f53e0135490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 --2- 172.21.15.98:0/2787240749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f53e0139ef0 0x7f53e0135490 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f53c80ae040 tx=0x7f53c80191e0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e4984640 1 -- 172.21.15.98:0/2787240749 reap_dead start 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e4984640 1 -- 172.21.15.98:0/2787240749 reap_dead start 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 shutdown_connections 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 >> 172.21.15.98:0/2787240749 conn(0x7f53e00ac790 msgr2=0x7f53e0135a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 shutdown_connections 2024-09-11T02:50:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 -- 172.21.15.98:0/2787240749 wait complete. 2024-09-11T02:50:12.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.503+0000 7f53e63e7640 1 librados: shutdown 2024-09-11T02:50:12.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:12.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:50:12.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:50:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient: get_monmap_and_config 2024-09-11T02:50:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient: monmap: 2024-09-11T02:50:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.666 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:12.670187+0000 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:12.670187+0000 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.667 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:12.668 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 Processor -- start 2024-09-11T02:50:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 -- start start 2024-09-11T02:50:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient: init 2024-09-11T02:50:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient: _add_conns ranks=[3,7,8,6,9,1,4,0,5,10,2] 2024-09-11T02:50:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c90139b10 0x7f0c90072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): picked mon.noname-g con 0x7f0c90139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0c90139ef0 0x7f0c900a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): picked mon.noname-i con 0x7f0c90139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0c9013a2d0 0x7f0c900a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): picked mon.noname-e con 0x7f0c9013a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0c900702b0 con 0x7f0c90139b10 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0c90071bf0 con 0x7f0c90139ef0 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0c90071350 con 0x7f0c9013a2d0 2024-09-11T02:50:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c949fa640 10 monclient(hunting): authenticate will time out at 966.550478s 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0c9013a2d0 0x7f0c900a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0c9013a2d0 0x7f0c900a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35096/0 (socket says 172.21.15.98:35096) 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/517873604 learned_addr learned my addr 172.21.15.98:0/517873604 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): get_auth_request con 0x7f0c9013a2d0 auth_method 0 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8dd9b640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0c90139ef0 0x7f0c900a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8dd9b640 10 monclient(hunting): get_auth_request con 0x7f0c90139ef0 auth_method 0 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_done global_id 4454 payload 274 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8e59c640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c90139b10 0x7f0c90072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0c90139ef0 msgr2=0x7f0c900a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0c90139ef0 0x7f0c900a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c90139b10 msgr2=0x7f0c90072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c90139b10 0x7f0c90072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 monclient: found mon.noname-e 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.667+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/517873604 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c900a9620 con 0x7f0c9013a2d0 2024-09-11T02:50:12.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8ed9d640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0c9013a2d0 0x7f0c900a7f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c800098a0 tx=0x7f0c8001c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a0832cac05c99f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 1 -- 172.21.15.98:0/517873604 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0c800b0070 con 0x7f0c9013a2d0 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: dump: 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 5 monclient: authenticate success, global_id 4454 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 1 -- 172.21.15.98:0/517873604 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c800aa810 con 0x7f0c9013a2d0 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 1 -- 172.21.15.98:0/517873604 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0c800a9300 con 0x7f0c9013a2d0 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 4 set_mon_vals no callback set 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c8d59a640 10 monclient: dump: 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient: shutdown 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 -- 172.21.15.98:0/517873604 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0c9013a2d0 msgr2=0x7f0c900a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0c9013a2d0 0x7f0c900a7f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c800098a0 tx=0x7f0c8001c440 comp rx=0 tx=0).stop 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 -- 172.21.15.98:0/517873604 shutdown_connections 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0c9013a2d0 0x7f0c900a7f50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0c90139ef0 0x7f0c900a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 --2- 172.21.15.98:0/517873604 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0c90139b10 0x7f0c90072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 -- 172.21.15.98:0/517873604 >> 172.21.15.98:0/517873604 conn(0x7f0c900ac790 msgr2=0x7f0c900a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 -- 172.21.15.98:0/517873604 shutdown_connections 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 -- 172.21.15.98:0/517873604 wait complete. 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient: monmap: 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 librados: starting msgr at 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 librados: starting objecter 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 Processor -- start 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 -- start start 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 librados: setting wanted keys 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 librados: calling monclient init 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient: init 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient: _add_conns ranks=[0,2,6,5,3,8,1,4,7] 2024-09-11T02:50:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c90139b10 0x7f0c9013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.671+0000 7f0c949fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0c90139b10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c90139ef0 0x7f0c9013abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0c90139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0c9013a2d0 0x7f0c9013b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): picked mon.noname-d con 0x7f0c9013a2d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0c90143460 con 0x7f0c90139b10 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0c900a88a0 con 0x7f0c90139ef0 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0c901435e0 con 0x7f0c9013a2d0 2024-09-11T02:50:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): authenticate will time out at 966.556096s 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c90139b10 0x7f0c9013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c90139ef0 0x7f0c9013abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c90139b10 0x7f0c9013a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37092/0 (socket says 172.21.15.98:37092) 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 1 -- 172.21.15.98:0/4022432545 learned_addr learned my addr 172.21.15.98:0/4022432545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request con 0x7f0c90139b10 auth_method 0 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0c9013a2d0 0x7f0c9013b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient(hunting): get_auth_request con 0x7f0c90139ef0 auth_method 0 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): get_auth_request con 0x7f0c9013a2d0 auth_method 0 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c90139ef0 msgr2=0x7f0c9013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c90139ef0 0x7f0c9013abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c90139b10 msgr2=0x7f0c9013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c90139b10 0x7f0c9013a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 monclient: found mon.noname-d 2024-09-11T02:50:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/4022432545 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9013d290 con 0x7f0c9013a2d0 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c90139b10 0x7f0c9013a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0c90139ef0 0x7f0c9013abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8ed9d640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0c9013a2d0 0x7f0c9013b130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c900714d0 tx=0x7f0c800191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 client.?.objecter ms_handle_connect 0x7f0c9013a2d0 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0c800b0040 con 0x7f0c9013a2d0 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: dump: 2024-09-11T02:50:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c800044b0 con 0x7f0c9013a2d0 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0c800aad30 con 0x7f0c9013a2d0 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 5 monclient: authenticate success, global_id 4443 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8cd99640 4 set_mon_vals no callback set 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: dump: 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient: _renew_subs 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c90135a10 con 0x7f0c9013a2d0 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient: _renew_subs 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c900aa510 con 0x7f0c9013a2d0 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 librados: init done 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063640 0x7f0c90063a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9013cc30 con 0x7f0c90063640 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0c800b3070 con 0x7f0c9013a2d0 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0c7003df30 0x7f0c700403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f0c800e8dd0 con 0x7f0c9013a2d0 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 client.4443.objecter ms_dispatch 0x7f0c90000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 3 client.4443.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 3 client.4443.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063640 0x7f0c90063a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 20 client.4443.objecter dump_active .. 0 homeless 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient: get_auth_request con 0x7f0c90063640 auth_method 0 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0c7003df30 0x7f0c700403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 10 monclient: get_auth_request con 0x7f0c7003df30 auth_method 0 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 1 -- 172.21.15.98:0/4022432545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c9006e680 con 0x7f0c90063640 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 client.4443.objecter ms_handle_connect 0x7f0c90063640 2024-09-11T02:50:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8e59c640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063640 0x7f0c90063a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800f400 tx=0x7f0c8800f8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 client.4443.objecter resend_mon_ops 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 client.4443.objecter ms_handle_connect 0x7f0c7003df30 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c8dd9b640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0c7003df30 0x7f0c700403f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0c900a5680 tx=0x7f0c78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0c8801c070 con 0x7f0c90063640 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.675+0000 7f0c86ffd640 10 monclient: dump: 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.679+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c8800c700 con 0x7f0c90063640 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.679+0000 7f0c86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.679+0000 7f0c86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.679+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063640 msgr2=0x7f0c90063a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.679+0000 7f0c86ffd640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063640 0x7f0c90063a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800f400 tx=0x7f0c8800f8c0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.679+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063640 msgr2=0x7f0c90063a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.679+0000 7f0c86ffd640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063640 0x7f0c90063a20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c949fa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c949fa640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063f60 0x7f0c900a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c949fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9006ca10 con 0x7f0c90063f60 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063f60 0x7f0c900a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient: get_auth_request con 0x7f0c90063f60 auth_method 0 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_done global_id 4438 payload 824 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 1 -- 172.21.15.98:0/4022432545 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c90072600 con 0x7f0c90063f60 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c86ffd640 10 client.4443.objecter ms_handle_connect 0x7f0c90063f60 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c8e59c640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063f60 0x7f0c900a0e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800f400 tx=0x7f0c88004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.683+0000 7f0c86ffd640 10 client.4443.objecter resend_mon_ops 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0c8801c040 con 0x7f0c90063f60 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 10 monclient: dump: 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f0c88018aa0 con 0x7f0c90063f60 2024-09-11T02:50:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063f60 msgr2=0x7f0c900a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063f60 0x7f0c900a0e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800f400 tx=0x7f0c88004a50 comp rx=0 tx=0).stop 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063f60 msgr2=0x7f0c900a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c86ffd640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c90063f60 0x7f0c900a0e60 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800f400 tx=0x7f0c88004a50 comp rx=0 tx=0).stop 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 28455, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 28, 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.728 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:12.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.730 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:12.731 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 10 librados: watch_flush enter 2024-09-11T02:50:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 10 librados: watch_flush exit 2024-09-11T02:50:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 20 client.4443.objecter shutdown clearing up homeless session... 2024-09-11T02:50:12.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 10 client.4443.objecter successfully canceled tick 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0c7003df30 msgr2=0x7f0c700403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0c7003df30 0x7f0c700403f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0c900a5680 tx=0x7f0c78008040 comp rx=0 tx=0).stop 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 10 monclient: shutdown 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0c9013a2d0 msgr2=0x7f0c9013b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c8ed9d640 1 -- 172.21.15.98:0/4022432545 reap_dead start 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0c9013a2d0 0x7f0c9013b130 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0c900714d0 tx=0x7f0c800191e0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 shutdown_connections 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 --2- 172.21.15.98:0/4022432545 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0c9013a2d0 0x7f0c9013b130 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 >> 172.21.15.98:0/4022432545 conn(0x7f0c900ac790 msgr2=0x7f0c900a7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 shutdown_connections 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 -- 172.21.15.98:0/4022432545 wait complete. 2024-09-11T02:50:12.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.687+0000 7f0c949fa640 1 librados: shutdown 2024-09-11T02:50:12.734 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:12.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:50:12.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient: get_monmap_and_config 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient: monmap: 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:12.857727+0000 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:12.857727+0000 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:12.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 1 Processor -- start 2024-09-11T02:50:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 1 -- start start 2024-09-11T02:50:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient: init 2024-09-11T02:50:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient: _add_conns ranks=[5,2,0,10,1,3,6,4,9,7,8] 2024-09-11T02:50:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 0x7f9874072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient(hunting): picked mon.noname-h con 0x7f9874139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874139ef0 0x7f98740a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient(hunting): picked mon.noname-b con 0x7f9874139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f987413a2d0 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98740702b0 con 0x7f987413a2d0 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9874071bf0 con 0x7f9874139ef0 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9874071350 con 0x7f9874139b10 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f987b5d5640 10 monclient(hunting): authenticate will time out at 966.738288s 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874139ef0 0x7f98740a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9879b72640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f987413a2d0 0x7f98740a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50946/0 (socket says 172.21.15.98:50946) 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9879b72640 1 -- 172.21.15.98:0/2031511252 learned_addr learned my addr 172.21.15.98:0/2031511252 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): get_auth_request con 0x7f9874139ef0 auth_method 0 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9879371640 1 --2- 172.21.15.98:0/2031511252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 0x7f9874072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9879371640 10 monclient(hunting): get_auth_request con 0x7f9874139b10 auth_method 0 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9879371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9879371640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9879371640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient(hunting): handle_auth_done global_id 4403 payload 274 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 1 -- 172.21.15.98:0/2031511252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 msgr2=0x7f9874072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f986bfff640 1 -- 172.21.15.98:0/2031511252 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028572079 0 0) 0x7f986c0014a0 con 0x7f987413a2d0 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 1 --2- 172.21.15.98:0/2031511252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 0x7f9874072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 1 -- 172.21.15.98:0/2031511252 >> v1:172.21.15.98:6789/0 conn(0x7f987413a2d0 legacy=0x7f98740a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 1 monclient: found mon.noname-b 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f9878b70640 1 -- 172.21.15.98:0/2031511252 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98740a9620 con 0x7f9874139ef0 2024-09-11T02:50:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.855+0000 7f986bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:50:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 --2- 172.21.15.98:0/2031511252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874139ef0 0x7f98740a7a10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f98640097f0 tx=0x7f986401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1b9c74b90b29587 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 1 -- 172.21.15.98:0/2031511252 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98640b0070 con 0x7f9874139ef0 2024-09-11T02:50:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: dump: 2024-09-11T02:50:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 5 monclient: authenticate success, global_id 4403 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 1 -- 172.21.15.98:0/2031511252 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98640aa810 con 0x7f9874139ef0 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 4 set_mon_vals no callback set 2024-09-11T02:50:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 1 -- 172.21.15.98:0/2031511252 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98640aae30 con 0x7f9874139ef0 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f986bfff640 10 monclient: dump: 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient: shutdown 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- 172.21.15.98:0/2031511252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874139ef0 msgr2=0x7f98740a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 --2- 172.21.15.98:0/2031511252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874139ef0 0x7f98740a7a10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f98640097f0 tx=0x7f986401c3a0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- 172.21.15.98:0/2031511252 shutdown_connections 2024-09-11T02:50:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 --2- 172.21.15.98:0/2031511252 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9874139ef0 0x7f98740a7a10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 --2- 172.21.15.98:0/2031511252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 0x7f9874072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- 172.21.15.98:0/2031511252 >> 172.21.15.98:0/2031511252 conn(0x7f98740ac790 msgr2=0x7f98740a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- 172.21.15.98:0/2031511252 shutdown_connections 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- 172.21.15.98:0/2031511252 wait complete. 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient: build_initial_monmap 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient: monmap: 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 librados: starting msgr at 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 librados: starting objecter 2024-09-11T02:50:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 Processor -- start 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- start start 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 librados: setting wanted keys 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 librados: calling monclient init 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient: init 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient: _add_conns ranks=[5,8,7,2,3,6,1,0,4] 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 0x7f987413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): picked mon.noname-h con 0x7f9874139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9874139ef0 0x7f987413ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): picked mon.noname-e con 0x7f9874139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f987413a2d0 0x7f987413b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): picked mon.noname-i con 0x7f987413a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9874144470 con 0x7f9874139b10 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f98740a8870 con 0x7f987413a2d0 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f98741445f0 con 0x7f9874139ef0 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f987b5d5640 10 monclient(hunting): authenticate will time out at 966.743270s 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9874139ef0 0x7f987413ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9874139ef0 0x7f987413ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35106/0 (socket says 172.21.15.98:35106) 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 -- 172.21.15.98:0/3958841823 learned_addr learned my addr 172.21.15.98:0/3958841823 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): get_auth_request con 0x7f9874139ef0 auth_method 0 2024-09-11T02:50:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879371640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 0x7f987413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879b72640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f987413a2d0 0x7f987413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879371640 10 monclient(hunting): get_auth_request con 0x7f9874139b10 auth_method 0 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879371640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879371640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient(hunting): handle_auth_done global_id 4463 payload 824 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f987413a2d0 msgr2=0x7f987413b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f987413a2d0 0x7f987413b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 msgr2=0x7f987413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9874139b10 0x7f987413a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient: _finish_hunting 0 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 monclient: found mon.noname-e 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 -- 172.21.15.98:0/3958841823 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987413d1e0 con 0x7f9874139ef0 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879b72640 10 monclient: get_auth_request con 0x7f987413a2d0 auth_method 0 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9879b72640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f987413a2d0 0x7f987413b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9878b70640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9874139ef0 0x7f987413ad10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9864009710 tx=0x7f9864019140 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9869ffb640 10 client.?.objecter ms_handle_connect 0x7f9874139ef0 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.859+0000 7f9869ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98640b0040 con 0x7f9874139ef0 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: dump: 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98640044b0 con 0x7f9874139ef0 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 5 monclient: authenticate success, global_id 4463 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 10 monclient: _renew_subs 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98740abc20 con 0x7f9874139ef0 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98640aae60 con 0x7f9874139ef0 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: dump: 2024-09-11T02:50:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f986b7fe640 4 set_mon_vals no callback set 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 10 monclient: _renew_subs 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f987413cf00 con 0x7f9874139ef0 2024-09-11T02:50:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 1 librados: init done 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9874133880 0x7f9874133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9874074a50 con 0x7f9874133880 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f98640b3070 con 0x7f9874139ef0 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9874133880 0x7f9874133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient: get_auth_request con 0x7f9874133880 auth_method 0 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985803df30 0x7f98580403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f98640e8920 con 0x7f9874139ef0 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 client.4463.objecter ms_dispatch 0x7f9874000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 3 client.4463.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 3 client.4463.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 20 client.4463.objecter dump_active .. 0 homeless 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879b72640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985803df30 0x7f98580403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879b72640 10 monclient: get_auth_request con 0x7f985803df30 auth_method 0 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 10 monclient(hunting): handle_auth_done global_id 4404 payload 824 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 1 -- 172.21.15.98:0/3958841823 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f987406e680 con 0x7f9874133880 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 client.4463.objecter ms_handle_connect 0x7f9874133880 2024-09-11T02:50:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879371640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9874133880 0x7f9874133c60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f987000f4b0 tx=0x7f987000f970 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 client.4463.objecter resend_mon_ops 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9870019020 con 0x7f9874133880 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: dump: 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 client.4463.objecter ms_handle_connect 0x7f985803df30 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9879b72640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985803df30 0x7f98580403f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f986c007fd0 tx=0x7f986c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9870014670 con 0x7f9874133880 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9874133880 msgr2=0x7f9874133c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9874133880 0x7f9874133c60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f987000f4b0 tx=0x7f987000f970 comp rx=0 tx=0).stop 2024-09-11T02:50:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9874133880 msgr2=0x7f9874133c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.863+0000 7f9869ffb640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9874133880 0x7f9874133c60 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f987000f4b0 tx=0x7f987000f970 comp rx=0 tx=0).stop 2024-09-11T02:50:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f987b5d5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f987b5d5640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98741341a0 0x7f98740a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f987b5d5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f987413c5d0 con 0x7f98741341a0 2024-09-11T02:50:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98741341a0 0x7f98740a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient: get_auth_request con 0x7f98741341a0 auth_method 0 2024-09-11T02:50:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 1 -- 172.21.15.98:0/3958841823 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f98740aa2a0 con 0x7f98741341a0 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9869ffb640 10 client.4463.objecter ms_handle_connect 0x7f98741341a0 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9869ffb640 10 client.4463.objecter resend_mon_ops 2024-09-11T02:50:12.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9879371640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98741341a0 0x7f98740a0e60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f987001a000 tx=0x7f98700046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9870019040 con 0x7f98741341a0 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9869ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9869ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.871+0000 7f9869ffb640 10 monclient: dump: 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:12.892 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f987001e420 con 0x7f98741341a0 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9869ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9869ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98741341a0 msgr2=0x7f98740a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9869ffb640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98741341a0 0x7f98740a0e60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f987001a000 tx=0x7f98700046d0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9869ffb640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98741341a0 msgr2=0x7f98740a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9869ffb640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98741341a0 0x7f98740a0e60 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f987001a000 tx=0x7f98700046d0 comp rx=0 tx=0).stop 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 28623, 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 28, 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:12.909 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:12.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:12.916 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.917 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:12.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:12.919 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:12.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:12.919 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:12.919 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 10 librados: watch_flush enter 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 10 librados: watch_flush exit 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 20 client.4463.objecter shutdown clearing up homeless session... 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 10 client.4463.objecter successfully canceled tick 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985803df30 msgr2=0x7f98580403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985803df30 0x7f98580403f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f986c007fd0 tx=0x7f986c009040 comp rx=0 tx=0).stop 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 10 monclient: shutdown 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9874139ef0 msgr2=0x7f987413ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 --2- 172.21.15.98:0/3958841823 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9874139ef0 0x7f987413ad10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9864009710 tx=0x7f9864019140 comp rx=0 tx=0).stop 2024-09-11T02:50:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9879b72640 1 -- 172.21.15.98:0/3958841823 reap_dead start 2024-09-11T02:50:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f9879b72640 1 -- 172.21.15.98:0/3958841823 reap_dead start 2024-09-11T02:50:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 shutdown_connections 2024-09-11T02:50:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 >> 172.21.15.98:0/3958841823 conn(0x7f98740ac790 msgr2=0x7f98740a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 shutdown_connections 2024-09-11T02:50:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 -- 172.21.15.98:0/3958841823 wait complete. 2024-09-11T02:50:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:12.875+0000 7f987b5d5640 1 librados: shutdown 2024-09-11T02:50:12.922 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:12.922 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:50:12.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:50:13.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.039+0000 7efddd041640 10 monclient: get_monmap_and_config 2024-09-11T02:50:13.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.039+0000 7efddd041640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient: monmap: 2024-09-11T02:50:13.041 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:13.045262+0000 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:13.045262+0000 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:13.043 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 1 Processor -- start 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 1 -- start start 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient: init 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient: _add_conns ranks=[0,4,7,6,5,9,1,2,10,8,3] 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efdd8139b10 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd813ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient(hunting): picked mon.noname-c con 0x7efdd8072600 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd813ef60 0x7efdd8141350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient(hunting): picked mon.noname-i con 0x7efdd813ef60 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdd813a600 con 0x7efdd8139b10 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7efdd80702b0 con 0x7efdd8072600 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7efdd8071bf0 con 0x7efdd813ef60 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efddd041640 10 monclient(hunting): authenticate will time out at 966.925519s 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd6d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efdd8139b10 0x7efdd8139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50950/0 (socket says 172.21.15.98:50950) 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd6d9d640 1 -- 172.21.15.98:0/1828169361 learned_addr learned my addr 172.21.15.98:0/1828169361 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 1 --2- 172.21.15.98:0/1828169361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd813ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): get_auth_request con 0x7efdd8072600 auth_method 0 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd759e640 1 --2- 172.21.15.98:0/1828169361 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd813ef60 0x7efdd8141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd759e640 10 monclient(hunting): get_auth_request con 0x7efdd813ef60 auth_method 0 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd5d9b640 1 -- 172.21.15.98:0/1828169361 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2140946451 0 0) 0x7efdd813a600 con 0x7efdd8139b10 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd5d9b640 10 monclient(hunting): _negotiate 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd5d9b640 10 monclient(hunting): my global_id is 4545 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd5d9b640 1 -- 172.21.15.98:0/1828169361 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdc0003530 con 0x7efdd8139b10 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient(hunting): handle_auth_done global_id 4414 payload 274 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 1 -- 172.21.15.98:0/1828169361 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd813ef60 msgr2=0x7efdd8141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 1 --2- 172.21.15.98:0/1828169361 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd813ef60 0x7efdd8141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 1 -- 172.21.15.98:0/1828169361 >> v1:172.21.15.98:6789/0 conn(0x7efdd8139b10 legacy=0x7efdd8139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 1 monclient: found mon.noname-c 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.043+0000 7efdd659c640 1 -- 172.21.15.98:0/1828169361 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd8071350 con 0x7efdd8072600 2024-09-11T02:50:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd659c640 1 --2- 172.21.15.98:0/1828169361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd813ea20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0097f0 tx=0x7efdcc01c370 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b16ca65562889866 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 1 -- 172.21.15.98:0/1828169361 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdcc0b0070 con 0x7efdd8072600 2024-09-11T02:50:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: dump: 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 1 -- 172.21.15.98:0/1828169361 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdcc0044a0 con 0x7efdd8072600 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 5 monclient: authenticate success, global_id 4414 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 1 -- 172.21.15.98:0/1828169361 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdcc003a90 con 0x7efdd8072600 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 4 set_mon_vals no callback set 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd5d9b640 10 monclient: dump: 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: shutdown 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- 172.21.15.98:0/1828169361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 msgr2=0x7efdd813ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 --2- 172.21.15.98:0/1828169361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd813ea20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0097f0 tx=0x7efdcc01c370 comp rx=0 tx=0).stop 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- 172.21.15.98:0/1828169361 shutdown_connections 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 --2- 172.21.15.98:0/1828169361 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd813ef60 0x7efdd8141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 --2- 172.21.15.98:0/1828169361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd813ea20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- 172.21.15.98:0/1828169361 >> 172.21.15.98:0/1828169361 conn(0x7efdd80ac790 msgr2=0x7efdd80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- 172.21.15.98:0/1828169361 shutdown_connections 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- 172.21.15.98:0/1828169361 wait complete. 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: monmap: 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:13.061 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 librados: starting msgr at 2024-09-11T02:50:13.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 librados: starting objecter 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 Processor -- start 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- start start 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 librados: setting wanted keys 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 librados: calling monclient init 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: init 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: _add_conns ranks=[4,7,2,1,0,8,3,6,5] 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd81d1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): picked mon.noname-c con 0x7efdd8072600 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd8139b10 0x7efdd81d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): picked mon.noname-i con 0x7efdd8139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd813ef60 0x7efdd80a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): picked mon.noname-b con 0x7efdd813ef60 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efdd8144230 con 0x7efdd813ef60 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7efdd81440b0 con 0x7efdd8072600 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7efdd81443b0 con 0x7efdd8139b10 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient(hunting): authenticate will time out at 966.930572s 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd81d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd813ef60 0x7efdd80a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd81d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39220/0 (socket says 172.21.15.98:39220) 2024-09-11T02:50:13.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 1 -- 172.21.15.98:0/1147362649 learned_addr learned my addr 172.21.15.98:0/1147362649 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 10 monclient(hunting): get_auth_request con 0x7efdd8072600 auth_method 0 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): get_auth_request con 0x7efdd813ef60 auth_method 0 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd659c640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd8139b10 0x7efdd81d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd659c640 10 monclient(hunting): get_auth_request con 0x7efdd8139b10 auth_method 0 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd8139b10 msgr2=0x7efdd81d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efdd8139b10 0x7efdd81d1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 msgr2=0x7efdd81d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd81d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 monclient: found mon.noname-b 2024-09-11T02:50:13.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 -- 172.21.15.98:0/1147362649 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdd80a5450 con 0x7efdd813ef60 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd6d9d640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efdd8072600 0x7efdd81d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd759e640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd813ef60 0x7efdd80a2d60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007cc0 tx=0x7efdd000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 client.?.objecter ms_handle_connect 0x7efdd813ef60 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdd000edd0 con 0x7efdd813ef60 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: dump: 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.067 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd0014890 con 0x7efdd813ef60 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdd0013380 con 0x7efdd813ef60 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdbf7fe640 10 monclient: dump: 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efdd559a640 4 set_mon_vals no callback set 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 5 monclient: authenticate success, global_id 4412 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: _renew_subs 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.047+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdd80aa3e0 con 0x7efdd813ef60 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 10 monclient: _renew_subs 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efdd8122f90 con 0x7efdd813ef60 2024-09-11T02:50:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 1 librados: init done 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a32a0 0x7efdd80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7efdd80a5180 con 0x7efdd80a32a0 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7efdd0014420 con 0x7efdd813ef60 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efdc403df40 0x7efdc4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7efdd0019020 con 0x7efdd813ef60 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 client.4412.objecter ms_dispatch 0x7efdd8000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 3 client.4412.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 3 client.4412.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a32a0 0x7efdd80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 20 client.4412.objecter dump_active .. 0 homeless 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient: get_auth_request con 0x7efdd80a32a0 auth_method 0 2024-09-11T02:50:13.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd659c640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efdc403df40 0x7efdc4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd659c640 10 monclient: get_auth_request con 0x7efdc403df40 auth_method 0 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 1 -- 172.21.15.98:0/1147362649 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdd806e680 con 0x7efdd80a32a0 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 client.4412.objecter ms_handle_connect 0x7efdc403df40 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd659c640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efdc403df40 0x7efdc4040400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0ad000 tx=0x7efdcc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 client.4412.objecter ms_handle_connect 0x7efdd80a32a0 2024-09-11T02:50:13.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 client.4412.objecter resend_mon_ops 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdd6d9d640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a32a0 0x7efdd80a3680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005890 tx=0x7efdc800d150 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdc800d880 con 0x7efdd80a32a0 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 monclient: dump: 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.074 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efdc8018070 con 0x7efdd80a32a0 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a32a0 msgr2=0x7efdd80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a32a0 0x7efdd80a3680 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005890 tx=0x7efdc800d150 comp rx=0 tx=0).stop 2024-09-11T02:50:13.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a32a0 msgr2=0x7efdd80a3680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:13.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.051+0000 7efdbf7fe640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a32a0 0x7efdd80a3680 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005890 tx=0x7efdc800d150 comp rx=0 tx=0).stop 2024-09-11T02:50:13.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efddd041640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:13.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efddd041640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a3bc0 0x7efdd813c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efddd041640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7efdd806ca10 con 0x7efdd80a3bc0 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a3bc0 0x7efdd813c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient: get_auth_request con 0x7efdd80a3bc0 auth_method 0 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 1 -- 172.21.15.98:0/1147362649 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efdd80a9f00 con 0x7efdd80a3bc0 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdd6d9d640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a3bc0 0x7efdd813c110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005890 tx=0x7efdc8005800 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdbf7fe640 10 client.4412.objecter ms_handle_connect 0x7efdd80a3bc0 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdbf7fe640 10 client.4412.objecter resend_mon_ops 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdc801d7c0 con 0x7efdd80a3bc0 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdbf7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.059+0000 7efdbf7fe640 10 monclient: dump: 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7efdc80073a0 con 0x7efdd80a3bc0 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdbf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdbf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a3bc0 msgr2=0x7efdd813c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdbf7fe640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a3bc0 0x7efdd813c110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005890 tx=0x7efdc8005800 comp rx=0 tx=0).stop 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdbf7fe640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a3bc0 msgr2=0x7efdd813c110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdbf7fe640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efdd80a3bc0 0x7efdd813c110 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7efdc8005890 tx=0x7efdc8005800 comp rx=0 tx=0).stop 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 28811, 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 28, 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:13.083 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:13.084 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:13.085 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:13.086 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.088 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:13.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.092 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:13.097 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:13.102 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.103 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.104 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.105 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:13.105 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 10 librados: watch_flush enter 2024-09-11T02:50:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 10 librados: watch_flush exit 2024-09-11T02:50:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 20 client.4412.objecter shutdown clearing up homeless session... 2024-09-11T02:50:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 10 client.4412.objecter successfully canceled tick 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efdc403df40 msgr2=0x7efdc4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efdc403df40 0x7efdc4040400 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0ad000 tx=0x7efdcc0a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 10 monclient: shutdown 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd813ef60 msgr2=0x7efdd80a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efdd759e640 1 -- 172.21.15.98:0/1147362649 reap_dead start 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd813ef60 0x7efdd80a2d60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007cc0 tx=0x7efdd000e4b0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 shutdown_connections 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 --2- 172.21.15.98:0/1147362649 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efdd813ef60 0x7efdd80a2d60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 >> 172.21.15.98:0/1147362649 conn(0x7efdd80ac790 msgr2=0x7efdd80a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 shutdown_connections 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 -- 172.21.15.98:0/1147362649 wait complete. 2024-09-11T02:50:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.063+0000 7efddd041640 1 librados: shutdown 2024-09-11T02:50:13.108 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:13.108 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:50:13.108 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:50:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient: get_monmap_and_config 2024-09-11T02:50:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient: monmap: 2024-09-11T02:50:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:13.226225+0000 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:13.226225+0000 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:13.224 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 Processor -- start 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 -- start start 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient: init 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient: _add_conns ranks=[9,6,8,3,10,7,0,2,1,5,4] 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab0c0a2be0 0x7fab0c0a20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): picked mon.noname-a con 0x7fab0c0a2be0 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab0c0a46a0 0x7fab0c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): picked mon.noname-d con 0x7fab0c0a46a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab0c0a4fc0 0x7fab0c140890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): picked mon.noname-e con 0x7fab0c0a4fc0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fab0c0a1760 con 0x7fab0c0a2be0 2024-09-11T02:50:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fab0c0a2620 con 0x7fab0c0a46a0 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fab0c077740 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab10edd640 10 monclient(hunting): authenticate will time out at 967.106541s 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab0c0a2be0 0x7fab0c0a20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab0c0a46a0 0x7fab0c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fab0c0a2be0 0x7fab0c0a20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37094/0 (socket says 172.21.15.98:37094) 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 1 -- 172.21.15.98:0/3205284263 learned_addr learned my addr 172.21.15.98:0/3205284263 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 10 monclient(hunting): get_auth_request con 0x7fab0c0a2be0 auth_method 0 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 1 --2- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab0c0a4fc0 0x7fab0c140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 10 monclient(hunting): get_auth_request con 0x7fab0c0a46a0 auth_method 0 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): get_auth_request con 0x7fab0c0a4fc0 auth_method 0 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_done global_id 4472 payload 274 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 1 -- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab0c0a46a0 msgr2=0x7fab0c0a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 1 --2- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab0c0a46a0 0x7fab0c0a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 1 -- 172.21.15.98:0/3205284263 >> v2:172.21.15.98:3300/0 conn(0x7fab0c0a2be0 msgr2=0x7fab0c0a20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 1 --2- 172.21.15.98:0/3205284263 >> v2:172.21.15.98:3300/0 conn(0x7fab0c0a2be0 0x7fab0c0a20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 1 monclient: found mon.noname-e 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0ad9d640 1 -- 172.21.15.98:0/3205284263 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c077040 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab0a59c640 1 --2- 172.21.15.98:0/3205284263 >> v2:172.21.15.98:3300/0 conn(0x7fab0c0a2be0 0x7fab0c0a20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.223+0000 7fab09d9b640 1 --2- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab0c0a46a0 0x7fab0c0a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0ad9d640 1 --2- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab0c0a4fc0 0x7fab0c140890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fab04009d90 tx=0x7fab0401c800 comp rx=0 tx=0).ready entity=mon.8 client_cookie=59d94521a72fc497 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 1 -- 172.21.15.98:0/3205284263 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab040b2070 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: dump: 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 1 -- 172.21.15.98:0/3205284263 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab040a5ab0 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 5 monclient: authenticate success, global_id 4472 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 1 -- 172.21.15.98:0/3205284263 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab040a5ce0 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab0959a640 10 monclient: dump: 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 4 set_mon_vals no callback set 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 10 monclient: shutdown 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 -- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab0c0a4fc0 msgr2=0x7fab0c140890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 --2- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab0c0a4fc0 0x7fab0c140890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fab04009d90 tx=0x7fab0401c800 comp rx=0 tx=0).stop 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 -- 172.21.15.98:0/3205284263 shutdown_connections 2024-09-11T02:50:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 --2- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fab0c0a4fc0 0x7fab0c140890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 --2- 172.21.15.98:0/3205284263 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fab0c0a46a0 0x7fab0c0a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 --2- 172.21.15.98:0/3205284263 >> v2:172.21.15.98:3300/0 conn(0x7fab0c0a2be0 0x7fab0c0a20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 -- 172.21.15.98:0/3205284263 >> 172.21.15.98:0/3205284263 conn(0x7fab0c06eed0 msgr2=0x7fab0c06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 -- 172.21.15.98:0/3205284263 shutdown_connections 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 -- 172.21.15.98:0/3205284263 wait complete. 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 10 monclient: monmap: 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 librados: starting msgr at 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 librados: starting objecter 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 Processor -- start 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 -- start start 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 librados: setting wanted keys 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 1 librados: calling monclient init 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.227+0000 7fab10edd640 10 monclient: init 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient: _add_conns ranks=[2,5,1,7,6,3,8,0,4] 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab0c0a2be0 0x7fab0c13d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): picked mon.noname-b con 0x7fab0c0a2be0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fab0c0a46a0 0x7fab0c13d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): picked mon.noname-h con 0x7fab0c0a46a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab0c0a4fc0 0x7fab0c125350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): picked mon.noname-f con 0x7fab0c0a4fc0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c0a2fc0 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fab0c070180 con 0x7fab0c0a2be0 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fab0c0a3140 con 0x7fab0c0a46a0 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab10edd640 10 monclient(hunting): authenticate will time out at 967.112586s 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab0c0a4fc0 0x7fab0c125350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab0c0a4fc0 0x7fab0c125350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:35752/0 (socket says 172.21.15.98:35752) 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 -- 172.21.15.98:0/407678708 learned_addr learned my addr 172.21.15.98:0/407678708 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): get_auth_request con 0x7fab0c0a4fc0 auth_method 0 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient(hunting): handle_auth_done global_id 4447 payload 824 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fab0c0a46a0 msgr2=0x7fab0c13d900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:13.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fab0c0a46a0 0x7fab0c13d900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab0c0a2be0 msgr2=0x7fab0c13d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fab0c0a2be0 0x7fab0c13d3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 monclient: found mon.noname-f 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 -- 172.21.15.98:0/407678708 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c09ebd0 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7fab0ad9d640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab0c0a4fc0 0x7fab0c125350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab040b0040 tx=0x7fab040195a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7faafaffd640 10 client.?.objecter ms_handle_connect 0x7fab0c0a4fc0 2024-09-11T02:50:13.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.231+0000 7faafaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.235+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab040bb020 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.235+0000 7faafaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.235+0000 7faafaffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.235+0000 7faafaffd640 10 monclient: dump: 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.246 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 5 monclient: authenticate success, global_id 4447 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 10 monclient: _renew_subs 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab040b3040 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab0c0a3ce0 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab040b2070 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: dump: 2024-09-11T02:50:13.248 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.249 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafbfff640 4 set_mon_vals no callback set 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 10 monclient: _renew_subs 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab0c128e20 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 1 librados: init done 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c127900 0x7fab0c126050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fab0c06ca10 con 0x7fab0c127900 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c127900 0x7fab0c126050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient: get_auth_request con 0x7fab0c127900 auth_method 0 2024-09-11T02:50:13.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fab04003c70 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7faaec03df40 0x7faaec040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 1 -- 172.21.15.98:0/407678708 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab0c13e500 con 0x7fab0c127900 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fab040e9c70 con 0x7fab0c0a4fc0 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 client.4447.objecter ms_dispatch 0x7fab0c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 3 client.4447.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:13.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 3 client.4447.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 20 client.4447.objecter dump_active .. 0 homeless 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab09d9b640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7faaec03df40 0x7faaec040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 client.4447.objecter ms_handle_connect 0x7fab0c127900 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 client.4447.objecter resend_mon_ops 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab09d9b640 10 monclient: get_auth_request con 0x7faaec03df40 auth_method 0 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab0a59c640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c127900 0x7fab0c126050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e770 tx=0x7faafc00eb70 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faafc0044a0 con 0x7fab0c127900 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: dump: 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.252 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 client.4447.objecter ms_handle_connect 0x7faaec03df40 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7fab09d9b640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7faaec03df40 0x7faaec040400 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fab00009580 tx=0x7fab00009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faafc004030 con 0x7fab0c127900 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c127900 msgr2=0x7fab0c126050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c127900 0x7fab0c126050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e770 tx=0x7faafc00eb70 comp rx=0 tx=0).stop 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c127900 msgr2=0x7fab0c126050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.239+0000 7faafaffd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c127900 0x7fab0c126050 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e770 tx=0x7faafc00eb70 comp rx=0 tx=0).stop 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab10edd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab10edd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c02d450 0x7fab0c02d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab10edd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fab0c06e870 con 0x7fab0c02d450 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c02d450 0x7fab0c02d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient: get_auth_request con 0x7fab0c02d450 auth_method 0 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 10 monclient(hunting): handle_auth_done global_id 4345 payload 824 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 1 -- 172.21.15.98:0/407678708 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fab0c071d20 con 0x7fab0c02d450 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7fab0a59c640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c02d450 0x7fab0c02d860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e770 tx=0x7faafc004740 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 10 client.4447.objecter ms_handle_connect 0x7fab0c02d450 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 10 client.4447.objecter resend_mon_ops 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faafc005c70 con 0x7fab0c02d450 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 10 monclient: dump: 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7faafc005c70 con 0x7fab0c02d450 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c02d450 msgr2=0x7fab0c02d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c02d450 0x7fab0c02d860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e770 tx=0x7faafc004740 comp rx=0 tx=0).stop 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c02d450 msgr2=0x7fab0c02d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.247+0000 7faafaffd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fab0c02d450 0x7fab0c02d860 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e770 tx=0x7faafc004740 comp rx=0 tx=0).stop 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 14, 2024-09-11T02:50:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 29019, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 28, 2024-09-11T02:50:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:13.262 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:13.263 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:13.264 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.268 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.270 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:13.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:13.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.275 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:13.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:13.280 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:13.281 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:13.282 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:13.282 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 10 librados: watch_flush enter 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 10 librados: watch_flush exit 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 20 client.4447.objecter shutdown clearing up homeless session... 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 10 client.4447.objecter successfully canceled tick 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7faaec03df40 msgr2=0x7faaec040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7faaec03df40 0x7faaec040400 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fab00009580 tx=0x7fab00009190 comp rx=0 tx=0).stop 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 10 monclient: shutdown 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab0c0a4fc0 msgr2=0x7fab0c125350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab0c0a4fc0 0x7fab0c125350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fab040b0040 tx=0x7fab040195a0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab0ad9d640 1 -- 172.21.15.98:0/407678708 reap_dead start 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 shutdown_connections 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 --2- 172.21.15.98:0/407678708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab0c0a4fc0 0x7fab0c125350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 >> 172.21.15.98:0/407678708 conn(0x7fab0c06eed0 msgr2=0x7fab0c09f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 shutdown_connections 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 -- 172.21.15.98:0/407678708 wait complete. 2024-09-11T02:50:13.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.251+0000 7fab10edd640 1 librados: shutdown 2024-09-11T02:50:13.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:50:13.286 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a', 'd'] 2024-09-11T02:50:13.286 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T02:50:13.286 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T02:50:13.286 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T02:50:13.286 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T02:50:13.286 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:50:13.297 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T02:50:13.297 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-09-11T02:50:13.297 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d store 2024-09-11T02:50:13.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d sync_force --yes-i-really-mean-it 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient: get_monmap_and_config 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient: monmap: 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:13.463233+0000 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:13.463233+0000 2024-09-11T02:50:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:13.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 1 Processor -- start 2024-09-11T02:50:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 1 -- start start 2024-09-11T02:50:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient: init 2024-09-11T02:50:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient: _add_conns ranks=[9,2,1,3,4,6,5,7,10,0,8] 2024-09-11T02:50:13.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0390139b80 0x7f0390072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient(hunting): picked mon.noname-a con 0x7f0390139b80 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0390139f60 0x7f03900a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient(hunting): picked mon.noname-b con 0x7f0390139f60 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f039013a380 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0390070320 con 0x7f039013a380 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f03900713c0 con 0x7f0390139b80 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0390072bd0 con 0x7f0390139f60 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f0394b6c640 10 monclient(hunting): authenticate will time out at 967.343447s 2024-09-11T02:50:13.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038e59c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f0390139b80 msgr2=0x7f0390072690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0390139b80 0x7f0390072690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0390139f60 0x7f03900a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0390139f60 0x7f03900a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48250/0 (socket says 172.21.15.98:48250) 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 -- 172.21.15.98:0/1010352685 learned_addr learned my addr 172.21.15.98:0/1010352685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): get_auth_request con 0x7f0390139f60 auth_method 0 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient(hunting): handle_auth_done global_id 4421 payload 274 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 -- 172.21.15.98:0/1010352685 >> v2:172.21.15.98:3300/0 conn(0x7f0390139b80 msgr2=0x7f0390072690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 --2- 172.21.15.98:0/1010352685 >> v2:172.21.15.98:3300/0 conn(0x7f0390139b80 0x7f0390072690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 -- 172.21.15.98:0/1010352685 >> v1:172.21.15.190:6789/0 conn(0x7f039013a380 legacy=0x7f03900a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 monclient: found mon.noname-b 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.459+0000 7f038dd9b640 1 -- 172.21.15.98:0/1010352685 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039006fff0 con 0x7f0390139f60 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038dd9b640 1 --2- 172.21.15.98:0/1010352685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0390139f60 0x7f03900a7a10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0378009870 tx=0x7f037801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffe4b39a8007d2c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 1 -- 172.21.15.98:0/1010352685 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03780b0070 con 0x7f0390139f60 2024-09-11T02:50:13.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: dump: 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 1 -- 172.21.15.98:0/1010352685 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03780aa810 con 0x7f0390139f60 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 5 monclient: authenticate success, global_id 4421 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:13.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 1 -- 172.21.15.98:0/1010352685 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03780a9300 con 0x7f0390139f60 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 4 set_mon_vals no callback set 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f038d59a640 10 monclient: dump: 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.477 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 10 monclient: shutdown 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 -- 172.21.15.98:0/1010352685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0390139f60 msgr2=0x7f03900a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 --2- 172.21.15.98:0/1010352685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0390139f60 0x7f03900a7a10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0378009870 tx=0x7f037801c440 comp rx=0 tx=0).stop 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 -- 172.21.15.98:0/1010352685 shutdown_connections 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 --2- 172.21.15.98:0/1010352685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0390139f60 0x7f03900a7a10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 --2- 172.21.15.98:0/1010352685 >> v2:172.21.15.98:3300/0 conn(0x7f0390139b80 0x7f0390072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 -- 172.21.15.98:0/1010352685 >> 172.21.15.98:0/1010352685 conn(0x7f03900ac7f0 msgr2=0x7f03900a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 -- 172.21.15.98:0/1010352685 shutdown_connections 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 -- 172.21.15.98:0/1010352685 wait complete. 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 10 monclient: monmap: 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:13.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 librados: starting msgr at 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 librados: starting objecter 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 Processor -- start 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 -- start start 2024-09-11T02:50:13.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 librados: setting wanted keys 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 1 librados: calling monclient init 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.463+0000 7f0394b6c640 10 monclient: init 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient: _add_conns ranks=[6,7,5,2,8,3,4,1,0] 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0390139b80 0x7f03900a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): picked mon.noname-d con 0x7f0390139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0390139f60 0x7f03900a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): picked mon.noname-i con 0x7f0390139f60 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f039013a380 0x7f03900a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): picked mon.noname-h con 0x7f039013a380 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f03901444e0 con 0x7f039013a380 2024-09-11T02:50:13.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f03900a8870 con 0x7f0390139b80 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0390144660 con 0x7f0390139f60 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): authenticate will time out at 967.348406s 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0390139b80 0x7f03900a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0390139b80 0x7f03900a4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:39666/0 (socket says 172.21.15.98:39666) 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 -- 172.21.15.98:0/1601941379 learned_addr learned my addr 172.21.15.98:0/1601941379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): get_auth_request con 0x7f0390139b80 auth_method 0 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0390139f60 0x7f03900a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): get_auth_request con 0x7f0390139f60 auth_method 0 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0390139f60 msgr2=0x7f03900a4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038ed9d640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f039013a380 0x7f03900a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0390139f60 0x7f03900a4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f039013a380 msgr2=0x7f03900a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f039013a380 0x7f03900a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 monclient: found mon.noname-d 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 -- 172.21.15.98:0/1601941379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f039013cfc0 con 0x7f0390139b80 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038e59c640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0390139b80 0x7f03900a4a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f038000b9e0 tx=0x7f038000bea0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 client.?.objecter ms_handle_connect 0x7f0390139b80 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f038000ce10 con 0x7f0390139b80 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:50:13.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: dump: 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 5 monclient: authenticate success, global_id 4461 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0380010070 con 0x7f0390139b80 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03800158e0 con 0x7f0390139b80 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:13.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038cd99640 4 set_mon_vals no callback set 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: dump: 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.487 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient: _renew_subs 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03900aba70 con 0x7f0390139b80 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient: _renew_subs 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f039013ac60 con 0x7f0390139b80 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 librados: init done 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1570 0x7f03901d1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f039006ca10 con 0x7f03901d1570 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1570 0x7f03901d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient: get_auth_request con 0x7f03901d1570 auth_method 0 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0380015470 con 0x7f0390139b80 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 10 monclient(hunting): handle_auth_done global_id 4470 payload 824 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 1 -- 172.21.15.98:0/1601941379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0390069670 con 0x7f03901d1570 2024-09-11T02:50:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f037003df40 0x7f0370040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f038004f140 con 0x7f0390139b80 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 client.4461.objecter ms_dispatch 0x7f0390000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 3 client.4461.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 3 client.4461.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 20 client.4461.objecter dump_active .. 0 homeless 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 client.4461.objecter ms_handle_connect 0x7f03901d1570 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 client.4461.objecter resend_mon_ops 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038dd9b640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1570 0x7f03901d1950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f039006fbb0 tx=0x7f03780191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03780b0050 con 0x7f03901d1570 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: dump: 2024-09-11T02:50:13.490 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.491 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038ed9d640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f037003df40 0x7f0370040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f038ed9d640 10 monclient: get_auth_request con 0x7f037003df40 auth_method 0 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f03780b1040 con 0x7f03901d1570 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1570 msgr2=0x7f03901d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1570 0x7f03901d1950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f039006fbb0 tx=0x7f03780191e0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1570 msgr2=0x7f03901d1950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.467+0000 7f0386ffd640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1570 0x7f03901d1950 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f039006fbb0 tx=0x7f03780191e0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.471+0000 7f0386ffd640 10 client.4461.objecter ms_handle_connect 0x7f037003df40 2024-09-11T02:50:13.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.471+0000 7f038ed9d640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f037003df40 0x7f0370040400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f039013bdf0 tx=0x7f0388009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0394b6c640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0394b6c640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1e90 0x7f03900a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0394b6c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f039006e870 con 0x7f03901d1e90 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1e90 0x7f03900a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient: get_auth_request con 0x7f03901d1e90 auth_method 0 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 10 monclient(hunting): handle_auth_done global_id 4479 payload 824 2024-09-11T02:50:13.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 1 -- 172.21.15.98:0/1601941379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f0390072690 con 0x7f03901d1e90 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0386ffd640 10 client.4461.objecter ms_handle_connect 0x7f03901d1e90 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f038dd9b640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1e90 0x7f03900a0e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f037801a070 tx=0x7f03780a8000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0386ffd640 10 client.4461.objecter resend_mon_ops 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03780bd750 con 0x7f03901d1e90 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0386ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0386ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.475+0000 7f0386ffd640 10 monclient: dump: 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.496 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f03780c1070 con 0x7f03901d1e90 2024-09-11T02:50:13.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0386ffd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0386ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1e90 msgr2=0x7f03900a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0386ffd640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1e90 0x7f03900a0e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f037801a070 tx=0x7f03780a8000 comp rx=0 tx=0).stop 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0386ffd640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1e90 msgr2=0x7f03900a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0386ffd640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f03901d1e90 0x7f03900a0e60 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f037801a070 tx=0x7f03780a8000 comp rx=0 tx=0).stop 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T02:50:13.498 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 10 librados: watch_flush enter 2024-09-11T02:50:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 10 librados: watch_flush exit 2024-09-11T02:50:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 20 client.4461.objecter shutdown clearing up homeless session... 2024-09-11T02:50:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 10 client.4461.objecter successfully canceled tick 2024-09-11T02:50:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f037003df40 msgr2=0x7f0370040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f037003df40 0x7f0370040400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f039013bdf0 tx=0x7f0388009040 comp rx=0 tx=0).stop 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 10 monclient: shutdown 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0390139b80 msgr2=0x7f03900a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 --2- 172.21.15.98:0/1601941379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0390139b80 0x7f03900a4a20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f038000b9e0 tx=0x7f038000bea0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f038ed9d640 1 -- 172.21.15.98:0/1601941379 reap_dead start 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f038ed9d640 1 -- 172.21.15.98:0/1601941379 reap_dead start 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 shutdown_connections 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 >> 172.21.15.98:0/1601941379 conn(0x7f03900ac7f0 msgr2=0x7f03900aa480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 shutdown_connections 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 -- 172.21.15.98:0/1601941379 wait complete. 2024-09-11T02:50:13.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.479+0000 7f0394b6c640 1 librados: shutdown 2024-09-11T02:50:13.502 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-11T02:50:13.502 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-11T02:50:13.502 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:50:13.513 INFO:tasks.ceph.mon.d:Stopped 2024-09-11T02:50:13.513 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-09-11T02:50:13.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:50:13.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient: get_monmap_and_config 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient: monmap: 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:13.671505+0000 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:13.671505+0000 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.668 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:13.669 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 Processor -- start 2024-09-11T02:50:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 -- start start 2024-09-11T02:50:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient: init 2024-09-11T02:50:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient: _add_conns ranks=[4,8,7,3,5,10,9,6,2,0,1] 2024-09-11T02:50:13.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f476806f6c0 0x7f4768072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): picked mon.noname-c con 0x7f476806f6c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f476806faa0 0x7f476806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): picked mon.noname-e con 0x7f476806faa0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f47680703c0 0x7f47681397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): picked mon.noname-i con 0x7f47680703c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4768071ff0 con 0x7f476806f6c0 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f47680a8750 con 0x7f47680703c0 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f47680716c0 con 0x7f476806faa0 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f476dc8e640 10 monclient(hunting): authenticate will time out at 967.551789s 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f47677fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f476806f6c0 0x7f4768072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.667+0000 7f4766ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f476806faa0 0x7f476806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f476806f6c0 0x7f4768072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39236/0 (socket says 172.21.15.98:39236) 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f476806faa0 0x7f476806fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35152/0 (socket says 172.21.15.98:35152) 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 -- 172.21.15.98:0/1241957419 learned_addr learned my addr 172.21.15.98:0/1241957419 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): get_auth_request con 0x7f476806f6c0 auth_method 0 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 10 monclient(hunting): get_auth_request con 0x7f476806faa0 auth_method 0 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4767fff640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f47680703c0 0x7f47681397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4767fff640 10 monclient(hunting): get_auth_request con 0x7f47680703c0 auth_method 0 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4767fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4767fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f4767fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient(hunting): handle_auth_done global_id 4432 payload 274 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 -- 172.21.15.98:0/1241957419 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f476806faa0 msgr2=0x7f476806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f476806faa0 0x7f476806fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 -- 172.21.15.98:0/1241957419 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f47680703c0 msgr2=0x7f47681397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f47680703c0 0x7f47681397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 monclient: found mon.noname-c 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:13.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 -- 172.21.15.98:0/1241957419 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680a64a0 con 0x7f476806f6c0 2024-09-11T02:50:13.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47677fe640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f476806f6c0 0x7f4768072940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4760009e80 tx=0x7f476001c650 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f783b7fd733db7cd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 1 -- 172.21.15.98:0/1241957419 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47600ac020 con 0x7f476806f6c0 2024-09-11T02:50:13.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:13.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: dump: 2024-09-11T02:50:13.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 1 -- 172.21.15.98:0/1241957419 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4760009110 con 0x7f476806f6c0 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 5 monclient: authenticate success, global_id 4432 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 4 set_mon_vals no callback set 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 1 -- 172.21.15.98:0/1241957419 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47600a6d40 con 0x7f476806f6c0 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f47667fc640 10 monclient: dump: 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.684 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 10 monclient: shutdown 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 -- 172.21.15.98:0/1241957419 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f476806f6c0 msgr2=0x7f4768072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f476806f6c0 0x7f4768072940 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4760009e80 tx=0x7f476001c650 comp rx=0 tx=0).stop 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 -- 172.21.15.98:0/1241957419 shutdown_connections 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f47680703c0 0x7f47681397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f476806faa0 0x7f476806fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 --2- 172.21.15.98:0/1241957419 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f476806f6c0 0x7f4768072940 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 -- 172.21.15.98:0/1241957419 >> 172.21.15.98:0/1241957419 conn(0x7f47680acac0 msgr2=0x7f47680a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 -- 172.21.15.98:0/1241957419 shutdown_connections 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.671+0000 7f476dc8e640 1 -- 172.21.15.98:0/1241957419 wait complete. 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: build_initial_monmap 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: monmap: 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:13.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 librados: starting msgr at 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 librados: starting objecter 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 Processor -- start 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 -- start start 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 librados: setting wanted keys 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 librados: calling monclient init 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: init 2024-09-11T02:50:13.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _add_conns ranks=[2,0,4,8,6,5,1,3,7] 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f476806f6c0 0x7f4768138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): picked mon.noname-b con 0x7f476806f6c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f476806faa0 0x7f4768138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): picked mon.noname-a con 0x7f476806faa0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47680703c0 0x7f476813ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): picked mon.noname-c con 0x7f47680703c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4768143840 con 0x7f476806faa0 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f47681436c0 con 0x7f476806f6c0 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:13.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f47681439c0 con 0x7f47680703c0 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient(hunting): authenticate will time out at 967.557278s 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4766ffd640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f476806faa0 msgr2=0x7f4768138b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4766ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f476806faa0 0x7f4768138b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f476806f6c0 0x7f4768138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f476806f6c0 0x7f4768138600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48280/0 (socket says 172.21.15.98:48280) 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 -- 172.21.15.98:0/2485781844 learned_addr learned my addr 172.21.15.98:0/2485781844 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): get_auth_request con 0x7f476806f6c0 auth_method 0 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47680703c0 0x7f476813ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 10 monclient(hunting): get_auth_request con 0x7f47680703c0 auth_method 0 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 -- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47680703c0 msgr2=0x7f476813ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47680703c0 0x7f476813ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 -- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f476806faa0 msgr2=0x7f4768138b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:13.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f476806faa0 0x7f4768138b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 monclient: found mon.noname-b 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 -- 172.21.15.98:0/2485781844 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680a55b0 con 0x7f476806f6c0 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4767fff640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47680703c0 0x7f476813ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f47677fe640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f476806f6c0 0x7f4768138600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f476001c480 tx=0x7f47600193f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 client.?.objecter ms_handle_connect 0x7f476806f6c0 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 1 -- 172.21.15.98:0/2485781844 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47600ac040 con 0x7f476806f6c0 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:13.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: dump: 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.692 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: _finish_auth 0 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 5 monclient: authenticate success, global_id 4430 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _renew_subs 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 1 -- 172.21.15.98:0/2485781844 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47600b6030 con 0x7f476806f6c0 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47681241f0 con 0x7f476806f6c0 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:13.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 1 -- 172.21.15.98:0/2485781844 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4760009510 con 0x7f476806f6c0 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4765ffb640 4 set_mon_vals no callback set 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 monclient: dump: 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:13.694 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _renew_subs 2024-09-11T02:50:13.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47681244d0 con 0x7f476806f6c0 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 librados: init done 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4768123c30 con 0x7f476806f6c0 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 1 -- 172.21.15.98:0/2485781844 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4760004680 con 0x7f476806f6c0 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f474803df40 0x7f4748040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 1 -- 172.21.15.98:0/2485781844 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f47600e91a0 con 0x7f476806f6c0 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 10 client.4430.objecter ms_dispatch 0x7f4768000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 3 client.4430.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:13.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 3 client.4430.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f475ffff640 20 client.4430.objecter dump_active .. 0 homeless 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4766ffd640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f474803df40 0x7f4748040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.675+0000 7f4766ffd640 10 monclient: get_auth_request con 0x7f474803df40 auth_method 0 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.679+0000 7f4766ffd640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f474803df40 0x7f4748040400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47500094e0 tx=0x7f4750009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.679+0000 7f475ffff640 10 client.4430.objecter ms_handle_connect 0x7f474803df40 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.683+0000 7f475ffff640 1 -- 172.21.15.98:0/2485781844 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f47600c0070 con 0x7f476806f6c0 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.683+0000 7f475ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:13.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.683+0000 7f475ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:13.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.907+0000 7f476dc8e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:50:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.907+0000 7f476dc8e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.907+0000 7f476dc8e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.907+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f47680a50a0 con 0x7f476806f6c0 2024-09-11T02:50:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f475ffff640 1 -- 172.21.15.98:0/2485781844 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f47680a55b0 con 0x7f476806f6c0 2024-09-11T02:50:13.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f475ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:13.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f475ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:13.910 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":29,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 10 librados: watch_flush enter 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 10 librados: watch_flush exit 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 20 client.4430.objecter shutdown clearing up homeless session... 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 10 client.4430.objecter successfully canceled tick 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f474803df40 msgr2=0x7f4748040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f474803df40 0x7f4748040400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47500094e0 tx=0x7f4750009190 comp rx=0 tx=0).stop 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 10 monclient: shutdown 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f476806f6c0 msgr2=0x7f4768138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:13.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f476806f6c0 0x7f4768138600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f476001c480 tx=0x7f47600193f0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 shutdown_connections 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f47680703c0 0x7f476813ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f476806faa0 0x7f4768138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f474803df40 0x7f4748040400 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 --2- 172.21.15.98:0/2485781844 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f476806f6c0 0x7f4768138600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 >> 172.21.15.98:0/2485781844 conn(0x7f47680acac0 msgr2=0x7f47680a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 shutdown_connections 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 -- 172.21.15.98:0/2485781844 wait complete. 2024-09-11T02:50:13.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:13.911+0000 7f476dc8e640 1 librados: shutdown 2024-09-11T02:50:16.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient: get_monmap_and_config 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient: build_initial_monmap 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient: monmap: 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:17.062700+0000 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:17.062700+0000 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:17.059 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:17.060 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 1 Processor -- start 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 1 -- start start 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient: init 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient: _add_conns ranks=[3,0,10,7,8,5,4,9,6,2,1] 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eac06f650 0x7f9eac072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient(hunting): picked mon.noname-g con 0x7f9eac06f650 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9eac0a8380 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9eac070140 0x7f9eac140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient(hunting): picked mon.noname-f con 0x7f9eac070140 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eac071a80 con 0x7f9eac0a8380 2024-09-11T02:50:17.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9eac0a8800 con 0x7f9eac070140 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9eac071d60 con 0x7f9eac06f650 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb3912640 10 monclient(hunting): authenticate will time out at 970.943118s 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb0ead640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9eac0a8380 legacy=0x7f9eac06fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9eac070140 0x7f9eac140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eac06f650 0x7f9eac072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9eac070140 0x7f9eac140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:35770/0 (socket says 172.21.15.98:35770) 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eac06f650 0x7f9eac072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:50946/0 (socket says 172.21.15.98:50946) 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 1 -- 172.21.15.98:0/585914024 learned_addr learned my addr 172.21.15.98:0/585914024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:17.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 10 monclient(hunting): get_auth_request con 0x7f9eac070140 auth_method 0 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 10 monclient(hunting): get_auth_request con 0x7f9eac06f650 auth_method 0 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb16ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:17.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.059+0000 7f9eb1eaf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 10 monclient(hunting): handle_auth_done global_id 4465 payload 274 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 1 -- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eac06f650 msgr2=0x7f9eac072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 1 --2- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eac06f650 0x7f9eac072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 1 -- 172.21.15.98:0/585914024 >> v1:172.21.15.98:6789/0 conn(0x7f9eac0a8380 legacy=0x7f9eac06fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 10 monclient: _finish_hunting 0 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 1 monclient: found mon.noname-f 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 1 -- 172.21.15.98:0/585914024 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac0714c0 con 0x7f9eac070140 2024-09-11T02:50:17.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb16ae640 1 --2- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eac06f650 0x7f9eac072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb1eaf640 1 --2- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eac070140 0x7f9eac140c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009870 tx=0x7f9ea401c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=507706562aa72ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 1 -- 172.21.15.98:0/585914024 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea40b1070 con 0x7f9eac070140 2024-09-11T02:50:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: dump: 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: _finish_auth 0 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 1 -- 172.21.15.98:0/585914024 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea40ab830 con 0x7f9eac070140 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 1 -- 172.21.15.98:0/585914024 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea40aa320 con 0x7f9eac070140 2024-09-11T02:50:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9e9ffff640 10 monclient: dump: 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 5 monclient: authenticate success, global_id 4465 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 4 set_mon_vals no callback set 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 10 monclient: shutdown 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 -- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eac070140 msgr2=0x7f9eac140c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 --2- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eac070140 0x7f9eac140c20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4009870 tx=0x7f9ea401c420 comp rx=0 tx=0).stop 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 -- 172.21.15.98:0/585914024 shutdown_connections 2024-09-11T02:50:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 --2- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9eac070140 0x7f9eac140c20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 --2- 172.21.15.98:0/585914024 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eac06f650 0x7f9eac072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 -- 172.21.15.98:0/585914024 >> 172.21.15.98:0/585914024 conn(0x7f9eac0ac8e0 msgr2=0x7f9eac0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 -- 172.21.15.98:0/585914024 shutdown_connections 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.063+0000 7f9eb3912640 1 -- 172.21.15.98:0/585914024 wait complete. 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: build_initial_monmap 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: monmap: 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 librados: starting msgr at 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 librados: starting objecter 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 Processor -- start 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 -- start start 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 librados: setting wanted keys 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 librados: calling monclient init 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: init 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _add_conns ranks=[6,2,7,4,8,5,1,0,3] 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9eac06f650 0x7f9eac128c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): picked mon.noname-d con 0x7f9eac06f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eac070140 0x7f9eac13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): picked mon.noname-b con 0x7f9eac070140 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eac0a8380 0x7f9eac126650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): picked mon.noname-i con 0x7f9eac0a8380 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9eac143d10 con 0x7f9eac070140 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9eac143b90 con 0x7f9eac06f650 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb16ae640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9eac06f650 msgr2=0x7f9eac128c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f9eac143e90 con 0x7f9eac0a8380 2024-09-11T02:50:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient(hunting): authenticate will time out at 970.949584s 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb16ae640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9eac06f650 0x7f9eac128c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eac070140 0x7f9eac13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eac070140 0x7f9eac13fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:48282/0 (socket says 172.21.15.98:48282) 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 -- 172.21.15.98:0/2421294494 learned_addr learned my addr 172.21.15.98:0/2421294494 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): get_auth_request con 0x7f9eac070140 auth_method 0 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:17.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eac0a8380 0x7f9eac126650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 10 monclient(hunting): get_auth_request con 0x7f9eac0a8380 auth_method 0 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb1eaf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 -- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eac0a8380 msgr2=0x7f9eac126650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eac0a8380 0x7f9eac126650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 -- 172.21.15.98:0/2421294494 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9eac06f650 msgr2=0x7f9eac128c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9eac06f650 0x7f9eac128c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient: _finish_hunting 0 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 monclient: found mon.noname-b 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 -- 172.21.15.98:0/2421294494 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac12a470 con 0x7f9eac070140 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb0ead640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eac070140 0x7f9eac13fae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800b4a0 tx=0x7f9ea800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 client.?.objecter ms_handle_connect 0x7f9eac070140 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 1 -- 172.21.15.98:0/2421294494 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea8003ba0 con 0x7f9eac070140 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: dump: 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 1 -- 172.21.15.98:0/2421294494 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea8015070 con 0x7f9eac070140 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 1 -- 172.21.15.98:0/2421294494 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea8003d80 con 0x7f9eac070140 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9f7fe640 4 set_mon_vals no callback set 2024-09-11T02:50:17.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 monclient: dump: 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:17.086 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 5 monclient: authenticate success, global_id 4439 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _renew_subs 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eac0733a0 con 0x7f9eac070140 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _renew_subs 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eac128110 con 0x7f9eac070140 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 librados: init done 2024-09-11T02:50:17.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eac072640 con 0x7f9eac070140 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 1 -- 172.21.15.98:0/2421294494 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9ea8005070 con 0x7f9eac070140 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9e9003df40 0x7f9e90040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 1 -- 172.21.15.98:0/2421294494 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f9ea800d040 con 0x7f9eac070140 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 10 client.4439.objecter ms_dispatch 0x7f9eac000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 3 client.4439.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.067+0000 7f9e9dffb640 3 client.4439.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.071+0000 7f9e9dffb640 20 client.4439.objecter dump_active .. 0 homeless 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.071+0000 7f9eb16ae640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9e9003df40 0x7f9e90040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:17.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.071+0000 7f9eb16ae640 10 monclient: get_auth_request con 0x7f9e9003df40 auth_method 0 2024-09-11T02:50:17.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.071+0000 7f9e9dffb640 10 client.4439.objecter ms_handle_connect 0x7f9e9003df40 2024-09-11T02:50:17.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.071+0000 7f9eb16ae640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9e9003df40 0x7f9e90040400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009bd0 tx=0x7f9ea000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:17.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.075+0000 7f9e9dffb640 1 -- 172.21.15.98:0/2421294494 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f9ea801adc0 con 0x7f9eac070140 2024-09-11T02:50:17.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.075+0000 7f9e9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:17.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.075+0000 7f9e9dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:17.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9eb3912640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:50:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9eb3912640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9eb3912640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:17.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9eac068f80 con 0x7f9eac070140 2024-09-11T02:50:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9e9dffb640 1 -- 172.21.15.98:0/2421294494 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f9ea801a980 con 0x7f9eac070140 2024-09-11T02:50:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9e9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:17.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9e9dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:17.293 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":32,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:50:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.291+0000 7f9eb3912640 10 librados: watch_flush enter 2024-09-11T02:50:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 10 librados: watch_flush exit 2024-09-11T02:50:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 20 client.4439.objecter shutdown clearing up homeless session... 2024-09-11T02:50:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 10 client.4439.objecter successfully canceled tick 2024-09-11T02:50:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9e9003df40 msgr2=0x7f9e90040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9e9003df40 0x7f9e90040400 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9ea0009bd0 tx=0x7f9ea000b040 comp rx=0 tx=0).stop 2024-09-11T02:50:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 10 monclient: shutdown 2024-09-11T02:50:17.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eac070140 msgr2=0x7f9eac13fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eac070140 0x7f9eac13fae0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ea800b4a0 tx=0x7f9ea800b960 comp rx=0 tx=0).stop 2024-09-11T02:50:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 shutdown_connections 2024-09-11T02:50:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9e9003df40 0x7f9e90040400 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eac070140 0x7f9eac13fae0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eac0a8380 0x7f9eac126650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 --2- 172.21.15.98:0/2421294494 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9eac06f650 0x7f9eac128c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 >> 172.21.15.98:0/2421294494 conn(0x7f9eac0ac8e0 msgr2=0x7f9eac0a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 shutdown_connections 2024-09-11T02:50:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 -- 172.21.15.98:0/2421294494 wait complete. 2024-09-11T02:50:17.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:17.295+0000 7f9eb3912640 1 librados: shutdown 2024-09-11T02:50:20.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:50:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient: get_monmap_and_config 2024-09-11T02:50:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient: build_initial_monmap 2024-09-11T02:50:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient: monmap: 2024-09-11T02:50:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:20.442894+0000 2024-09-11T02:50:20.439 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:20.442894+0000 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:20.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:20.441 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:20.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:20.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 Processor -- start 2024-09-11T02:50:20.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 -- start start 2024-09-11T02:50:20.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient: init 2024-09-11T02:50:20.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:20.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient: _add_conns ranks=[3,8,2,5,0,9,6,4,7,1,10] 2024-09-11T02:50:20.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66f00a5930 0x7f66f00a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): picked mon.noname-g con 0x7f66f00a5930 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66f00a1870 0x7f66f00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): picked mon.noname-e con 0x7f66f00a1870 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f01376e0 0x7f66f0137b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): picked mon.noname-b con 0x7f66f01376e0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f66f00a4170 con 0x7f66f01376e0 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f66f00a6790 con 0x7f66f00a5930 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:20.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f66f0071d90 con 0x7f66f00a1870 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f7cc2640 10 monclient(hunting): authenticate will time out at 974.323263s 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66f00a1870 0x7f66f00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f01376e0 0x7f66f0137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66f00a1870 0x7f66f00a6250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:46252/0 (socket says 172.21.15.98:46252) 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f01376e0 0x7f66f0137b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54876/0 (socket says 172.21.15.98:54876) 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 1 -- 172.21.15.98:0/4114018893 learned_addr learned my addr 172.21.15.98:0/4114018893 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:20.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 10 monclient(hunting): get_auth_request con 0x7f66f00a1870 auth_method 0 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): get_auth_request con 0x7f66f01376e0 auth_method 0 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66f00a5930 0x7f66f00a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 10 monclient(hunting): get_auth_request con 0x7f66f00a5930 auth_method 0 2024-09-11T02:50:20.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f525d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f5a5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient(hunting): handle_auth_done global_id 4448 payload 274 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 -- 172.21.15.98:0/4114018893 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66f00a1870 msgr2=0x7f66f00a6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66f00a1870 0x7f66f00a6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 -- 172.21.15.98:0/4114018893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66f00a5930 msgr2=0x7f66f00a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66f00a5930 0x7f66f00a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient: _finish_hunting 0 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 monclient: found mon.noname-b 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.439+0000 7f66f625f640 1 -- 172.21.15.98:0/4114018893 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f0073b50 con 0x7f66f01376e0 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f625f640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f01376e0 0x7f66f0137b40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009820 tx=0x7f66e801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d60803857821c11f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:20.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 1 -- 172.21.15.98:0/4114018893 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66e80b0070 con 0x7f66f01376e0 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: dump: 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:20.456 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: _finish_auth 0 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: _check_auth_tickets 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:20.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 5 monclient: authenticate success, global_id 4448 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 1 -- 172.21.15.98:0/4114018893 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e80aa810 con 0x7f66f01376e0 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 4 set_mon_vals no callback set 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 1 -- 172.21.15.98:0/4114018893 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66e80a9300 con 0x7f66f01376e0 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f4a5c640 10 monclient: dump: 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:20.458 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:20.459 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 10 monclient: shutdown 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 -- 172.21.15.98:0/4114018893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f01376e0 msgr2=0x7f66f0137b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f01376e0 0x7f66f0137b40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009820 tx=0x7f66e801c410 comp rx=0 tx=0).stop 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 -- 172.21.15.98:0/4114018893 shutdown_connections 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f01376e0 0x7f66f0137b40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66f00a1870 0x7f66f00a6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/4114018893 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66f00a5930 0x7f66f00a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 -- 172.21.15.98:0/4114018893 >> 172.21.15.98:0/4114018893 conn(0x7f66f00aa520 msgr2=0x7f66f00a6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 -- 172.21.15.98:0/4114018893 shutdown_connections 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 1 -- 172.21.15.98:0/4114018893 wait complete. 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 10 monclient: build_initial_monmap 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.443+0000 7f66f7cc2640 10 monclient: monmap: 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:20.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:20.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 librados: starting msgr at 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 librados: starting objecter 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 Processor -- start 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 -- start start 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 librados: setting wanted keys 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 librados: calling monclient init 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: init 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _add_conns ranks=[2,7,6,4,1,0,8,5,3] 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f00a1870 0x7f66f0136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): picked mon.noname-b con 0x7f66f00a1870 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f66f00a5930 0x7f66f01366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:20.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): picked mon.noname-i con 0x7f66f00a5930 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f66f01376e0 0x7f66f013b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): picked mon.noname-d con 0x7f66f01376e0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f66f0141150 con 0x7f66f00a1870 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f66f00a7420 con 0x7f66f01376e0 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f66f01412d0 con 0x7f66f00a5930 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient(hunting): authenticate will time out at 974.328845s 2024-09-11T02:50:20.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f625f640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f66f01376e0 msgr2=0x7f66f013b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f625f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f66f01376e0 0x7f66f013b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f00a1870 0x7f66f0136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f00a1870 0x7f66f0136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54878/0 (socket says 172.21.15.98:54878) 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 -- 172.21.15.98:0/3496290211 learned_addr learned my addr 172.21.15.98:0/3496290211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): get_auth_request con 0x7f66f00a1870 auth_method 0 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:20.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f525d640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f66f00a5930 0x7f66f01366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f525d640 10 monclient(hunting): get_auth_request con 0x7f66f00a5930 auth_method 0 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f525d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f525d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f525d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient(hunting): handle_auth_done global_id 4457 payload 824 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 -- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f66f00a5930 msgr2=0x7f66f01366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f66f00a5930 0x7f66f01366c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 -- 172.21.15.98:0/3496290211 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f66f01376e0 msgr2=0x7f66f013b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f66f01376e0 0x7f66f013b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient: _finish_hunting 0 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 monclient: found mon.noname-b 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 -- 172.21.15.98:0/3496290211 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f0121c20 con 0x7f66f00a1870 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f5a5e640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f00a1870 0x7f66f0136180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f66e000e930 tx=0x7f66e000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 client.?.objecter ms_handle_connect 0x7f66f00a1870 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 1 -- 172.21.15.98:0/3496290211 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66e0017070 con 0x7f66f00a1870 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:20.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: dump: 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:20.468 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 5 monclient: authenticate success, global_id 4457 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 1 -- 172.21.15.98:0/3496290211 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e0013470 con 0x7f66f00a1870 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 1 -- 172.21.15.98:0/3496290211 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66e0013a90 con 0x7f66f00a1870 2024-09-11T02:50:20.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _renew_subs 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f00a92a0 con 0x7f66f00a1870 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e7fff640 4 set_mon_vals no callback set 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 monclient: dump: 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:20.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:20.471 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _renew_subs 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f0136c00 con 0x7f66f00a1870 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 librados: init done 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 1 -- 172.21.15.98:0/3496290211 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f66e0013c30 con 0x7f66f00a1870 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f0137b40 con 0x7f66f00a1870 2024-09-11T02:50:20.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66dc03df70 0x7f66dc040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 1 -- 172.21.15.98:0/3496290211 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f66e000f040 con 0x7f66f00a1870 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 10 client.4457.objecter ms_dispatch 0x7f66f0000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 3 client.4457.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 3 client.4457.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66e67fc640 20 client.4457.objecter dump_active .. 0 homeless 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f525d640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66dc03df70 0x7f66dc040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.447+0000 7f66f525d640 10 monclient: get_auth_request con 0x7f66dc03df70 auth_method 0 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.451+0000 7f66e67fc640 10 client.4457.objecter ms_handle_connect 0x7f66dc03df70 2024-09-11T02:50:20.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.451+0000 7f66f525d640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66dc03df70 0x7f66dc040430 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66f013d5e0 tx=0x7f66ec00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.451+0000 7f66e67fc640 1 -- 172.21.15.98:0/3496290211 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f66e0021070 con 0x7f66f00a1870 2024-09-11T02:50:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.451+0000 7f66e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:20.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.451+0000 7f66e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:20.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:50:20.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:20.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:20.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f66f0069090 con 0x7f66f00a1870 2024-09-11T02:50:20.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66e67fc640 1 -- 172.21.15.98:0/3496290211 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f66e003dc90 con 0x7f66f00a1870 2024-09-11T02:50:20.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:20.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:20.691 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":14,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":36,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 10 librados: watch_flush enter 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 10 librados: watch_flush exit 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 20 client.4457.objecter shutdown clearing up homeless session... 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 10 client.4457.objecter successfully canceled tick 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66dc03df70 msgr2=0x7f66dc040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66dc03df70 0x7f66dc040430 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66f013d5e0 tx=0x7f66ec00b040 comp rx=0 tx=0).stop 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 10 monclient: shutdown 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f00a1870 msgr2=0x7f66f0136180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:20.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f00a1870 0x7f66f0136180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f66e000e930 tx=0x7f66e000edf0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 shutdown_connections 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f66f01376e0 0x7f66f013b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f66f00a5930 0x7f66f01366c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66dc03df70 0x7f66dc040430 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 --2- 172.21.15.98:0/3496290211 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f66f00a1870 0x7f66f0136180 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f66e000e930 tx=0x7f66e000edf0 comp rx=0 tx=0).stop 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 >> 172.21.15.98:0/3496290211 conn(0x7f66f00aa520 msgr2=0x7f66f00a2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 shutdown_connections 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 -- 172.21.15.98:0/3496290211 wait complete. 2024-09-11T02:50:20.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:20.691+0000 7f66f7cc2640 1 librados: shutdown 2024-09-11T02:50:23.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:50:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient: get_monmap_and_config 2024-09-11T02:50:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient: build_initial_monmap 2024-09-11T02:50:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient: monmap: 2024-09-11T02:50:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:23.843014+0000 2024-09-11T02:50:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:23.843014+0000 2024-09-11T02:50:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:23.841 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 1 Processor -- start 2024-09-11T02:50:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 1 -- start start 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient: init 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient: _add_conns ranks=[0,8,7,4,3,10,5,2,1,6,9] 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e8c0a3280 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient(hunting): picked mon.noname-e con 0x7f5e8c0a1c30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient(hunting): picked mon.noname-i con 0x7f5e8c0a3d70 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8c0a4600 con 0x7f5e8c0a3280 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f5e8c0a4cc0 con 0x7f5e8c0a3d70 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5e8c09c5e0 con 0x7f5e8c0a1c30 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e917d7640 10 monclient(hunting): authenticate will time out at 977.723889s 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.839+0000 7f5e8affd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f5e8c0a3280 legacy=0x7f5e8c0a3660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:46260/0 (socket says 172.21.15.98:46260) 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 learned_addr learned my addr 172.21.15.98:0/2794564829 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a1c30 auth_method 0 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a3d70 auth_method 0 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:50:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:23.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:23.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:23.843+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:24.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.043+0000 7f5e8affd640 1 -- 172.21.15.98:0/2794564829 >> v1:172.21.15.98:6789/0 conn(0x7f5e8c0a3280 legacy=0x7f5e8c0a3660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a3d70 auth_method 0 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:50:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a1c30 auth_method 0 2024-09-11T02:50:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:50:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.047+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:50:24.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.447+0000 7f5e8affd640 1 -- 172.21.15.98:0/2794564829 >> v1:172.21.15.98:6789/0 conn(0x7f5e8c0a3280 legacy=0x7f5e8c0a3660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:50:24.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a1c30 auth_method 0 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a3d70 auth_method 0 2024-09-11T02:50:24.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:24.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:24.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:24.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:50:24.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:24.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:24.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:24.451+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:50:25.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.251+0000 7f5e8affd640 1 -- 172.21.15.98:0/2794564829 >> v1:172.21.15.98:6789/0 conn(0x7f5e8c0a3280 legacy=0x7f5e8c0a3660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:50:25.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:25.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a1c30 auth_method 0 2024-09-11T02:50:25.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e8c0a3d70 auth_method 0 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:50:25.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:25.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:25.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:25.255+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:50:26.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): tick 2024-09-11T02:50:26.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:50:26.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 monclient(hunting): continuing hunt 2024-09-11T02:50:26.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:50:26.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 msgr2=0x7f5e8c13cb40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:26.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5e8c0a1c30 0x7f5e8c13cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 msgr2=0x7f5e8c13f090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e8c0a3d70 0x7f5e8c13f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 -- 172.21.15.98:0/2794564829 >> v1:172.21.15.98:6789/0 conn(0x7f5e8c0a3280 legacy=0x7f5e8c0a3660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient: _add_conns ranks=[9,4,3,5,10,2,1,6] 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 --2- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 0x7f5e6c005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5e6c0037f0 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): picked mon.noname-c con 0x7f5e6c006240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): picked mon.noname-g con 0x7f5e6c008d30 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:26.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 -- 172.21.15.98:0/2794564829 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5e8c0a4cc0 con 0x7f5e6c0037f0 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 -- 172.21.15.98:0/2794564829 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8c09c5e0 con 0x7f5e6c008d30 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 1 -- 172.21.15.98:0/2794564829 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5e6c00b960 con 0x7f5e6c006240 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e897fa640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8affd640 1 -- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 msgr2=0x7f5e6c005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8affd640 1 --2- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 0x7f5e6c005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e6c006240 auth_method 0 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:26.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e6c008d30 auth_method 0 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 msgr2=0x7f5e6c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 msgr2=0x7f5e6c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 msgr2=0x7f5e6c00b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:50:26.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 msgr2=0x7f5e6c00b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:26.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:26.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:26.843+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:27.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8affd640 1 -- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 msgr2=0x7f5e6c005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:27.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8affd640 1 --2- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 0x7f5e6c005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e6c006240 auth_method 0 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e6c008d30 auth_method 0 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:27.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 msgr2=0x7f5e6c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 msgr2=0x7f5e6c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 msgr2=0x7f5e6c00b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 msgr2=0x7f5e6c00b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.047+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e6c006240 auth_method 0 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e6c008d30 auth_method 0 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:27.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 msgr2=0x7f5e6c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 msgr2=0x7f5e6c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 msgr2=0x7f5e6c00b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 msgr2=0x7f5e6c00b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8affd640 1 -- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 msgr2=0x7f5e6c005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:27.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:27.451+0000 7f5e8affd640 1 --2- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 0x7f5e6c005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:50:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8affd640 1 -- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 msgr2=0x7f5e6c005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:28.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8affd640 1 --2- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 0x7f5e6c005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:50:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e6c006240 auth_method 0 2024-09-11T02:50:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request con 0x7f5e6c008d30 auth_method 0 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:50:28.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient(hunting): handle_auth_done global_id 4450 payload 274 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 msgr2=0x7f5e6c00b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5e6c008d30 0x7f5e6c00b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 msgr2=0x7f5e6c005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> v2:172.21.15.98:3300/0 conn(0x7f5e6c0037f0 0x7f5e6c005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient: _finish_hunting 0 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 1 monclient: found mon.noname-c 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e8b7fe640 1 -- 172.21.15.98:0/2794564829 reap_dead start 2024-09-11T02:50:28.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.255+0000 7f5e827fc640 1 -- 172.21.15.98:0/2794564829 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8c0a2180 con 0x7f5e6c006240 2024-09-11T02:50:28.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.259+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0a94d0 tx=0x7f5e7401dd60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d2d99ec3cf29996d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 1 -- 172.21.15.98:0/2794564829 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5e740ac070 con 0x7f5e6c006240 2024-09-11T02:50:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:28.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: dump: 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.272 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:28.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 1 -- 172.21.15.98:0/2794564829 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e740aa070 con 0x7f5e6c006240 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 1 -- 172.21.15.98:0/2794564829 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5e7400f040 con 0x7f5e6c006240 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 5 monclient: authenticate success, global_id 4450 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 4 set_mon_vals no callback set 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e8a7fc640 10 monclient: dump: 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.276 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 10 monclient: shutdown 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 1 -- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 msgr2=0x7f5e6c0086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0a94d0 tx=0x7f5e7401dd60 comp rx=0 tx=0).stop 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 1 -- 172.21.15.98:0/2794564829 shutdown_connections 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2794564829 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5e6c006240 0x7f5e6c0086d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 1 -- 172.21.15.98:0/2794564829 >> 172.21.15.98:0/2794564829 conn(0x7f5e8c0a86f0 msgr2=0x7f5e8c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:28.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 1 -- 172.21.15.98:0/2794564829 shutdown_connections 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.271+0000 7f5e917d7640 1 -- 172.21.15.98:0/2794564829 wait complete. 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient: build_initial_monmap 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient: monmap: 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:28.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 librados: starting msgr at 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 librados: starting objecter 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 Processor -- start 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 -- start start 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 librados: setting wanted keys 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 librados: calling monclient init 2024-09-11T02:50:28.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient: init 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient: _add_conns ranks=[7,2,6,3,8,5,4,0,1] 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e6c006240 0x7f5e8c1cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): picked mon.noname-i con 0x7f5e6c006240 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e8c1d0320 0x7f5e8c132350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): picked mon.noname-b con 0x7f5e8c1d0320 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5e8c1d0720 0x7f5e8c132890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): picked mon.noname-d con 0x7f5e8c1d0720 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8c09bb40 con 0x7f5e8c1d0320 2024-09-11T02:50:28.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5e8c09b9c0 con 0x7f5e8c1d0720 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f5e8c09bcc0 con 0x7f5e6c006240 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient(hunting): authenticate will time out at 982.158640s 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8b7fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5e8c1d0720 msgr2=0x7f5e8c132890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8b7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5e8c1d0720 0x7f5e8c132890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e8c1d0320 0x7f5e8c132350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e8c1d0320 0x7f5e8c132350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:44362/0 (socket says 172.21.15.98:44362) 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 -- 172.21.15.98:0/2117784849 learned_addr learned my addr 172.21.15.98:0/2117784849 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:28.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): get_auth_request con 0x7f5e8c1d0320 auth_method 0 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e6c006240 0x7f5e8c1cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 10 monclient(hunting): get_auth_request con 0x7f5e6c006240 auth_method 0 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient(hunting): handle_auth_done global_id 4466 payload 824 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 -- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e6c006240 msgr2=0x7f5e8c1cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e6c006240 0x7f5e8c1cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 -- 172.21.15.98:0/2117784849 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5e8c1d0720 msgr2=0x7f5e8c132890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5e8c1d0720 0x7f5e8c132890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient: _finish_hunting 0 2024-09-11T02:50:28.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 monclient: found mon.noname-b 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 -- 172.21.15.98:0/2117784849 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8c074370 con 0x7f5e8c1d0320 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e827fc640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e8c1d0320 0x7f5e8c132350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e7401db90 tx=0x7f5e74008a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 client.?.objecter ms_handle_connect 0x7f5e8c1d0320 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 1 -- 172.21.15.98:0/2117784849 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5e740ac040 con 0x7f5e8c1d0320 2024-09-11T02:50:28.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: dump: 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.290 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 5 monclient: authenticate success, global_id 4466 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 1 -- 172.21.15.98:0/2117784849 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e740a68a0 con 0x7f5e8c1d0320 2024-09-11T02:50:28.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 1 -- 172.21.15.98:0/2117784849 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5e74003620 con 0x7f5e8c1d0320 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e83fff640 10 monclient: dump: 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.292 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e89ffb640 4 set_mon_vals no callback set 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient: _renew_subs 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:28.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.275+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8c135990 con 0x7f5e8c1d0320 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 10 monclient: _renew_subs 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e8c1344d0 con 0x7f5e8c1d0320 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 1 librados: init done 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e8c13ab30 con 0x7f5e8c1d0320 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 1 -- 172.21.15.98:0/2117784849 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5e740b8070 con 0x7f5e8c1d0320 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5e6c03f180 0x7f5e6c041640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 1 -- 172.21.15.98:0/2117784849 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f5e740e99d0 con 0x7f5e8c1d0320 2024-09-11T02:50:28.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 10 client.4466.objecter ms_dispatch 0x7f5e8c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:28.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 3 client.4466.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:28.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 3 client.4466.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:28.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 20 client.4466.objecter dump_active .. 0 homeless 2024-09-11T02:50:28.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e8affd640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5e6c03f180 0x7f5e6c041640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e8affd640 10 monclient: get_auth_request con 0x7f5e6c03f180 auth_method 0 2024-09-11T02:50:28.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e83fff640 10 client.4466.objecter ms_handle_connect 0x7f5e6c03f180 2024-09-11T02:50:28.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.279+0000 7f5e8affd640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5e6c03f180 0x7f5e6c041640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00a610 tx=0x7f5e7c00a5a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.283+0000 7f5e83fff640 1 -- 172.21.15.98:0/2117784849 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f5e8c074370 con 0x7f5e8c1d0320 2024-09-11T02:50:28.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.283+0000 7f5e83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:28.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.283+0000 7f5e83fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.531+0000 7f5e917d7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:50:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.531+0000 7f5e917d7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.531+0000 7f5e917d7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.531+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e8c135cd0 con 0x7f5e8c1d0320 2024-09-11T02:50:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.531+0000 7f5e83fff640 1 -- 172.21.15.98:0/2117784849 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f5e7400a930 con 0x7f5e8c1d0320 2024-09-11T02:50:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.531+0000 7f5e83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:50:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.531+0000 7f5e83fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:28.532 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":22,"quorum":[1,2,3,4,5,7,8],"quorum_names":["f","b","g","c","h","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:50:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 10 librados: watch_flush enter 2024-09-11T02:50:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 10 librados: watch_flush exit 2024-09-11T02:50:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 20 client.4466.objecter shutdown clearing up homeless session... 2024-09-11T02:50:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 10 client.4466.objecter successfully canceled tick 2024-09-11T02:50:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5e6c03f180 msgr2=0x7f5e6c041640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5e6c03f180 0x7f5e6c041640 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c00a610 tx=0x7f5e7c00a5a0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 10 monclient: shutdown 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e8c1d0320 msgr2=0x7f5e8c132350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e8c1d0320 0x7f5e8c132350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e7401db90 tx=0x7f5e74008a40 comp rx=0 tx=0).stop 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 shutdown_connections 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5e6c03f180 0x7f5e6c041640 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5e8c1d0320 0x7f5e8c132350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5e8c1d0720 0x7f5e8c132890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 --2- 172.21.15.98:0/2117784849 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5e6c006240 0x7f5e8c1cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 >> 172.21.15.98:0/2117784849 conn(0x7f5e8c0a86f0 msgr2=0x7f5e8c09aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:28.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 shutdown_connections 2024-09-11T02:50:28.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 -- 172.21.15.98:0/2117784849 wait complete. 2024-09-11T02:50:28.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.535+0000 7f5e917d7640 1 librados: shutdown 2024-09-11T02:50:28.552 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-09-11T02:50:28.552 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-09-11T02:50:28.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:50:28.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.667+0000 7f9a32ac5640 10 monclient: get_monmap_and_config 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.667+0000 7f9a32ac5640 10 monclient: build_initial_monmap 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.667+0000 7f9a32ac5640 10 monclient: monmap: 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:28.673027+0000 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:28.673027+0000 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:28.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 Processor -- start 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 -- start start 2024-09-11T02:50:28.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient: init 2024-09-11T02:50:28.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:28.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient: _add_conns ranks=[2,4,10,1,3,8,9,7,6,0,5] 2024-09-11T02:50:28.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c139b10 0x7f9a2c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2c139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a2c139ef0 0x7f9a2c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): picked mon.noname-c con 0x7f9a2c139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9a2c13a2d0 0x7f9a2c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): picked mon.noname-f con 0x7f9a2c13a2d0 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9a2c0702b0 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a2c071bf0 con 0x7f9a2c139b10 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9a2c071350 con 0x7f9a2c139ef0 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:28.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a32ac5640 10 monclient(hunting): authenticate will time out at 982.553268s 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c139b10 0x7f9a2c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a2c139ef0 0x7f9a2c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c139b10 0x7f9a2c072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:44366/0 (socket says 172.21.15.98:44366) 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a2c139ef0 0x7f9a2c0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:36676/0 (socket says 172.21.15.98:36676) 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 -- 172.21.15.98:0/318566105 learned_addr learned my addr 172.21.15.98:0/318566105 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): get_auth_request con 0x7f9a2c139b10 auth_method 0 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 10 monclient(hunting): get_auth_request con 0x7f9a2c139ef0 auth_method 0 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient(hunting): handle_auth_done global_id 4475 payload 274 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a31062640 1 --2- 172.21.15.98:0/318566105 >> v2:172.21.15.190:3300/0 conn(0x7f9a2c13a2d0 0x7f9a2c0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 -- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a2c139ef0 msgr2=0x7f9a2c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 --2- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a2c139ef0 0x7f9a2c0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 -- 172.21.15.98:0/318566105 >> v2:172.21.15.190:3300/0 conn(0x7f9a2c13a2d0 msgr2=0x7f9a2c0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:28.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 --2- 172.21.15.98:0/318566105 >> v2:172.21.15.190:3300/0 conn(0x7f9a2c13a2d0 0x7f9a2c0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient: _finish_hunting 0 2024-09-11T02:50:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 monclient: found mon.noname-b 2024-09-11T02:50:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:28.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 -- 172.21.15.98:0/318566105 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2c0a9620 con 0x7f9a2c139b10 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a31062640 10 monclient: get_auth_request con 0x7f9a2c13a2d0 auth_method 0 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a2bfff640 1 --2- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a2c139ef0 0x7f9a2c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a31062640 1 --2- 172.21.15.98:0/318566105 >> v2:172.21.15.190:3300/0 conn(0x7f9a2c13a2d0 0x7f9a2c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.671+0000 7f9a30861640 1 --2- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c139b10 0x7f9a2c072600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0098a0 tx=0x7f9a1c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3b728708ca8046 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 1 -- 172.21.15.98:0/318566105 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a1c0b0070 con 0x7f9a2c139b10 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: dump: 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 1 -- 172.21.15.98:0/318566105 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a1c0aa810 con 0x7f9a2c139b10 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 5 monclient: authenticate success, global_id 4475 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 1 -- 172.21.15.98:0/318566105 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a1c0a9300 con 0x7f9a2c139b10 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 4 set_mon_vals no callback set 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:28.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2b7fe640 10 monclient: dump: 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.690 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: shutdown 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c139b10 msgr2=0x7f9a2c072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c139b10 0x7f9a2c072600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0098a0 tx=0x7f9a1c01c400 comp rx=0 tx=0).stop 2024-09-11T02:50:28.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/318566105 shutdown_connections 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/318566105 >> v2:172.21.15.190:3300/0 conn(0x7f9a2c13a2d0 0x7f9a2c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a2c139ef0 0x7f9a2c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/318566105 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c139b10 0x7f9a2c072600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/318566105 >> 172.21.15.98:0/318566105 conn(0x7f9a2c0ac790 msgr2=0x7f9a2c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/318566105 shutdown_connections 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/318566105 wait complete. 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: build_initial_monmap 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: monmap: 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:28.692 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:28.693 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 librados: starting msgr at 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 librados: starting objecter 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 Processor -- start 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- start start 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 librados: setting wanted keys 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 librados: calling monclient init 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: init 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: _add_conns ranks=[7,0,8,5,4,6,3,2,1] 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a2c139b10 0x7f9a2c1319a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): picked mon.noname-i con 0x7f9a2c139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2c139ef0 0x7f9a2c131ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): picked mon.noname-a con 0x7f9a2c139ef0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9a2c13a2d0 0x7f9a2c0a3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): picked mon.noname-e con 0x7f9a2c13a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2c143460 con 0x7f9a2c139ef0 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f9a2c0a88a0 con 0x7f9a2c139b10 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9a2c1435e0 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): authenticate will time out at 982.558327s 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2bfff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2c139ef0 msgr2=0x7f9a2c131ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2c139ef0 0x7f9a2c131ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9a2c13a2d0 0x7f9a2c0a3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9a2c13a2d0 0x7f9a2c0a3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55664/0 (socket says 172.21.15.98:55664) 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 -- 172.21.15.98:0/4078285973 learned_addr learned my addr 172.21.15.98:0/4078285973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): get_auth_request con 0x7f9a2c13a2d0 auth_method 0 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a30861640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a2c139b10 0x7f9a2c1319a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a30861640 10 monclient(hunting): get_auth_request con 0x7f9a2c139b10 auth_method 0 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a30861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a30861640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a30861640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a2c139b10 msgr2=0x7f9a2c1319a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a2c139b10 0x7f9a2c1319a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2c139ef0 msgr2=0x7f9a2c131ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a2c139ef0 0x7f9a2c131ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient: _finish_hunting 0 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 monclient: found mon.noname-e 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 -- 172.21.15.98:0/4078285973 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2c135980 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a31062640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9a2c13a2d0 0x7f9a2c0a3390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400cd80 tx=0x7f9a2400a650 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 client.?.objecter ms_handle_connect 0x7f9a2c13a2d0 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a2400f8f0 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: dump: 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.699 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: _finish_auth 0 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: _check_auth_tickets 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 5 monclient: authenticate success, global_id 4499 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a2400fad0 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: _renew_subs 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2c13b610 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a2400fc80 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a2a7fc640 4 set_mon_vals no callback set 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a297fa640 10 monclient: dump: 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: _renew_subs 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2c0a9f90 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 librados: init done 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a38d0 0x7f9a2c0a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.675+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a2c06ca10 con 0x7f9a2c0a38d0 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a38d0 0x7f9a2c0a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient: get_auth_request con 0x7f9a2c0a38d0 auth_method 0 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9a24020030 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 10 monclient(hunting): handle_auth_done global_id 4484 payload 824 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 1 -- 172.21.15.98:0/4078285973 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a2c0aa4e0 con 0x7f9a2c0a38d0 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a1803df40 0x7f9a18040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f9a2404f9f0 con 0x7f9a2c13a2d0 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 client.4499.objecter ms_dispatch 0x7f9a2c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 3 client.4499.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 3 client.4499.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a30861640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a38d0 0x7f9a2c0a3cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0a9a70 tx=0x7f9a1c0191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 20 client.4499.objecter dump_active .. 0 homeless 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 client.4499.objecter ms_handle_connect 0x7f9a2c0a38d0 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 client.4499.objecter resend_mon_ops 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a1c0b9030 con 0x7f9a2c0a38d0 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 monclient: dump: 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.705 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a2bfff640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a1803df40 0x7f9a18040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a2bfff640 10 monclient: get_auth_request con 0x7f9a1803df40 auth_method 0 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a1c0b1040 con 0x7f9a2c0a38d0 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a38d0 msgr2=0x7f9a2c0a3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a38d0 0x7f9a2c0a3cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0a9a70 tx=0x7f9a1c0191a0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a38d0 msgr2=0x7f9a2c0a3cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:28.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a38d0 0x7f9a2c0a3cb0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0a9a70 tx=0x7f9a1c0191a0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a297fa640 10 client.4499.objecter ms_handle_connect 0x7f9a1803df40 2024-09-11T02:50:28.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.679+0000 7f9a2bfff640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a1803df40 0x7f9a18040400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0a54c0 tx=0x7f9a20009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a32ac5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a41f0 0x7f9a2c135b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a32ac5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a2c06e870 con 0x7f9a2c0a41f0 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a41f0 0x7f9a2c135b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient: get_auth_request con 0x7f9a2c0a41f0 auth_method 0 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 1 -- 172.21.15.98:0/4078285973 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a2c06e680 con 0x7f9a2c0a41f0 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a297fa640 10 client.4499.objecter ms_handle_connect 0x7f9a2c0a41f0 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a30861640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a41f0 0x7f9a2c135b90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c01cca0 tx=0x7f9a1c0a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a297fa640 10 client.4499.objecter resend_mon_ops 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a1c0b9070 con 0x7f9a2c0a41f0 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.683+0000 7f9a297fa640 10 monclient: dump: 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.710 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9411 (secure 0 0 0) 0x7f9a1c0ab610 con 0x7f9a2c0a41f0 2024-09-11T02:50:28.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a297fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a41f0 msgr2=0x7f9a2c135b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a297fa640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a41f0 0x7f9a2c135b90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c01cca0 tx=0x7f9a1c0a8000 comp rx=0 tx=0).stop 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a297fa640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a41f0 msgr2=0x7f9a2c135b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a297fa640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a2c0a41f0 0x7f9a2c135b90 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c01cca0 tx=0x7f9a1c0a8000 comp rx=0 tx=0).stop 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:28.712 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 22, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 44467, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.713 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:28.714 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:28.715 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:28.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:28.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:28.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:28.721 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:28.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:28.727 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:28.730 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.731 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:28.732 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:28.733 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:28.734 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.735 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.736 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.737 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:28.737 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:28.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 10 librados: watch_flush enter 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 10 librados: watch_flush exit 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 20 client.4499.objecter shutdown clearing up homeless session... 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 10 client.4499.objecter successfully canceled tick 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a1803df40 msgr2=0x7f9a18040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a1803df40 0x7f9a18040400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0a54c0 tx=0x7f9a20009190 comp rx=0 tx=0).stop 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 10 monclient: shutdown 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9a2c13a2d0 msgr2=0x7f9a2c0a3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 --2- 172.21.15.98:0/4078285973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9a2c13a2d0 0x7f9a2c0a3390 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a2400cd80 tx=0x7f9a2400a650 comp rx=0 tx=0).stop 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a31062640 1 -- 172.21.15.98:0/4078285973 reap_dead start 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a31062640 1 -- 172.21.15.98:0/4078285973 reap_dead start 2024-09-11T02:50:28.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 shutdown_connections 2024-09-11T02:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 >> 172.21.15.98:0/4078285973 conn(0x7f9a2c0ac790 msgr2=0x7f9a2c0aa7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 shutdown_connections 2024-09-11T02:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 -- 172.21.15.98:0/4078285973 wait complete. 2024-09-11T02:50:28.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.687+0000 7f9a32ac5640 1 librados: shutdown 2024-09-11T02:50:28.742 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:50:28.742 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-09-11T02:50:28.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:50:28.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient: get_monmap_and_config 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient: build_initial_monmap 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient: monmap: 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:28.861546+0000 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:28.861546+0000 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:28.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:28.859 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 Processor -- start 2024-09-11T02:50:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 -- start start 2024-09-11T02:50:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient: init 2024-09-11T02:50:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient: _add_conns ranks=[9,10,8,4,1,5,3,0,2,6,7] 2024-09-11T02:50:28.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2558139b80 0x7f2558139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): picked mon.noname-a con 0x7f2558139b80 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f255806fe50 0x7f25580a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): picked mon.noname-f con 0x7f255806fe50 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2558072640 0x7f25580a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): picked mon.noname-e con 0x7f2558072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f25580719c0 con 0x7f2558139b80 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2558071c10 con 0x7f255806fe50 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f25580702d0 con 0x7f2558072640 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:28.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255cbce640 10 monclient(hunting): authenticate will time out at 982.741848s 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255659c640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f2558139b80 msgr2=0x7f2558139f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f255659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2558139b80 0x7f2558139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2558072640 0x7f25580a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2558072640 0x7f25580a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55678/0 (socket says 172.21.15.98:55678) 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 -- 172.21.15.98:0/461354612 learned_addr learned my addr 172.21.15.98:0/461354612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): get_auth_request con 0x7f2558072640 auth_method 0 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2555d9b640 1 --2- 172.21.15.98:0/461354612 >> v2:172.21.15.190:3300/0 conn(0x7f255806fe50 0x7f25580a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2555d9b640 10 monclient(hunting): get_auth_request con 0x7f255806fe50 auth_method 0 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2555d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2555d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2555d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient(hunting): handle_auth_done global_id 4508 payload 274 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 -- 172.21.15.98:0/461354612 >> v2:172.21.15.190:3300/0 conn(0x7f255806fe50 msgr2=0x7f25580a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 --2- 172.21.15.98:0/461354612 >> v2:172.21.15.190:3300/0 conn(0x7f255806fe50 0x7f25580a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 -- 172.21.15.98:0/461354612 >> v2:172.21.15.98:3300/0 conn(0x7f2558139b80 msgr2=0x7f2558139f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 --2- 172.21.15.98:0/461354612 >> v2:172.21.15.98:3300/0 conn(0x7f2558139b80 0x7f2558139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 monclient: found mon.noname-e 2024-09-11T02:50:28.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:28.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:28.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.859+0000 7f2556d9d640 1 -- 172.21.15.98:0/461354612 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2558071370 con 0x7f2558072640 2024-09-11T02:50:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2556d9d640 1 --2- 172.21.15.98:0/461354612 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2558072640 0x7f25580a7f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f25580704e0 tx=0x7f255001c8d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=8215bb7939396cf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 1 -- 172.21.15.98:0/461354612 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25500090d0 con 0x7f2558072640 2024-09-11T02:50:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: dump: 2024-09-11T02:50:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.872 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.873 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: _finish_auth 0 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 1 -- 172.21.15.98:0/461354612 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25500a5ad0 con 0x7f2558072640 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 5 monclient: authenticate success, global_id 4508 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 1 -- 172.21.15.98:0/461354612 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25500a5cc0 con 0x7f2558072640 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255559a640 10 monclient: dump: 2024-09-11T02:50:28.874 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.875 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 4 set_mon_vals no callback set 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient: shutdown 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- 172.21.15.98:0/461354612 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2558072640 msgr2=0x7f25580a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 --2- 172.21.15.98:0/461354612 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2558072640 0x7f25580a7f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f25580704e0 tx=0x7f255001c8d0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- 172.21.15.98:0/461354612 shutdown_connections 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 --2- 172.21.15.98:0/461354612 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2558072640 0x7f25580a7f50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 --2- 172.21.15.98:0/461354612 >> v2:172.21.15.190:3300/0 conn(0x7f255806fe50 0x7f25580a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 --2- 172.21.15.98:0/461354612 >> v2:172.21.15.98:3300/0 conn(0x7f2558139b80 0x7f2558139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- 172.21.15.98:0/461354612 >> 172.21.15.98:0/461354612 conn(0x7f25580ac730 msgr2=0x7f25580a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- 172.21.15.98:0/461354612 shutdown_connections 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- 172.21.15.98:0/461354612 wait complete. 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient: build_initial_monmap 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient: monmap: 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 librados: starting msgr at 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 librados: starting objecter 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 Processor -- start 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- start start 2024-09-11T02:50:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 librados: setting wanted keys 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 librados: calling monclient init 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient: init 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient: _add_conns ranks=[3,7,0,5,6,8,2,1,4] 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f255806fe50 0x7f25581d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): picked mon.noname-g con 0x7f255806fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2558072640 0x7f25581d1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): picked mon.noname-i con 0x7f2558072640 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2558139b80 0x7f255813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): picked mon.noname-a con 0x7f2558139b80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25581435c0 con 0x7f2558139b80 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2558143440 con 0x7f255806fe50 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f2558143740 con 0x7f2558072640 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255cbce640 10 monclient(hunting): authenticate will time out at 982.747333s 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2556d9d640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2558139b80 msgr2=0x7f255813a6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2556d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2558139b80 0x7f255813a6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2558072640 0x7f25581d1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2558072640 0x7f25581d1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:47362/0 (socket says 172.21.15.98:47362) 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f255806fe50 0x7f25581d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 1 -- 172.21.15.98:0/3630044784 learned_addr learned my addr 172.21.15.98:0/3630044784 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 10 monclient(hunting): get_auth_request con 0x7f2558072640 auth_method 0 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 10 monclient(hunting): get_auth_request con 0x7f255806fe50 auth_method 0 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f2555d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.863+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f255806fe50 msgr2=0x7f25581d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f255806fe50 0x7f25581d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2558139b80 msgr2=0x7f255813a6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2558139b80 0x7f255813a6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 1 monclient: found mon.noname-i 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 1 -- 172.21.15.98:0/3630044784 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255813def0 con 0x7f2558072640 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f255806fe50 0x7f25581d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2555d9b640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2558072640 0x7f25581d1e30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f254c00d800 tx=0x7f254c00dcc0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 client.?.objecter ms_handle_connect 0x7f2558072640 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f254c00bb30 con 0x7f2558072640 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: dump: 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 5 monclient: authenticate success, global_id 4408 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 10 monclient: _renew_subs 2024-09-11T02:50:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255813d650 con 0x7f2558072640 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 10 monclient: _renew_subs 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f255813c6f0 con 0x7f2558072640 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 1 librados: init done 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f254c019070 con 0x7f2558072640 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f254c00bcd0 con 0x7f2558072640 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2554d99640 4 set_mon_vals no callback set 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:50:28.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: dump: 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255813e8d0 0x7f255813b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f255806ca10 con 0x7f255813e8d0 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255813e8d0 0x7f255813b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient: get_auth_request con 0x7f255813e8d0 auth_method 0 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 10 monclient(hunting): handle_auth_done global_id 4468 payload 824 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 1 -- 172.21.15.98:0/3630044784 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2558140440 con 0x7f255813e8d0 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 client.4408.objecter ms_handle_connect 0x7f255813e8d0 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 client.4408.objecter resend_mon_ops 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f255659c640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255813e8d0 0x7f255813b3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2548005280 tx=0x7f2548005740 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.7 v2:172.21.15.190:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f254c013070 con 0x7f2558072640 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f253803df40 0x7f2538040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.7 v2:172.21.15.190:3303/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f254c00f040 con 0x7f2558072640 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 client.4408.objecter ms_dispatch 0x7f2558000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:28.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 3 client.4408.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 3 client.4408.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 20 client.4408.objecter dump_active .. 0 homeless 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2548016070 con 0x7f255813e8d0 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: dump: 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.889 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f254800ea30 con 0x7f255813e8d0 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255813e8d0 msgr2=0x7f255813b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255813e8d0 0x7f255813b3b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2548005280 tx=0x7f2548005740 comp rx=0 tx=0).stop 2024-09-11T02:50:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255813e8d0 msgr2=0x7f255813b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255813e8d0 0x7f255813b3b0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2548005280 tx=0x7f2548005740 comp rx=0 tx=0).stop 2024-09-11T02:50:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2556d9d640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f253803df40 0x7f2538040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2556d9d640 10 monclient: get_auth_request con 0x7f253803df40 auth_method 0 2024-09-11T02:50:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2546ffd640 10 client.4408.objecter ms_handle_connect 0x7f253803df40 2024-09-11T02:50:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.867+0000 7f2556d9d640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f253803df40 0x7f2538040400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f25500ae000 tx=0x7f25500a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255802d450 0x7f255802d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f255806e870 con 0x7f255802d450 2024-09-11T02:50:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255802d450 0x7f255802d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient: get_auth_request con 0x7f255802d450 auth_method 0 2024-09-11T02:50:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 1 -- 172.21.15.98:0/3630044784 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f25580a2570 con 0x7f255802d450 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 10 client.4408.objecter ms_handle_connect 0x7f255802d450 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 10 client.4408.objecter resend_mon_ops 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255659c640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255802d450 0x7f255802d860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f25480059b0 tx=0x7f25480059e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2548016040 con 0x7f255802d450 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 10 monclient: dump: 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9244 (secure 0 0 0) 0x7f254800ee80 con 0x7f255802d450 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255802d450 msgr2=0x7f255802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255802d450 0x7f255802d860 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f25480059b0 tx=0x7f25480059e0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255802d450 msgr2=0x7f255802d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2546ffd640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f255802d450 0x7f255802d860 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f25480059b0 tx=0x7f25480059e0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 22, 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 44659, 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:28.897 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:28.898 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:28.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:28.900 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:28.901 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.902 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:28.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:28.909 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:28.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:28.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:28.916 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.918 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:28.919 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:28.920 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:28.920 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:28.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:28.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:28.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:28.920 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:28.920 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 10 librados: watch_flush enter 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 10 librados: watch_flush exit 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 20 client.4408.objecter shutdown clearing up homeless session... 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 10 client.4408.objecter successfully canceled tick 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f253803df40 msgr2=0x7f2538040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f253803df40 0x7f2538040400 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f25500ae000 tx=0x7f25500a7070 comp rx=0 tx=0).stop 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 10 monclient: shutdown 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2558072640 msgr2=0x7f25581d1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:28.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 --2- 172.21.15.98:0/3630044784 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2558072640 0x7f25581d1e30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f254c00d800 tx=0x7f254c00dcc0 comp rx=0 tx=0).stop 2024-09-11T02:50:28.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2556d9d640 1 -- 172.21.15.98:0/3630044784 reap_dead start 2024-09-11T02:50:28.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f2556d9d640 1 -- 172.21.15.98:0/3630044784 reap_dead start 2024-09-11T02:50:28.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 shutdown_connections 2024-09-11T02:50:28.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 >> 172.21.15.98:0/3630044784 conn(0x7f25580ac730 msgr2=0x7f2558138140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:28.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 shutdown_connections 2024-09-11T02:50:28.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 -- 172.21.15.98:0/3630044784 wait complete. 2024-09-11T02:50:28.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:28.875+0000 7f255cbce640 1 librados: shutdown 2024-09-11T02:50:28.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:50:28.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-09-11T02:50:28.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:50:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient: get_monmap_and_config 2024-09-11T02:50:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient: monmap: 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:29.045522+0000 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:29.045522+0000 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.042 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:29.043 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:29.043 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 1 Processor -- start 2024-09-11T02:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 1 -- start start 2024-09-11T02:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient: init 2024-09-11T02:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient: _add_conns ranks=[0,2,6,5,9,3,8,7,4,10,1] 2024-09-11T02:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa02c0a3de0 addr v1:172.21.15.98:6789/0 2024-09-11T02:50:29.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c0a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa02c0a48d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c13c9f0 0x7fa02c13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient(hunting): picked mon.noname-d con 0x7fa02c13c9f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa02c068fa0 con 0x7fa02c0a3de0 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa02c0a0170 con 0x7fa02c0a48d0 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03106b640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fa02c0a3de0 legacy=0x7fa02c0a41c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa02c068b40 con 0x7fa02c13c9f0 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa0332cf640 10 monclient(hunting): authenticate will time out at 982.925825s 2024-09-11T02:50:29.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03186c640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c13c9f0 msgr2=0x7fa02c13ede0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03186c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c13c9f0 0x7fa02c13ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c0a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c0a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:44398/0 (socket says 172.21.15.98:44398) 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 -- 172.21.15.98:0/3731388574 learned_addr learned my addr 172.21.15.98:0/3731388574 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): get_auth_request con 0x7fa02c0a48d0 auth_method 0 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient(hunting): handle_auth_done global_id 4502 payload 274 2024-09-11T02:50:29.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 -- 172.21.15.98:0/3731388574 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c13c9f0 msgr2=0x7fa02c13ede0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 --2- 172.21.15.98:0/3731388574 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c13c9f0 0x7fa02c13ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 -- 172.21.15.98:0/3731388574 >> v1:172.21.15.98:6789/0 conn(0x7fa02c0a3de0 legacy=0x7fa02c0a41c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 monclient: found mon.noname-b 2024-09-11T02:50:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:29.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.043+0000 7fa03086a640 1 -- 172.21.15.98:0/3731388574 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02c09eaf0 con 0x7fa02c0a48d0 2024-09-11T02:50:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa03086a640 1 --2- 172.21.15.98:0/3731388574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c0a0570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa028009eb0 tx=0x7fa02801c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e268ee5da45279e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 1 -- 172.21.15.98:0/3731388574 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0280090d0 con 0x7fa02c0a48d0 2024-09-11T02:50:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: dump: 2024-09-11T02:50:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.056 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 5 monclient: authenticate success, global_id 4502 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 1 -- 172.21.15.98:0/3731388574 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0280a5a70 con 0x7fa02c0a48d0 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 4 set_mon_vals no callback set 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 1 -- 172.21.15.98:0/3731388574 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0280a5c60 con 0x7fa02c0a48d0 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa023fff640 10 monclient: dump: 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 10 monclient: shutdown 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 -- 172.21.15.98:0/3731388574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 msgr2=0x7fa02c0a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 --2- 172.21.15.98:0/3731388574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c0a0570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa028009eb0 tx=0x7fa02801c910 comp rx=0 tx=0).stop 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 -- 172.21.15.98:0/3731388574 shutdown_connections 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 --2- 172.21.15.98:0/3731388574 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c13c9f0 0x7fa02c13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 --2- 172.21.15.98:0/3731388574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c0a0570 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 -- 172.21.15.98:0/3731388574 >> 172.21.15.98:0/3731388574 conn(0x7fa02c0a4ed0 msgr2=0x7fa02c0a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 -- 172.21.15.98:0/3731388574 shutdown_connections 2024-09-11T02:50:29.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 -- 172.21.15.98:0/3731388574 wait complete. 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 10 monclient: monmap: 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.060 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 librados: starting msgr at 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 librados: starting objecter 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 Processor -- start 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 -- start start 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 librados: setting wanted keys 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 1 librados: calling monclient init 2024-09-11T02:50:29.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.047+0000 7fa0332cf640 10 monclient: init 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient: _add_conns ranks=[6,2,1,8,5,7,4,3,0] 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c0a3de0 0x7fa02c12d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): picked mon.noname-d con 0x7fa02c0a3de0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c12d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa02c0a48d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa02c13c9f0 0x7fa02c12df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): picked mon.noname-f con 0x7fa02c13c9f0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c141ed0 con 0x7fa02c13c9f0 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa02c141d50 con 0x7fa02c0a48d0 2024-09-11T02:50:29.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c0a3de0 msgr2=0x7fa02c12d490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa02c142050 con 0x7fa02c0a3de0 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): authenticate will time out at 982.932346s 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c0a3de0 0x7fa02c12d490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c12d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c12d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:44408/0 (socket says 172.21.15.98:44408) 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 -- 172.21.15.98:0/2916402523 learned_addr learned my addr 172.21.15.98:0/2916402523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): get_auth_request con 0x7fa02c0a48d0 auth_method 0 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa02c13c9f0 0x7fa02c12df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 10 monclient(hunting): get_auth_request con 0x7fa02c13c9f0 auth_method 0 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient(hunting): handle_auth_done global_id 4511 payload 824 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c0a3de0 msgr2=0x7fa02c12d490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa02c0a3de0 0x7fa02c12d490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa02c13c9f0 msgr2=0x7fa02c12df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa02c13c9f0 0x7fa02c12df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 monclient: found mon.noname-b 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 -- 172.21.15.98:0/2916402523 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa02c130160 con 0x7fa02c0a48d0 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03086a640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c12d9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa028009e20 tx=0x7fa0280196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 client.?.objecter ms_handle_connect 0x7fa02c0a48d0 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0280bc020 con 0x7fa02c0a48d0 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: dump: 2024-09-11T02:50:29.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.066 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 5 monclient: authenticate success, global_id 4511 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa028003bb0 con 0x7fa02c0a48d0 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient: _renew_subs 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa02c0a1260 con 0x7fa02c0a48d0 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa028003d60 con 0x7fa02c0a48d0 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0237fe640 4 set_mon_vals no callback set 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: dump: 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.068 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient: _renew_subs 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa02c12fe80 con 0x7fa02c0a48d0 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 librados: init done 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b0b0 0x7fa02c09b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa02c06ca10 con 0x7fa02c09b0b0 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa028004040 con 0x7fa02c0a48d0 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b0b0 0x7fa02c09b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient: get_auth_request con 0x7fa02c09b0b0 auth_method 0 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa01803df40 0x7fa018040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fa0280ea410 con 0x7fa02c0a48d0 2024-09-11T02:50:29.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 client.4511.objecter ms_dispatch 0x7fa02c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 3 client.4511.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 3 client.4511.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 10 monclient(hunting): handle_auth_done global_id 4517 payload 824 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 1 -- 172.21.15.98:0/2916402523 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa02c0a1620 con 0x7fa02c09b0b0 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 20 client.4511.objecter dump_active .. 0 homeless 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa01803df40 0x7fa018040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 10 monclient: get_auth_request con 0x7fa01803df40 auth_method 0 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03106b640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b0b0 0x7fa02c09b490 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa024003990 tx=0x7fa02400b810 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 client.4511.objecter ms_handle_connect 0x7fa02c09b0b0 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 client.4511.objecter resend_mon_ops 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa024016070 con 0x7fa02c09b0b0 2024-09-11T02:50:29.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: dump: 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.072 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 client.4511.objecter ms_handle_connect 0x7fa01803df40 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa03186c640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa01803df40 0x7fa018040400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa01c003940 tx=0x7fa01c006e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa024003bd0 con 0x7fa02c09b0b0 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b0b0 msgr2=0x7fa02c09b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b0b0 0x7fa02c09b490 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa024003990 tx=0x7fa02400b810 comp rx=0 tx=0).stop 2024-09-11T02:50:29.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b0b0 msgr2=0x7fa02c09b490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.051+0000 7fa021ffb640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b0b0 0x7fa02c09b490 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa024003990 tx=0x7fa02400b810 comp rx=0 tx=0).stop 2024-09-11T02:50:29.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b9d0 0x7fa02c1387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa02c06e870 con 0x7fa02c09b9d0 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b9d0 0x7fa02c1387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient: get_auth_request con 0x7fa02c09b9d0 auth_method 0 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 10 monclient(hunting): handle_auth_done global_id 4526 payload 824 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 1 -- 172.21.15.98:0/2916402523 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa02c0a0570 con 0x7fa02c09b9d0 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03106b640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b9d0 0x7fa02c1387c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa024003990 tx=0x7fa024003800 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 10 client.4511.objecter ms_handle_connect 0x7fa02c09b9d0 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 10 client.4511.objecter resend_mon_ops 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa024016030 con 0x7fa02c09b9d0 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 10 monclient: dump: 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9411 (secure 0 0 0) 0x7fa024019940 con 0x7fa02c09b9d0 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b9d0 msgr2=0x7fa02c1387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b9d0 0x7fa02c1387c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa024003990 tx=0x7fa024003800 comp rx=0 tx=0).stop 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b9d0 msgr2=0x7fa02c1387c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa021ffb640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa02c09b9d0 0x7fa02c1387c0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa024003990 tx=0x7fa024003800 comp rx=0 tx=0).stop 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:29.079 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 22, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 44851, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.081 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.082 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.083 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.084 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.085 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:29.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.090 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.092 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.094 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:29.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.101 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:29.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:29.103 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 10 librados: watch_flush enter 2024-09-11T02:50:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 10 librados: watch_flush exit 2024-09-11T02:50:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 20 client.4511.objecter shutdown clearing up homeless session... 2024-09-11T02:50:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 10 client.4511.objecter successfully canceled tick 2024-09-11T02:50:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa01803df40 msgr2=0x7fa018040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa01803df40 0x7fa018040400 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa01c003940 tx=0x7fa01c006e10 comp rx=0 tx=0).stop 2024-09-11T02:50:29.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 10 monclient: shutdown 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 msgr2=0x7fa02c12d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 --2- 172.21.15.98:0/2916402523 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02c0a48d0 0x7fa02c12d9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa028009e20 tx=0x7fa0280196b0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03186c640 1 -- 172.21.15.98:0/2916402523 reap_dead start 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa03186c640 1 -- 172.21.15.98:0/2916402523 reap_dead start 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 shutdown_connections 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 >> 172.21.15.98:0/2916402523 conn(0x7fa02c0a4ed0 msgr2=0x7fa02c13cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 shutdown_connections 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 -- 172.21.15.98:0/2916402523 wait complete. 2024-09-11T02:50:29.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.059+0000 7fa0332cf640 1 librados: shutdown 2024-09-11T02:50:29.107 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:50:29.107 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-09-11T02:50:29.107 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:50:29.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.227+0000 7f28a14c1640 10 monclient: get_monmap_and_config 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.227+0000 7f28a14c1640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.227+0000 7f28a14c1640 10 monclient: monmap: 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:29.233010+0000 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:29.233010+0000 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.230 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:29.231 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:29.232 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 Processor -- start 2024-09-11T02:50:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 -- start start 2024-09-11T02:50:29.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient: init 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient: _add_conns ranks=[3,4,9,1,6,5,2,10,7,0,8] 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f289c139b80 0x7f289c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): picked mon.noname-g con 0x7f289c139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c06fe50 0x7f289c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): picked mon.noname-c con 0x7f289c06fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f289c072640 0x7f289c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): picked mon.noname-a con 0x7f289c072640 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f289c0719c0 con 0x7f289c072640 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f289c071c10 con 0x7f289c139b80 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f289c0702d0 con 0x7f289c06fe50 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f28a14c1640 10 monclient(hunting): authenticate will time out at 983.113255s 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289b7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f289c072640 msgr2=0x7f289c0a7f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289b7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f289c072640 0x7f289c0a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c06fe50 0x7f289c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c06fe50 0x7f289c0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:36698/0 (socket says 172.21.15.98:36698) 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 -- 172.21.15.98:0/3473052484 learned_addr learned my addr 172.21.15.98:0/3473052484 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): get_auth_request con 0x7f289c06fe50 auth_method 0 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289affd640 1 --2- 172.21.15.98:0/3473052484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f289c139b80 0x7f289c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient(hunting): handle_auth_done global_id 4486 payload 274 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 -- 172.21.15.98:0/3473052484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f289c139b80 msgr2=0x7f289c139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 --2- 172.21.15.98:0/3473052484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f289c139b80 0x7f289c139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 -- 172.21.15.98:0/3473052484 >> v2:172.21.15.98:3300/0 conn(0x7f289c072640 msgr2=0x7f289c0a7f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 --2- 172.21.15.98:0/3473052484 >> v2:172.21.15.98:3300/0 conn(0x7f289c072640 0x7f289c0a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 monclient: found mon.noname-c 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 -- 172.21.15.98:0/3473052484 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289c071370 con 0x7f289c06fe50 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.231+0000 7f289a7fc640 1 --2- 172.21.15.98:0/3473052484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c06fe50 0x7f289c0a7a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28840098a0 tx=0x7f288401c420 comp rx=0 tx=0).ready entity=mon.4 client_cookie=54cc0b30ecb6acbe server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 1 -- 172.21.15.98:0/3473052484 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f28840b0070 con 0x7f289c06fe50 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: dump: 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.245 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 1 -- 172.21.15.98:0/3473052484 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28840aa810 con 0x7f289c06fe50 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 5 monclient: authenticate success, global_id 4486 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 1 -- 172.21.15.98:0/3473052484 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f28840a9300 con 0x7f289c06fe50 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 4 set_mon_vals no callback set 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f2899ffb640 10 monclient: dump: 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.247 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.248 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: shutdown 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- 172.21.15.98:0/3473052484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c06fe50 msgr2=0x7f289c0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- 172.21.15.98:0/3473052484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c06fe50 0x7f289c0a7a10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f28840098a0 tx=0x7f288401c420 comp rx=0 tx=0).stop 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- 172.21.15.98:0/3473052484 shutdown_connections 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- 172.21.15.98:0/3473052484 >> v2:172.21.15.98:3300/0 conn(0x7f289c072640 0x7f289c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- 172.21.15.98:0/3473052484 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c06fe50 0x7f289c0a7a10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- 172.21.15.98:0/3473052484 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f289c139b80 0x7f289c139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- 172.21.15.98:0/3473052484 >> 172.21.15.98:0/3473052484 conn(0x7f289c0ac730 msgr2=0x7f289c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- 172.21.15.98:0/3473052484 shutdown_connections 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- 172.21.15.98:0/3473052484 wait complete. 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: monmap: 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:29.250 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:29.251 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.251 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.251 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.251 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.251 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.251 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.251 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 librados: starting msgr at 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 librados: starting objecter 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 Processor -- start 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- start start 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 librados: setting wanted keys 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 librados: calling monclient init 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: init 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: _add_conns ranks=[0,5,4,8,1,7,2,3,6] 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f289c06fe50 0x7f289c13f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): picked mon.noname-a con 0x7f289c06fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f289c072640 0x7f289c13fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): picked mon.noname-h con 0x7f289c072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c139b80 0x7f289c13a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): picked mon.noname-c con 0x7f289c139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f289c1435c0 con 0x7f289c06fe50 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f289c143440 con 0x7f289c139b80 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f289c143740 con 0x7f289c072640 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient(hunting): authenticate will time out at 983.118305s 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289affd640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f289c06fe50 msgr2=0x7f289c13f9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f289c06fe50 0x7f289c13f9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c139b80 0x7f289c13a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c139b80 0x7f289c13a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:36712/0 (socket says 172.21.15.98:36712) 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 -- 172.21.15.98:0/2374723114 learned_addr learned my addr 172.21.15.98:0/2374723114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): get_auth_request con 0x7f289c139b80 auth_method 0 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f289c072640 0x7f289c13fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 10 monclient(hunting): get_auth_request con 0x7f289c072640 auth_method 0 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f289c072640 msgr2=0x7f289c13fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f289c072640 0x7f289c13fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f289c06fe50 msgr2=0x7f289c13f9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f289c06fe50 0x7f289c13f9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 monclient: found mon.noname-c 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 -- 172.21.15.98:0/2374723114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289c140420 con 0x7f289c139b80 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f289b7fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c139b80 0x7f289c13a6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f288c00be90 tx=0x7f288c00bf80 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 client.?.objecter ms_handle_connect 0x7f289c139b80 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f288c00cd20 con 0x7f289c139b80 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: dump: 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.256 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.257 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f288c010030 con 0x7f289c139b80 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 5 monclient: authenticate success, global_id 4495 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: _renew_subs 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f289c0aa090 con 0x7f289c139b80 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f288c015cc0 con 0x7f289c139b80 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:29.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28937fe640 10 monclient: dump: 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.259 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: _renew_subs 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28997fa640 4 set_mon_vals no callback set 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f289c0a5960 con 0x7f289c139b80 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 librados: init done 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.235+0000 7f28a14c1640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c13abf0 0x7f289c13afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f289c06ca10 con 0x7f289c13abf0 2024-09-11T02:50:29.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f288c008070 con 0x7f289c139b80 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c13abf0 0x7f289c13afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient: get_auth_request con 0x7f289c13abf0 auth_method 0 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f287c03df40 0x7f287c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f288c04f150 con 0x7f289c139b80 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 client.4495.objecter ms_dispatch 0x7f289c000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 3 client.4495.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:29.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 3 client.4495.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 10 monclient(hunting): handle_auth_done global_id 4492 payload 824 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 20 client.4495.objecter dump_active .. 0 homeless 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 1 -- 172.21.15.98:0/2374723114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f289c13b510 con 0x7f289c13abf0 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289a7fc640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f287c03df40 0x7f287c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289a7fc640 10 monclient: get_auth_request con 0x7f287c03df40 auth_method 0 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 client.4495.objecter ms_handle_connect 0x7f289c13abf0 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 client.4495.objecter resend_mon_ops 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289affd640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c13abf0 0x7f289c13afd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f289400c4c0 tx=0x7f289400c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f28940044a0 con 0x7f289c13abf0 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 monclient: dump: 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.262 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.263 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 client.4495.objecter ms_handle_connect 0x7f287c03df40 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f289a7fc640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f287c03df40 0x7f287c040400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28840ad000 tx=0x7f28840a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2894004030 con 0x7f289c13abf0 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c13abf0 msgr2=0x7f289c13afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c13abf0 0x7f289c13afd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f289400c4c0 tx=0x7f289400c980 comp rx=0 tx=0).stop 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c13abf0 msgr2=0x7f289c13afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.239+0000 7f28937fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c13abf0 0x7f289c13afd0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f289400c4c0 tx=0x7f289400c980 comp rx=0 tx=0).stop 2024-09-11T02:50:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.247+0000 7f28a14c1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.247+0000 7f28a14c1640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c0a0e60 0x7f289c0a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.247+0000 7f28a14c1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.247+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f289c06e870 con 0x7f289c0a0e60 2024-09-11T02:50:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c0a0e60 0x7f289c0a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient: get_auth_request con 0x7f289c0a0e60 auth_method 0 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 10 monclient(hunting): handle_auth_done global_id 4501 payload 824 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 1 -- 172.21.15.98:0/2374723114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f289c139f60 con 0x7f289c0a0e60 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 10 client.4495.objecter ms_handle_connect 0x7f289c0a0e60 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 10 client.4495.objecter resend_mon_ops 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289affd640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c0a0e60 0x7f289c0a1270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2894016000 tx=0x7f2894004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2894011c80 con 0x7f289c0a0e60 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 10 monclient: dump: 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.268 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9413 (secure 0 0 0) 0x7f2894011810 con 0x7f289c0a0e60 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c0a0e60 msgr2=0x7f289c0a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c0a0e60 0x7f289c0a1270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2894016000 tx=0x7f2894004740 comp rx=0 tx=0).stop 2024-09-11T02:50:29.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c0a0e60 msgr2=0x7f289c0a1270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28937fe640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f289c0a0e60 0x7f289c0a1270 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2894016000 tx=0x7f2894004740 comp rx=0 tx=0).stop 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 22, 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 45019, 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:29.270 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.271 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:29.272 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:29.274 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.275 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:29.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.279 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.281 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:29.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:29.283 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.284 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.286 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.288 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.290 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.292 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.293 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:29.294 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 10 librados: watch_flush enter 2024-09-11T02:50:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 10 librados: watch_flush exit 2024-09-11T02:50:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 20 client.4495.objecter shutdown clearing up homeless session... 2024-09-11T02:50:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 10 client.4495.objecter successfully canceled tick 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f287c03df40 msgr2=0x7f287c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f287c03df40 0x7f287c040400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f28840ad000 tx=0x7f28840a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 10 monclient: shutdown 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c139b80 msgr2=0x7f289c13a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c139b80 0x7f289c13a6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f288c00be90 tx=0x7f288c00bf80 comp rx=0 tx=0).stop 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f289b7fe640 1 -- 172.21.15.98:0/2374723114 reap_dead start 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 shutdown_connections 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 --2- 172.21.15.98:0/2374723114 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f289c139b80 0x7f289c13a6b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 >> 172.21.15.98:0/2374723114 conn(0x7f289c0ac730 msgr2=0x7f289c0a6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 shutdown_connections 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 -- 172.21.15.98:0/2374723114 wait complete. 2024-09-11T02:50:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.251+0000 7f28a14c1640 1 librados: shutdown 2024-09-11T02:50:29.298 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:50:29.298 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon g 2024-09-11T02:50:29.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:50:29.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient: get_monmap_and_config 2024-09-11T02:50:29.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient: monmap: 2024-09-11T02:50:29.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.418 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:29.421566+0000 2024-09-11T02:50:29.418 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:29.421566+0000 2024-09-11T02:50:29.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:29.419 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:29.420 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 1 Processor -- start 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 1 -- start start 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient: init 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient: _add_conns ranks=[10,1,4,7,6,8,0,9,2,3,5] 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7f60139b80 0x7f7f60139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient(hunting): picked mon.noname-f con 0x7f7f60139b80 addr v2:172.21.15.190:3300/0 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7f6006fe50 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f60072640 0x7f7f600a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient(hunting): picked mon.noname-c con 0x7f7f60072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f60071370 con 0x7f7f6006fe50 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7f600719c0 con 0x7f7f60139b80 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7f60071c10 con 0x7f7f60072640 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f64d14640 10 monclient(hunting): authenticate will time out at 983.301786s 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f60072640 0x7f7f600a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f60072640 0x7f7f600a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:36728/0 (socket says 172.21.15.98:36728) 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 -- 172.21.15.98:0/3039066948 learned_addr learned my addr 172.21.15.98:0/3039066948 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): get_auth_request con 0x7f7f60072640 auth_method 0 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 1 --2- 172.21.15.98:0/3039066948 >> v2:172.21.15.190:3300/0 conn(0x7f7f60139b80 0x7f7f60139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 10 monclient(hunting): get_auth_request con 0x7f7f60139b80 auth_method 0 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient(hunting): handle_auth_done global_id 4504 payload 274 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 -- 172.21.15.98:0/3039066948 >> v2:172.21.15.190:3300/0 conn(0x7f7f60139b80 msgr2=0x7f7f60139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 --2- 172.21.15.98:0/3039066948 >> v2:172.21.15.190:3300/0 conn(0x7f7f60139b80 0x7f7f60139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 -- 172.21.15.98:0/3039066948 >> v1:172.21.15.190:6789/0 conn(0x7f7f6006fe50 legacy=0x7f7f600a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 monclient: found mon.noname-c 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.419+0000 7f7f5ed9d640 1 -- 172.21.15.98:0/3039066948 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f600a9600 con 0x7f7f60072640 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5ed9d640 1 --2- 172.21.15.98:0/3039066948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f60072640 0x7f7f600a8120 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009870 tx=0x7f7f4801c020 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2e7d440e0a098908 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 1 -- 172.21.15.98:0/3039066948 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f480af870 con 0x7f7f60072640 2024-09-11T02:50:29.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: dump: 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.435 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 1 -- 172.21.15.98:0/3039066948 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f480afa30 con 0x7f7f60072640 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 1 -- 172.21.15.98:0/3039066948 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f480afc20 con 0x7f7f60072640 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5d59a640 10 monclient: dump: 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.437 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 5 monclient: authenticate success, global_id 4504 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 4 set_mon_vals no callback set 2024-09-11T02:50:29.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient: shutdown 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- 172.21.15.98:0/3039066948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f60072640 msgr2=0x7f7f600a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 --2- 172.21.15.98:0/3039066948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f60072640 0x7f7f600a8120 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009870 tx=0x7f7f4801c020 comp rx=0 tx=0).stop 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- 172.21.15.98:0/3039066948 shutdown_connections 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 --2- 172.21.15.98:0/3039066948 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f60072640 0x7f7f600a8120 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 --2- 172.21.15.98:0/3039066948 >> v2:172.21.15.190:3300/0 conn(0x7f7f60139b80 0x7f7f60139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- 172.21.15.98:0/3039066948 >> 172.21.15.98:0/3039066948 conn(0x7f7f600ac730 msgr2=0x7f7f600a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- 172.21.15.98:0/3039066948 shutdown_connections 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- 172.21.15.98:0/3039066948 wait complete. 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient: monmap: 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 librados: starting msgr at 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 librados: starting objecter 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 Processor -- start 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- start start 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 librados: setting wanted keys 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 librados: calling monclient init 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient: init 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient: _add_conns ranks=[3,1,0,6,8,5,2,7,4] 2024-09-11T02:50:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f6006fe50 0x7f7f6013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): picked mon.noname-g con 0x7f7f6006fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f60072640 0x7f7f6013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): picked mon.noname-f con 0x7f7f60072640 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f60139b80 0x7f7f6013b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): picked mon.noname-a con 0x7f7f60139b80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60144df0 con 0x7f7f60139b80 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7f60144c70 con 0x7f7f60072640 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7f600517e0 con 0x7f7f6006fe50 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f64d14640 10 monclient(hunting): authenticate will time out at 983.307296s 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5ed9d640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f60139b80 msgr2=0x7f7f6013b250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f60139b80 0x7f7f6013b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f6006fe50 0x7f7f6013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f60072640 0x7f7f6013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f60072640 0x7f7f6013ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:46148/0 (socket says 172.21.15.98:46148) 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f6006fe50 0x7f7f6013a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:54498/0 (socket says 172.21.15.98:54498) 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 -- 172.21.15.98:0/2323011360 learned_addr learned my addr 172.21.15.98:0/2323011360 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): get_auth_request con 0x7f7f60072640 auth_method 0 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 10 monclient(hunting): get_auth_request con 0x7f7f6006fe50 auth_method 0 2024-09-11T02:50:29.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient(hunting): handle_auth_done global_id 4528 payload 824 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f6006fe50 msgr2=0x7f7f6013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f6006fe50 0x7f7f6013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f60139b80 msgr2=0x7f7f6013b250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f60139b80 0x7f7f6013b250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 monclient: found mon.noname-f 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.423+0000 7f7f5dd9b640 1 -- 172.21.15.98:0/2323011360 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f600aa2c0 con 0x7f7f60072640 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5dd9b640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f60072640 0x7f7f6013ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800caa0 tx=0x7f7f5800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 client.?.objecter ms_handle_connect 0x7f7f60072640 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f58003e10 con 0x7f7f60072640 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: dump: 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.446 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 5 monclient: authenticate success, global_id 4528 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f5800a8c0 con 0x7f7f60072640 2024-09-11T02:50:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 10 monclient: _renew_subs 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f6013cd80 con 0x7f7f60072640 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f580050d0 con 0x7f7f60072640 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5cd99640 4 set_mon_vals no callback set 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: dump: 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 10 monclient: _renew_subs 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f6013d340 con 0x7f7f60072640 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 1 librados: init done 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60133760 0x7f7f60133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7f6006ca10 con 0x7f7f60133760 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60133760 0x7f7f60133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient: get_auth_request con 0x7f7f60133760 auth_method 0 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f7f580039a0 con 0x7f7f60072640 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 1 -- 172.21.15.98:0/2323011360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f600a9f00 con 0x7f7f60133760 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f4003df40 0x7f7f40040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f7f5804f130 con 0x7f7f60072640 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 client.4528.objecter ms_dispatch 0x7f7f60000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5e59c640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60133760 0x7f7f60133b40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7f50011900 tx=0x7f7f50011dc0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 3 client.4528.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 3 client.4528.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 20 client.4528.objecter dump_active .. 0 homeless 2024-09-11T02:50:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 client.4528.objecter ms_handle_connect 0x7f7f60133760 2024-09-11T02:50:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 client.4528.objecter resend_mon_ops 2024-09-11T02:50:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5ed9d640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f4003df40 0x7f7f40040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5ed9d640 10 monclient: get_auth_request con 0x7f7f4003df40 auth_method 0 2024-09-11T02:50:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f50019070 con 0x7f7f60133760 2024-09-11T02:50:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:50:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: dump: 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f50004300 con 0x7f7f60133760 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60133760 msgr2=0x7f7f60133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60133760 0x7f7f60133b40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7f50011900 tx=0x7f7f50011dc0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60133760 msgr2=0x7f7f60133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60133760 0x7f7f60133b40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f56ffd640 10 client.4528.objecter ms_handle_connect 0x7f7f4003df40 2024-09-11T02:50:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.427+0000 7f7f5ed9d640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f4003df40 0x7f7f40040400 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f480ae000 tx=0x7f7f480a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60134080 0x7f7f600a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7f6006e870 con 0x7f7f60134080 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60134080 0x7f7f600a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient: get_auth_request con 0x7f7f60134080 auth_method 0 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 10 monclient(hunting): handle_auth_done global_id 4476 payload 824 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 1 -- 172.21.15.98:0/2323011360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f60134890 con 0x7f7f60134080 2024-09-11T02:50:29.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5e59c640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60134080 0x7f7f600a0e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7f50000c10 tx=0x7f7f500046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 10 client.4528.objecter ms_handle_connect 0x7f7f60134080 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 10 client.4528.objecter resend_mon_ops 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f50019040 con 0x7f7f60134080 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 10 monclient: dump: 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.458 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9411 (secure 0 0 0) 0x7f7f5001e450 con 0x7f7f60134080 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60134080 msgr2=0x7f7f600a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60134080 0x7f7f600a0e60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7f50000c10 tx=0x7f7f500046d0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60134080 msgr2=0x7f7f600a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f56ffd640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7f60134080 0x7f7f600a0e60 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7f50000c10 tx=0x7f7f500046d0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 22, 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 45183, 2024-09-11T02:50:29.460 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:29.461 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:29.462 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:29.464 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.465 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.467 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:29.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.474 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:29.484 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 10 librados: watch_flush enter 2024-09-11T02:50:29.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 10 librados: watch_flush exit 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 20 client.4528.objecter shutdown clearing up homeless session... 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 10 client.4528.objecter successfully canceled tick 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f4003df40 msgr2=0x7f7f40040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f4003df40 0x7f7f40040400 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f480ae000 tx=0x7f7f480a6000 comp rx=0 tx=0).stop 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 10 monclient: shutdown 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f60072640 msgr2=0x7f7f6013ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 --2- 172.21.15.98:0/2323011360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f60072640 0x7f7f6013ad10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7f5800caa0 tx=0x7f7f5800cf60 comp rx=0 tx=0).stop 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5ed9d640 1 -- 172.21.15.98:0/2323011360 reap_dead start 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f5ed9d640 1 -- 172.21.15.98:0/2323011360 reap_dead start 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 shutdown_connections 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 >> 172.21.15.98:0/2323011360 conn(0x7f7f600ac730 msgr2=0x7f7f600a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 shutdown_connections 2024-09-11T02:50:29.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 -- 172.21.15.98:0/2323011360 wait complete. 2024-09-11T02:50:29.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.435+0000 7f7f64d14640 1 librados: shutdown 2024-09-11T02:50:29.487 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:50:29.487 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-09-11T02:50:29.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:50:29.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.603+0000 7f6207fab640 10 monclient: get_monmap_and_config 2024-09-11T02:50:29.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.603+0000 7f6207fab640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.603+0000 7f6207fab640 10 monclient: monmap: 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:29.609062+0000 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:29.609062+0000 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:29.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:29.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 Processor -- start 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 -- start start 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient: init 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient: _add_conns ranks=[9,8,3,4,7,6,5,2,1,0,10] 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f62000a46a0 0x7f62000a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): picked mon.noname-a con 0x7f62000a46a0 addr v2:172.21.15.98:3300/0 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f62000a4fc0 0x7f6200139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): picked mon.noname-e con 0x7f62000a4fc0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6200139e70 0x7f620013a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): picked mon.noname-g con 0x7f6200139e70 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f62000775b0 con 0x7f62000a46a0 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f62000a11d0 con 0x7f6200139e70 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f62000a0750 con 0x7f62000a4fc0 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6207fab640 10 monclient(hunting): authenticate will time out at 983.489326s 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205d47640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f62000a46a0 msgr2=0x7f62000a4a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:29.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205d47640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f62000a46a0 0x7f62000a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f62000a4fc0 0x7f6200139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f62000a4fc0 0x7f6200139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55708/0 (socket says 172.21.15.98:55708) 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 -- 172.21.15.98:0/1062842997 learned_addr learned my addr 172.21.15.98:0/1062842997 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): get_auth_request con 0x7f62000a4fc0 auth_method 0 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 1 --2- 172.21.15.98:0/1062842997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6200139e70 0x7f620013a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 10 monclient(hunting): get_auth_request con 0x7f6200139e70 auth_method 0 2024-09-11T02:50:29.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6206548640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient(hunting): handle_auth_done global_id 4535 payload 274 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 -- 172.21.15.98:0/1062842997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6200139e70 msgr2=0x7f620013a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 --2- 172.21.15.98:0/1062842997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6200139e70 0x7f620013a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 -- 172.21.15.98:0/1062842997 >> v2:172.21.15.98:3300/0 conn(0x7f62000a46a0 msgr2=0x7f62000a4a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:29.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 --2- 172.21.15.98:0/1062842997 >> v2:172.21.15.98:3300/0 conn(0x7f62000a46a0 0x7f62000a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 monclient: found mon.noname-e 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.607+0000 7f6205546640 1 -- 172.21.15.98:0/1062842997 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6200077160 con 0x7f62000a4fc0 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6205546640 1 --2- 172.21.15.98:0/1062842997 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f62000a4fc0 0x7f6200139930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009870 tx=0x7f61fc01c560 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a00ba2ce37af2ae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 1 -- 172.21.15.98:0/1062842997 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61fc0aa8d0 con 0x7f62000a4fc0 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: dump: 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.616 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.617 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 1 -- 172.21.15.98:0/1062842997 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc0af070 con 0x7f62000a4fc0 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 5 monclient: authenticate success, global_id 4535 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 1 -- 172.21.15.98:0/1062842997 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61fc0a93c0 con 0x7f62000a4fc0 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 4 set_mon_vals no callback set 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:29.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6204d45640 10 monclient: dump: 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.619 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient: shutdown 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- 172.21.15.98:0/1062842997 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f62000a4fc0 msgr2=0x7f6200139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 --2- 172.21.15.98:0/1062842997 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f62000a4fc0 0x7f6200139930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009870 tx=0x7f61fc01c560 comp rx=0 tx=0).stop 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- 172.21.15.98:0/1062842997 shutdown_connections 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 --2- 172.21.15.98:0/1062842997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6200139e70 0x7f620013a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 --2- 172.21.15.98:0/1062842997 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f62000a4fc0 0x7f6200139930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 --2- 172.21.15.98:0/1062842997 >> v2:172.21.15.98:3300/0 conn(0x7f62000a46a0 0x7f62000a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- 172.21.15.98:0/1062842997 >> 172.21.15.98:0/1062842997 conn(0x7f620006ed00 msgr2=0x7f620006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- 172.21.15.98:0/1062842997 shutdown_connections 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- 172.21.15.98:0/1062842997 wait complete. 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient: monmap: 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:29.621 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 librados: starting msgr at 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 librados: starting objecter 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 Processor -- start 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- start start 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 librados: setting wanted keys 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 librados: calling monclient init 2024-09-11T02:50:29.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient: init 2024-09-11T02:50:29.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient: _add_conns ranks=[6,0,4,3,5,7,2,8,1] 2024-09-11T02:50:29.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62000a46a0 0x7f62001d1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): picked mon.noname-d con 0x7f62000a46a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62000a4fc0 0x7f62001d19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): picked mon.noname-a con 0x7f62000a4fc0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6200139e70 0x7f620013d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): picked mon.noname-c con 0x7f6200139e70 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6200070090 con 0x7f62000a4fc0 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6205d47640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62000a46a0 msgr2=0x7f62001d1470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f62000a14b0 con 0x7f6200139e70 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f62000a53c0 con 0x7f62000a46a0 2024-09-11T02:50:29.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6205d47640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62000a46a0 0x7f62001d1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6207fab640 10 monclient(hunting): authenticate will time out at 983.495220s 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6205546640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62000a4fc0 msgr2=0x7f62001d19b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6205546640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62000a4fc0 0x7f62001d19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6200139e70 0x7f620013d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6200139e70 0x7f620013d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:36732/0 (socket says 172.21.15.98:36732) 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 -- 172.21.15.98:0/1411514813 learned_addr learned my addr 172.21.15.98:0/1411514813 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): get_auth_request con 0x7f6200139e70 auth_method 0 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient(hunting): handle_auth_done global_id 4513 payload 824 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62000a46a0 msgr2=0x7f62001d1470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62000a46a0 0x7f62001d1470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62000a4fc0 msgr2=0x7f62001d19b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f62000a4fc0 0x7f62001d19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 monclient: found mon.noname-c 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 -- 172.21.15.98:0/1411514813 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6200123f50 con 0x7f6200139e70 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f6206548640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6200139e70 0x7f620013d930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61f800efc0 tx=0x7f61f800c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f61f27fc640 10 client.?.objecter ms_handle_connect 0x7f6200139e70 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.611+0000 7f61f27fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61f8018020 con 0x7f6200139e70 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: dump: 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61f800cd30 con 0x7f6200139e70 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61f8014d60 con 0x7f6200139e70 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 5 monclient: authenticate success, global_id 4513 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 10 monclient: _renew_subs 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6200124230 con 0x7f6200139e70 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f3fff640 4 set_mon_vals no callback set 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:50:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: dump: 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 10 monclient: _renew_subs 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f620013a320 con 0x7f6200139e70 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 1 librados: init done 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013e630 0x7f620002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f62001400c0 con 0x7f620013e630 2024-09-11T02:50:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f61f8014480 con 0x7f6200139e70 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013e630 0x7f620002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient: get_auth_request con 0x7f620013e630 auth_method 0 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f61e403df40 0x7f61e4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f61f804fd90 con 0x7f6200139e70 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 client.4513.objecter ms_dispatch 0x7f6200000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 3 client.4513.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 3 client.4513.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 10 monclient(hunting): handle_auth_done global_id 4424 payload 824 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 1 -- 172.21.15.98:0/1411514813 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62000a8960 con 0x7f620013e630 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 20 client.4513.objecter dump_active .. 0 homeless 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205546640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f61e403df40 0x7f61e4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205546640 10 monclient: get_auth_request con 0x7f61e403df40 auth_method 0 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 client.4513.objecter ms_handle_connect 0x7f620013e630 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 client.4513.objecter resend_mon_ops 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205d47640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013e630 0x7f620002d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61f4003990 tx=0x7f61f400b810 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61f4016070 con 0x7f620013e630 2024-09-11T02:50:29.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f6205546640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f61e403df40 0x7f61e4040400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0ae000 tx=0x7f61fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: dump: 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 client.4513.objecter ms_handle_connect 0x7f61e403df40 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f61f4003bd0 con 0x7f620013e630 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013e630 msgr2=0x7f620002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013e630 0x7f620002d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61f4003990 tx=0x7f61f400b810 comp rx=0 tx=0).stop 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013e630 msgr2=0x7f620002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.615+0000 7f61f27fc640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013e630 0x7f620002d450 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61f4003990 tx=0x7f61f400b810 comp rx=0 tx=0).stop 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013ea10 0x7f620002d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f620006ca10 con 0x7f620013ea10 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013ea10 0x7f620002d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient: get_auth_request con 0x7f620013ea10 auth_method 0 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 10 monclient(hunting): handle_auth_done global_id 4433 payload 824 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 1 -- 172.21.15.98:0/1411514813 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f62000a4a80 con 0x7f620013ea10 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6205d47640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013ea10 0x7f620002d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61f4003990 tx=0x7f61f4003800 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 10 client.4513.objecter ms_handle_connect 0x7f620013ea10 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 10 client.4513.objecter resend_mon_ops 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f61f4016050 con 0x7f620013ea10 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 10 monclient: dump: 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.644 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9411 (secure 0 0 0) 0x7f61f4021410 con 0x7f620013ea10 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:29.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013ea10 msgr2=0x7f620002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013ea10 0x7f620002d990 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61f4003990 tx=0x7f61f4003800 comp rx=0 tx=0).stop 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013ea10 msgr2=0x7f620002d990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f61f27fc640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f620013ea10 0x7f620002d990 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:29.646 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 22, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 45371, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:50:29.647 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.648 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.649 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:29.650 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.651 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.655 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.658 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.662 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.664 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:50:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:29.670 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 10 librados: watch_flush enter 2024-09-11T02:50:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 10 librados: watch_flush exit 2024-09-11T02:50:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 20 client.4513.objecter shutdown clearing up homeless session... 2024-09-11T02:50:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 10 client.4513.objecter successfully canceled tick 2024-09-11T02:50:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f61e403df40 msgr2=0x7f61e4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f61e403df40 0x7f61e4040400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f61fc0ae000 tx=0x7f61fc008040 comp rx=0 tx=0).stop 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 10 monclient: shutdown 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6200139e70 msgr2=0x7f620013d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 --2- 172.21.15.98:0/1411514813 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6200139e70 0x7f620013d930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61f800efc0 tx=0x7f61f800c490 comp rx=0 tx=0).stop 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6206548640 1 -- 172.21.15.98:0/1411514813 reap_dead start 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6206548640 1 -- 172.21.15.98:0/1411514813 reap_dead start 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 shutdown_connections 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 >> 172.21.15.98:0/1411514813 conn(0x7f620006ed00 msgr2=0x7f6200072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 shutdown_connections 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 -- 172.21.15.98:0/1411514813 wait complete. 2024-09-11T02:50:29.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.623+0000 7f6207fab640 1 librados: shutdown 2024-09-11T02:50:29.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:50:29.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-09-11T02:50:29.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:50:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient: get_monmap_and_config 2024-09-11T02:50:29.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient: monmap: 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:50:29.793374+0000 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:50:29.793374+0000 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:50:29.790 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:50:29.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 1 Processor -- start 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 1 -- start start 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient: init 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient: _add_conns ranks=[8,6,1,7,2,3,5,0,10,4,9] 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6240a2a30 0x7fc6240a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient(hunting): picked mon.noname-e con 0x7fc6240a2a30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6240a46a0 0x7fc6240a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient(hunting): picked mon.noname-d con 0x7fc6240a46a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:50:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc6240a4fc0 addr v1:172.21.15.190:6789/0 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6240774c0 con 0x7fc6240a4fc0 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc624077740 con 0x7fc6240a46a0 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc6240a0e30 con 0x7fc6240a2a30 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc62a06f640 10 monclient(hunting): authenticate will time out at 983.673809s 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc622ffd640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6240a46a0 msgr2=0x7fc6240a4a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc622ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6240a46a0 0x7fc6240a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6240a2a30 0x7fc6240a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6240a2a30 0x7fc6240a2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55722/0 (socket says 172.21.15.98:55722) 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 -- 172.21.15.98:0/3640502759 learned_addr learned my addr 172.21.15.98:0/3640502759 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): get_auth_request con 0x7fc6240a2a30 auth_method 0 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient(hunting): handle_auth_done global_id 4544 payload 274 2024-09-11T02:50:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 -- 172.21.15.98:0/3640502759 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6240a46a0 msgr2=0x7fc6240a4a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:50:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 --2- 172.21.15.98:0/3640502759 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6240a46a0 0x7fc6240a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 -- 172.21.15.98:0/3640502759 >> v1:172.21.15.190:6789/0 conn(0x7fc6240a4fc0 legacy=0x7fc624140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 monclient: found mon.noname-e 2024-09-11T02:50:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:50:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.791+0000 7fc6237fe640 1 -- 172.21.15.98:0/3640502759 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6240a1110 con 0x7fc6240a2a30 2024-09-11T02:50:29.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6237fe640 1 --2- 172.21.15.98:0/3640502759 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6240a2a30 0x7fc6240a2e10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0098d0 tx=0x7fc61c01c4f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a561dc08e1e4aac9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 1 -- 172.21.15.98:0/3640502759 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc61c0b1070 con 0x7fc6240a2a30 2024-09-11T02:50:29.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: dump: 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.806 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 1 -- 172.21.15.98:0/3640502759 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61c0ab830 con 0x7fc6240a2a30 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 5 monclient: authenticate success, global_id 4544 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 1 -- 172.21.15.98:0/3640502759 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc61c0aa320 con 0x7fc6240a2a30 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient: get_monmap_and_config success 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 4 set_mon_vals no callback set 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6227fc640 10 monclient: dump: 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient: shutdown 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- 172.21.15.98:0/3640502759 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6240a2a30 msgr2=0x7fc6240a2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3640502759 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6240a2a30 0x7fc6240a2e10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0098d0 tx=0x7fc61c01c4f0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- 172.21.15.98:0/3640502759 shutdown_connections 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3640502759 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6240a46a0 0x7fc6240a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3640502759 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6240a2a30 0x7fc6240a2e10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- 172.21.15.98:0/3640502759 >> 172.21.15.98:0/3640502759 conn(0x7fc62406eed0 msgr2=0x7fc62406f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- 172.21.15.98:0/3640502759 shutdown_connections 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- 172.21.15.98:0/3640502759 wait complete. 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient: build_initial_monmap 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient: monmap: 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:50:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 librados: starting msgr at 2024-09-11T02:50:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 librados: starting objecter 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 Processor -- start 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- start start 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 librados: setting wanted keys 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 librados: calling monclient init 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient: init 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient: _reopen_session rank -1 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient: _add_conns ranks=[5,7,3,6,1,4,8,0,2] 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc6240a2a30 0x7fc6241d17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): picked mon.noname-h con 0x7fc6240a2a30 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc6240a46a0 0x7fc62413c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): picked mon.noname-i con 0x7fc6240a46a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240a4fc0 0x7fc6241236f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): picked mon.noname-g con 0x7fc6240a4fc0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc6240a2260 con 0x7fc6240a4fc0 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc6240a20e0 con 0x7fc6240a2a30 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc6241447d0 con 0x7fc6240a46a0 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): _renew_subs 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc62a06f640 10 monclient(hunting): authenticate will time out at 983.679753s 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6237fe640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc6240a2a30 0x7fc6241d17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.795+0000 7fc6237fe640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc6240a2a30 0x7fc6241d17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:42372/0 (socket says 172.21.15.98:42372) 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 -- 172.21.15.98:0/3095967420 learned_addr learned my addr 172.21.15.98:0/3095967420 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): get_auth_request con 0x7fc6240a2a30 auth_method 0 2024-09-11T02:50:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc6240a46a0 0x7fc62413c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): get_auth_request con 0x7fc6240a46a0 auth_method 0 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240a4fc0 0x7fc6241236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient(hunting): get_auth_request con 0x7fc6240a4fc0 auth_method 0 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient(hunting): handle_auth_done global_id 4442 payload 824 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc6240a46a0 msgr2=0x7fc62413c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc6240a46a0 0x7fc62413c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240a4fc0 msgr2=0x7fc6241236f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc6240a4fc0 0x7fc6241236f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient: _finish_hunting 0 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 monclient: found mon.noname-h 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 -- 172.21.15.98:0/3095967420 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc624126d20 con 0x7fc6240a2a30 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6237fe640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc6240a2a30 0x7fc6241d17c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc61c01cd90 tx=0x7fc61c019290 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 client.?.objecter ms_handle_connect 0x7fc6240a2a30 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc61c0ba070 con 0x7fc6240a2a30 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: dump: 2024-09-11T02:50:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: _finish_auth 0 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 5 monclient: authenticate success, global_id 4442 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 10 monclient: _renew_subs 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc624127000 con 0x7fc6240a2a30 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc61c0abe40 con 0x7fc6240a2a30 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:50:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 10 monclient: _renew_subs 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62409ec40 con 0x7fc6240a2a30 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 1 librados: init done 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc6217fa640 4 set_mon_vals no callback set 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc61c0b1040 con 0x7fc6240a2a30 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: dump: 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc62413ce70 0x7fc624123c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc62406ca10 con 0x7fc62413ce70 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc62413ce70 0x7fc624123c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient: get_auth_request con 0x7fc62413ce70 auth_method 0 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 10 monclient(hunting): handle_auth_done global_id 4426 payload 824 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 1 -- 172.21.15.98:0/3095967420 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6240a2e10 con 0x7fc62413ce70 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc61c004040 con 0x7fc6240a2a30 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc622ffd640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc62413ce70 0x7fc624123c30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc624141d50 tx=0x7fc60c007660 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc60803df70 0x7fc608040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 client.4442.objecter ms_handle_connect 0x7fc62413ce70 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 client.4442.objecter resend_mon_ops 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fc61c0e8360 con 0x7fc6240a2a30 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 client.4442.objecter ms_dispatch 0x7fc624000ba0 osd_map(20..20 src has 1..20) 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 3 client.4442.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 3 client.4442.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 20 client.4442.objecter dump_active .. 0 homeless 2024-09-11T02:50:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc60c01a070 con 0x7fc62413ce70 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: dump: 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc60803df70 0x7fc608040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 10 monclient: get_auth_request con 0x7fc60803df70 auth_method 0 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc60c0052e0 con 0x7fc62413ce70 2024-09-11T02:50:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:50:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:50:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc62413ce70 msgr2=0x7fc624123c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc62413ce70 0x7fc624123c30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc624141d50 tx=0x7fc60c007660 comp rx=0 tx=0).stop 2024-09-11T02:50:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc62413ce70 msgr2=0x7fc624123c30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc62413ce70 0x7fc624123c30 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc624141d50 tx=0x7fc60c007660 comp rx=0 tx=0).stop 2024-09-11T02:50:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc623fff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc60803df70 0x7fc608040430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc614007fd0 tx=0x7fc614009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.799+0000 7fc61bfff640 10 client.4442.objecter ms_handle_connect 0x7fc60803df70 2024-09-11T02:50:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc624124170 0x7fc62409d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc624124580 con 0x7fc624124170 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc624124170 0x7fc62409d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient: get_auth_request con 0x7fc624124170 auth_method 0 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:50:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 1 -- 172.21.15.98:0/3095967420 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc624124f80 con 0x7fc624124170 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 10 client.4442.objecter ms_handle_connect 0x7fc624124170 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 10 client.4442.objecter resend_mon_ops 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc622ffd640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc624124170 0x7fc62409d250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc624141d50 tx=0x7fc60c005d30 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc60c01a050 con 0x7fc624124170 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 10 monclient: dump: 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:50:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:50:29.829 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9244 (secure 0 0 0) 0x7fc60c01b040 con 0x7fc624124170 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc624124170 msgr2=0x7fc62409d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc624124170 0x7fc62409d250 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc624141d50 tx=0x7fc60c005d30 comp rx=0 tx=0).stop 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc624124170 msgr2=0x7fc62409d250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc61bfff640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc624124170 0x7fc62409d250 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc624141d50 tx=0x7fc60c005d30 comp rx=0 tx=0).stop 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:29.830 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 22, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 45575, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:50:29.831 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.832 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.833 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:50:29.834 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:50:29.835 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:50:29.841 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.845 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:50:29.847 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:50:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:50:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:50:29.855 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:50:29.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 10 librados: watch_flush enter 2024-09-11T02:50:29.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 10 librados: watch_flush exit 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 20 client.4442.objecter shutdown clearing up homeless session... 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 10 client.4442.objecter successfully canceled tick 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc60803df70 msgr2=0x7fc608040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc60803df70 0x7fc608040430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc614007fd0 tx=0x7fc614009040 comp rx=0 tx=0).stop 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 10 monclient: shutdown 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc6240a2a30 msgr2=0x7fc6241d17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc6240a2a30 0x7fc6241d17c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc61c01cd90 tx=0x7fc61c019290 comp rx=0 tx=0).stop 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc623fff640 1 -- 172.21.15.98:0/3095967420 reap_dead start 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 shutdown_connections 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 --2- 172.21.15.98:0/3095967420 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc6240a2a30 0x7fc6241d17c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:50:29.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 >> 172.21.15.98:0/3095967420 conn(0x7fc62406eed0 msgr2=0x7fc62409f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:50:29.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 shutdown_connections 2024-09-11T02:50:29.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 -- 172.21.15.98:0/3095967420 wait complete. 2024-09-11T02:50:29.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:50:29.807+0000 7fc62a06f640 1 librados: shutdown 2024-09-11T02:50:29.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:50:29.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr:Note: switching to '8f219c476cb4f26fc5719c661c0323719a45cb30'. 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T02:50:59.810 INFO:tasks.workunit.client.0.smithi190.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr: git switch -c 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr:Or undo this operation with: 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr: git switch - 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr: 2024-09-11T02:50:59.811 INFO:tasks.workunit.client.0.smithi190.stderr:HEAD is now at 8f219c476cb Merge branch 'mgr-vol-mod-disable' of https://github.com/rishabh-d-dave/ceph into wip-hemanth-testing-2024-09-10-0723 2024-09-11T02:50:59.815 DEBUG:teuthology.orchestra.run.smithi190:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T02:50:59.862 INFO:tasks.workunit.client.0.smithi190.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T02:50:59.866 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T02:50:59.866 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T02:51:00.682 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T02:51:00.764 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T02:51:00.812 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T02:51:00.813 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T02:51:00.813 INFO:tasks.workunit.client.0.smithi190.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T02:51:00.860 INFO:tasks.workunit.client.0.smithi190.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T02:51:00.863 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-09-11T02:51:00.864 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T02:51:00.910 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-11T02:51:00.911 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-11T02:51:00.912 DEBUG:teuthology.orchestra.run.smithi190:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8f219c476cb4f26fc5719c661c0323719a45cb30 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ parallel=1 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' '' = --serial ']' 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ crimson=0 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' '' = --crimson ']' 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ color= 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -t 1 ']' 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-11T02:51:00.962 INFO:tasks.workunit.client.0.smithi190.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-11T02:51:00.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ declare -A pids 2024-09-11T02:51:00.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.964 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_aio 2024-09-11T02:51:00.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_aio' 2024-09-11T02:51:00.965 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_aio 2024-09-11T02:51:00.965 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:00.967 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_aio 2024-09-11T02:51:00.967 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28156 2024-09-11T02:51:00.967 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_aio on pid 28156' 2024-09-11T02:51:00.967 INFO:tasks.workunit.client.0.smithi190.stdout:test api_aio on pid 28156 2024-09-11T02:51:00.968 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28156 2024-09-11T02:51:00.968 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_aio_pp 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_aio_pp' 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_aio_pp 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-11T02:51:00.969 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_aio: /' 2024-09-11T02:51:00.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_aio_pp 2024-09-11T02:51:00.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28164 2024-09-11T02:51:00.971 INFO:tasks.workunit.client.0.smithi190.stdout:test api_aio_pp on pid 28164 2024-09-11T02:51:00.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_aio_pp on pid 28164' 2024-09-11T02:51:00.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28164 2024-09-11T02:51:00.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-11T02:51:00.972 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_io 2024-09-11T02:51:00.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_io' 2024-09-11T02:51:00.972 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_io 2024-09-11T02:51:00.972 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:00.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:00.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:00.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-11T02:51:00.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-11T02:51:00.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-11T02:51:00.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_io 2024-09-11T02:51:00.974 INFO:tasks.workunit.client.0.smithi190.stdout:test api_io on pid 28172 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28172 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_io on pid 28172' 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28172 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_io_pp 2024-09-11T02:51:00.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_io_pp' 2024-09-11T02:51:00.976 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_io_pp 2024-09-11T02:51:00.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:00.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:00.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.977 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:00.977 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-11T02:51:00.977 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_io.log 2024-09-11T02:51:00.977 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_io: /' 2024-09-11T02:51:00.977 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_io_pp 2024-09-11T02:51:00.978 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28180 2024-09-11T02:51:00.978 INFO:tasks.workunit.client.0.smithi190.stdout:test api_io_pp on pid 28180 2024-09-11T02:51:00.978 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_io_pp on pid 28180' 2024-09-11T02:51:00.978 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28180 2024-09-11T02:51:00.979 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.979 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.979 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-11T02:51:00.979 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_asio 2024-09-11T02:51:00.979 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_asio' 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_asio 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-11T02:51:00.980 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-11T02:51:00.982 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_asio 2024-09-11T02:51:00.982 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28188 2024-09-11T02:51:00.982 INFO:tasks.workunit.client.0.smithi190.stdout:test api_asio on pid 28188 2024-09-11T02:51:00.982 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_asio on pid 28188' 2024-09-11T02:51:00.983 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28188 2024-09-11T02:51:00.983 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.983 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.983 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-11T02:51:00.983 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_list 2024-09-11T02:51:00.983 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_list' 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_list 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-11T02:51:00.984 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-11T02:51:00.985 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_asio: /' 2024-09-11T02:51:00.985 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_list 2024-09-11T02:51:00.986 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28196 2024-09-11T02:51:00.986 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_list on pid 28196' 2024-09-11T02:51:00.986 INFO:tasks.workunit.client.0.smithi190.stdout:test api_list on pid 28196 2024-09-11T02:51:00.986 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28196 2024-09-11T02:51:00.986 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.987 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.987 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-11T02:51:00.987 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:00.987 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.988 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:00.988 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.988 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-11T02:51:00.988 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_list.log 2024-09-11T02:51:00.988 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_list: /' 2024-09-11T02:51:00.989 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_lock 2024-09-11T02:51:00.990 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_lock' 2024-09-11T02:51:00.990 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_lock 2024-09-11T02:51:00.990 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:00.993 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_lock 2024-09-11T02:51:00.993 INFO:tasks.workunit.client.0.smithi190.stdout:test api_lock on pid 28204 2024-09-11T02:51:00.993 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28204 2024-09-11T02:51:00.994 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_lock on pid 28204' 2024-09-11T02:51:00.994 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28204 2024-09-11T02:51:00.994 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-11T02:51:00.994 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:00.994 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:00.994 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_lock_pp 2024-09-11T02:51:00.995 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_lock_pp' 2024-09-11T02:51:00.995 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:00.995 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.995 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:00.995 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:00.995 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-11T02:51:00.995 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-11T02:51:01.005 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.007 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_lock_pp 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stdout:test api_lock_pp on pid 28219 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_lock_pp 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28219 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_lock_pp on pid 28219' 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28219 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.009 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_misc 2024-09-11T02:51:01.010 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_misc' 2024-09-11T02:51:01.010 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_lock: /' 2024-09-11T02:51:01.010 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-11T02:51:01.010 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_misc 2024-09-11T02:51:01.011 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.011 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.011 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.011 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.011 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.012 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-11T02:51:01.012 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-11T02:51:01.013 INFO:tasks.workunit.client.0.smithi190.stdout:test api_misc on pid 28231 2024-09-11T02:51:01.013 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_misc 2024-09-11T02:51:01.014 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-11T02:51:01.014 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28231 2024-09-11T02:51:01.014 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_misc on pid 28231' 2024-09-11T02:51:01.014 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28231 2024-09-11T02:51:01.014 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.014 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.014 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-11T02:51:01.015 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_misc_pp 2024-09-11T02:51:01.015 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_misc_pp' 2024-09-11T02:51:01.015 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.015 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.015 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.015 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.015 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-11T02:51:01.016 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-11T02:51:01.016 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.016 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_misc_pp 2024-09-11T02:51:01.017 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_misc: /' 2024-09-11T02:51:01.021 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_misc_pp 2024-09-11T02:51:01.021 INFO:tasks.workunit.client.0.smithi190.stdout:test api_misc_pp on pid 28259 2024-09-11T02:51:01.022 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28259 2024-09-11T02:51:01.022 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_misc_pp on pid 28259' 2024-09-11T02:51:01.022 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28259 2024-09-11T02:51:01.022 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.022 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.022 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_tier_pp 2024-09-11T02:51:01.023 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_tier_pp' 2024-09-11T02:51:01.025 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-11T02:51:01.026 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_tier_pp 2024-09-11T02:51:01.027 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.028 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.028 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.028 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.028 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-11T02:51:01.037 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-11T02:51:01.038 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.041 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] Global test environment set-up. 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.041+0000 7f06ae12b880 10 monclient: get_monmap_and_config 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.041+0000 7f06ae12b880 10 monclient: build_initial_monmap 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.041+0000 7f06ae12b880 10 monclient: monmap: 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 0 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 0.000000 2024-09-11T02:51:01.043 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 0.000000 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 1 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.044 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.041+0000 7f06ae12b880 1 Processor -- start 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 1 -- start start 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient: init 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient: _reopen_session rank -1 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient: _add_conns ranks=[8,1,7,3,6,4,0,5,2] 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4d50b30 0x563eb4dded60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): picked mon.noname-e con 0x563eb4d50b30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563eb4de5ed0 0x563eb4d5db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): picked mon.noname-f con 0x563eb4de5ed0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x563eb4d5ad80 0x563eb4d5b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): picked mon.noname-i con 0x563eb4d5ad80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): start opening mon connection 2024-09-11T02:51:01.045 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x563eb4d50f10 con 0x563eb4de5ed0 2024-09-11T02:51:01.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): start opening mon connection 2024-09-11T02:51:01.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x563eb4c7d9f0 con 0x563eb4d5ad80 2024-09-11T02:51:01.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): start opening mon connection 2024-09-11T02:51:01.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x563eb4dc7050 con 0x563eb4d50b30 2024-09-11T02:51:01.046 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): _renew_subs 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient(hunting): authenticate will time out at 1021.713094s 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4d50b30 0x563eb4dded60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4d50b30 0x563eb4dded60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.190:50478/0 (socket says 172.21.15.190:50478) 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 -- 172.21.15.190:0/27072707 learned_addr learned my addr 172.21.15.190:0/27072707 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): get_auth_request con 0x563eb4d50b30 auth_method 0 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient(hunting): handle_auth_done global_id 4562 payload 274 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 -- 172.21.15.190:0/27072707 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x563eb4d5ad80 msgr2=0x563eb4d5b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:51:01.047 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 --2- 172.21.15.190:0/27072707 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x563eb4d5ad80 0x563eb4d5b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 -- 172.21.15.190:0/27072707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563eb4de5ed0 msgr2=0x563eb4d5db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:51:01.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 --2- 172.21.15.190:0/27072707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563eb4de5ed0 0x563eb4d5db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient: _finish_hunting 0 2024-09-11T02:51:01.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 monclient: found mon.noname-e 2024-09-11T02:51:01.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:51:01.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:51:01.048 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 -- 172.21.15.190:0/27072707 --> [v2:172.21.15.98test api_tier_pp on pid 28327 2024-09-11T02:51:01.049 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_tier_pp 2024-09-11T02:51:01.049 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28327 2024-09-11T02:51:01.049 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_tier_pp on pid 28327' 2024-09-11T02:51:01.050 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28327 2024-09-11T02:51:01.050 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.050 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.050 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_pool 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout::3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb4c7ca10 con 0x563eb4d50b30 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a77fe640 1 --2- 172.21.15.190:0/27072707 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4d50b30 0x563eb4dded60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f069c00a600 tx=0x7f069c00aac0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=9ff66e02578983d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 1 -- 172.21.15.190:0/27072707 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f069c008ca0 con 0x563eb4d50b30 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 10 monclient: dump: 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 1 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.051 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 19 (squid) 2024-09-11T02:51:01.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 3 2024-09-11T02:51:01.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:51:01.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:51:01.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:51:01.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:51:01.052 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 10 monclient: _finish_auth 0 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 1 -- 172.21.15.190:0/27072707 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f069c0123e0 con 0x563eb4d50b30 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 5 monclient: authenticate success, global_id 4562 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:51:01.053 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 10 monclient: get_monmap_and_config success 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.045+0000 7f06ae12b880 4 set_mon_vals no callback set 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a5ffb640 1 -- 172.21.15.190:0/27072707 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f069c008e60 con 0x563eb4d50b30 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a5ffb640 10 monclient: dump: 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 1 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:51:01.054 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.055 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_pool' 2024-09-11T02:51:01.055 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.055 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.055 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 19 (squid) 2024-09-11T02:51:01.055 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 3 2024-09-11T02:51:01.055 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:51:01.055 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:51:01.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:51:01.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:51:01.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:51:01.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient: shutdown 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- 172.21.15.190:0/27072707 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4d50b30 msgr2=0x563eb4dded60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 --2- 172.21.15.190:0/27072707 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4d50b30 0x563eb4dded60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f069c00a600 tx=0x7f069c00aac0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- 172.21.15.190:0/27072707 shutdown_connections 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 --2- 172.21.15.190:0/27072707 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x563eb4d5ad80 0x563eb4d5b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 --2- 172.21.15.190:0/27072707 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563eb4de5ed0 0x563eb4d5db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 --2- 172.21.15.190:0/27072707 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4d50b30 0x563eb4dded60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- 172.21.15.190:0/27072707 >> 172.21.15.190:0/27072707 conn(0x563eb4d2fd50 msgr2=0x563eb4de50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- 172.21.15.190:0/27072707 shutdown_connections 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- 172.21.15.190:0/27072707 wait complete. 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient: build_initial_monmap 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient: monmap: 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 0 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 0.000000 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 0.000000 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-11T02:51:01.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 1 2024-09-11T02:51:01.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:51:01.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:51:01.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:51:01.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:51:01.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:51:01.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:51:01.059 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 librados: starting msgr at 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 librados: starting objecter 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 Processor -- start 2024-09-11T02:51:01.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- start start 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 librados: setting wanted keys 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 librados: calling monclient init 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient: init 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient: _reopen_session rank -1 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient: _add_conns ranks=[6,4,8,1,3,2,5,7,0] 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x563eb4d50b30 0x563eb4dec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): picked mon.noname-d con 0x563eb4d50b30 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x563eb4d5ad80 0x563eb4dec8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): picked mon.noname-c con 0x563eb4d5ad80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4de5ed0 0x563eb4de8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): picked mon.noname-e con 0x563eb4de5ed0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): start opening mon connection 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x563eb4de29d0 con 0x563eb4d5ad80 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): start opening mon connection 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x563eb4d5c4d0 con 0x563eb4d50b30 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): start opening mon connection 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x563eb4dc6650 con 0x563eb4de5ed0 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): _renew_subs 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06ae12b880 10 monclient(hunting): authenticate will time out at 1021.718764s 2024-09-11T02:51:01.062 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a77fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x563eb4d50b30 msgr2=0x563eb4dec3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:51:01.063 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_pool 2024-09-11T02:51:01.063 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x563eb4d50b30 0x563eb4dec3b0 unknown :-1 s=START_CONNEtest api_pool on pid 28343 2024-09-11T02:51:01.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_pool 2024-09-11T02:51:01.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28343 2024-09-11T02:51:01.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_pool on pid 28343' 2024-09-11T02:51:01.063 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28343 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stdout:CT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x563eb4d5ad80 0x563eb4dec8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x563eb4d5ad80 0x563eb4dec8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.190:60192/0 (socket says 172.21.15.190:60192) 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 learned_addr learned my addr 172.21.15.190:0/4261071606 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 10 monclient(hunting): get_auth_request con 0x563eb4d5ad80 auth_method 0 2024-09-11T02:51:01.064 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4de5ed0 0x563eb4de8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): get_auth_request con 0x563eb4de5ed0 auth_method 0 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:51:01.065 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:51:01.066 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:51:01.066 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:51:01.066 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:51:01.066 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 10 monclient(hunting): handle_auth_done global_id 4571 payload 824 2024-09-11T02:51:01.066 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.049+0000 7f06a7fff640 1 -- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x563eb4d50b30 msgr2=0x563eb4dec3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:51:01.066 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x563eb4d50b30 0x563eb4dec3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.066 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 1 -- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x563eb4d5ad80 msgr2=0x563eb4dec8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:51:01.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.067 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.067 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x563eb4d5ad80 0x563eb4dec8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:01.067 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 10 monclient: _finish_hunting 0 2024-09-11T02:51:01.067 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 1 monclient: found mon.noname-e 2024-09-11T02:51:01.067 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb4c56c00 con 0x563eb4de5ed0 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a6ffd640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x563eb4d5ad80 0x563eb4dec8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a7fff640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4de5ed0 0x563eb4de8020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005e40 tx=0x7f06a0008430 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 client.?.objecter ms_handle_connect 0x563eb4de5ed0 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06a0005bf0 con 0x563eb4de5ed0 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:51:01.068 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: dump: 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 1 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 19 (squid) 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 3 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:51:01.069 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: _finish_auth 0 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 5 monclient: authenticate success, global_id 4571 2024-09-11T02:51:01.070 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 10 monclient: _renew_subs 2024-09-11T02:51:01.071 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.071 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-11T02:51:01.071 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-11T02:51:01.071 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x563eb4dece30 con 0x563eb4de5ed0 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 10 monclient: _renew_subs 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x563eb4deaf00 con 0x563eb4de5ed0 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 1 librados: init done 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06ae12b880 10 librados: wait_for_osdmap waiting 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a0015070 con 0x563eb4de5ed0 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a57fa640 4 set_mon_vals no callback set 2024-09-11T02:51:01.072 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06a0008e10 con 0x563eb4de5ed0 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 10 monclient: dump: 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: epoch 1 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: min_mon_release 19 (squid) 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: election_strategy: 3 2024-09-11T02:51:01.073 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f06a0010440 con 0x563eb4de5ed0 2024-09-11T02:51:01.074 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a4ff9640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f069003c4d0 0x7f069003e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.075 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.053+0000 7f06a77fe640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f069003c4d0 0x7f069003e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_snapshots 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_snapshots' 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.076 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-11T02:51:01.077 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_pool: /' 2024-09-11T02:51:01.077 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_snapshots 2024-09-11T02:51:01.077 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-11T02:51:01.077 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.077 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-11T02:51:01.077 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:test api_snapshots on pid 28380 2024-09-11T02:51:01.079 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_snapshots 2024-09-11T02:51:01.079 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28380 2024-09-11T02:51:01.079 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_snapshots on pid 28380' 2024-09-11T02:51:01.079 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28380 2024-09-11T02:51:01.079 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.079 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.080 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_snapshots_pp 2024-09-11T02:51:01.080 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_snapshots_pp' 2024-09-11T02:51:01.080 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-11T02:51:01.080 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.080 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.080 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_snapshots_pp 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_snapshots_pp 2024-09-11T02:51:01.081 INFO:tasks.workunit.client.0.smithi190.stdout:test api_snapshots_pp on pid 28398 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28398 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_snapshots_pp on pid 28398' 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28398 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_stat 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_stat' 2024-09-11T02:51:01.083 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_stat 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_stat 2024-09-11T02:51:01.084 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28416 2024-09-11T02:51:01.085 INFO:tasks.workunit.client.0.smithi190.stdout:test api_stat on pid 28416 2024-09-11T02:51:01.086 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_stat on pid 28416' 2024-09-11T02:51:01.086 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28416 2024-09-11T02:51:01.086 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_stat_pp 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_stat_pp' 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_stat: /' 2024-09-11T02:51:01.087 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.088 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_stat_pp 2024-09-11T02:51:01.088 INFO:tasks.workunit.client.0.smithi190.stdout:test api_stat_pp on pid 28429 2024-09-11T02:51:01.089 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_stat_pp 2024-09-11T02:51:01.089 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28429 2024-09-11T02:51:01.089 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_stat_pp on pid 28429' 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28429 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-11T02:51:01.090 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-11T02:51:01.091 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_watch_notify 2024-09-11T02:51:01.091 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_watch_notify' 2024-09-11T02:51:01.091 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.092 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_watch_notify 2024-09-11T02:51:01.092 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-11T02:51:01.092 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_watch_notify 2024-09-11T02:51:01.092 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28449 2024-09-11T02:51:01.092 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_watch_notify on pid 28449' 2024-09-11T02:51:01.092 INFO:tasks.workunit.client.0.smithi190.stdout:test api_watch_notify on pid 28449 2024-09-11T02:51:01.093 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28449 2024-09-11T02:51:01.093 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.093 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.093 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_watch_notify_pp 2024-09-11T02:51:01.093 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_watch_notify_pp' 2024-09-11T02:51:01.094 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.095 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_watch_notify_pp 2024-09-11T02:51:01.096 INFO:tasks.workunit.client.0.smithi190.stdout:test api_watch_notify_pp on pid 28461 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_watch_notify_pp 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28461 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_watch_notify_pp on pid 28461' 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28461 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_cmd 2024-09-11T02:51:01.097 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_cmd' 2024-09-11T02:51:01.098 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-11T02:51:01.098 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-11T02:51:01.098 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-11T02:51:01.100 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.100 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.101 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.101 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.101 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-11T02:51:01.101 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_cmd 2024-09-11T02:51:01.102 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-11T02:51:01.103 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.103 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.103 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.103 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.104 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-11T02:51:01.104 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-11T02:51:01.105 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.107 INFO:tasks.workunit.client.0.smithi190.stdout:test api_cmd on pid 28482 2024-09-11T02:51:01.107 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_cmd 2024-09-11T02:51:01.108 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28482 2024-09-11T02:51:01.108 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_cmd on pid 28482' 2024-09-11T02:51:01.108 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28482 2024-09-11T02:51:01.108 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.108 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.108 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-11T02:51:01.109 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_cmd_pp 2024-09-11T02:51:01.109 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-11T02:51:01.109 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_cmd_pp' 2024-09-11T02:51:01.110 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_cmd_pp 2024-09-11T02:51:01.110 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-11T02:51:01.111 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-11T02:51:01.113 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.113 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.113 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.113 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.113 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-11T02:51:01.117 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.117 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-11T02:51:01.118 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_cmd: /' 2024-09-11T02:51:01.122 INFO:tasks.workunit.client.0.smithi190.stdout:test api_cmd_pp on pid 28519 2024-09-11T02:51:01.123 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_cmd_pp 2024-09-11T02:51:01.123 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28519 2024-09-11T02:51:01.123 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_cmd_pp on pid 28519' 2024-09-11T02:51:01.123 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28519 2024-09-11T02:51:01.123 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.123 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.131 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_service 2024-09-11T02:51:01.131 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_service' 2024-09-11T02:51:01.132 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-11T02:51:01.133 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.133 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.133 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.133 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.135 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.137 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-11T02:51:01.138 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-11T02:51:01.139 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_service 2024-09-11T02:51:01.141 INFO:tasks.workunit.client.0.smithi190.stdout:test api_service on pid 28574 2024-09-11T02:51:01.141 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_service 2024-09-11T02:51:01.141 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28574 2024-09-11T02:51:01.141 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_service on pid 28574' 2024-09-11T02:51:01.141 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28574 2024-09-11T02:51:01.142 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.142 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.142 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-11T02:51:01.142 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_service_pp 2024-09-11T02:51:01.142 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_service_pp' 2024-09-11T02:51:01.143 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_service_pp 2024-09-11T02:51:01.146 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-11T02:51:01.146 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.149 INFO:tasks.workunit.client.0.smithi190.stdout:test api_service_pp on pid 28608 2024-09-11T02:51:01.150 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_service_pp 2024-09-11T02:51:01.150 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28608 2024-09-11T02:51:01.150 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_service_pp on pid 28608' 2024-09-11T02:51:01.150 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28608 2024-09-11T02:51:01.150 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.150 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.151 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.151 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.151 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.151 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.151 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-11T02:51:01.151 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_service: /' 2024-09-11T02:51:01.153 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.153 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.153 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.153 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.153 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_c_write_operations 2024-09-11T02:51:01.153 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-11T02:51:01.153 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_c_write_operations' 2024-09-11T02:51:01.154 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-11T02:51:01.154 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_c_write_operations 2024-09-11T02:51:01.154 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-11T02:51:01.154 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.155 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_service.log 2024-09-11T02:51:01.156 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-11T02:51:01.158 INFO:tasks.workunit.client.0.smithi190.stdout:test api_c_write_operations on pid 28631 2024-09-11T02:51:01.158 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_c_write_operations 2024-09-11T02:51:01.158 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28631 2024-09-11T02:51:01.158 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_c_write_operations on pid 28631' 2024-09-11T02:51:01.159 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28631 2024-09-11T02:51:01.159 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.159 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.159 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s api_c_read_operations 2024-09-11T02:51:01.159 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' api_c_read_operations' 2024-09-11T02:51:01.159 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo api_c_read_operations 2024-09-11T02:51:01.162 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.162 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-11T02:51:01.165 INFO:tasks.workunit.client.0.smithi190.stdout:test api_c_read_operations on pid 28663 2024-09-11T02:51:01.165 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=api_c_read_operations 2024-09-11T02:51:01.166 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28663 2024-09-11T02:51:01.166 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test api_c_read_operations on pid 28663' 2024-09-11T02:51:01.166 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28663 2024-09-11T02:51:01.166 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.166 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.167 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-11T02:51:01.167 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s list_parallel 2024-09-11T02:51:01.167 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' list_parallel' 2024-09-11T02:51:01.168 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo list_parallel 2024-09-11T02:51:01.170 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.170 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.170 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.170 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.171 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-11T02:51:01.171 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.171 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-11T02:51:01.172 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.172 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.172 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.172 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.172 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-11T02:51:01.174 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-11T02:51:01.174 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-11T02:51:01.174 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=list_parallel 2024-09-11T02:51:01.174 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28692 2024-09-11T02:51:01.175 INFO:tasks.workunit.client.0.smithi190.stdout:test list_parallel on pid 28692 2024-09-11T02:51:01.175 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test list_parallel on pid 28692' 2024-09-11T02:51:01.175 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28692 2024-09-11T02:51:01.175 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.176 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.176 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-11T02:51:01.177 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s open_pools_parallel 2024-09-11T02:51:01.177 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-11T02:51:01.178 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' open_pools_parallel' 2024-09-11T02:51:01.179 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.179 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.179 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.179 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.179 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-11T02:51:01.179 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-11T02:51:01.179 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.181 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ list_parallel: /' 2024-09-11T02:51:01.183 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo open_pools_parallel 2024-09-11T02:51:01.184 INFO:tasks.workunit.client.0.smithi190.stdout:test open_pools_parallel on pid 28710 2024-09-11T02:51:01.184 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=open_pools_parallel 2024-09-11T02:51:01.184 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28710 2024-09-11T02:51:01.184 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test open_pools_parallel on pid 28710' 2024-09-11T02:51:01.184 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28710 2024-09-11T02:51:01.184 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-11T02:51:01.185 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.185 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-11T02:51:01.185 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.185 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.186 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.186 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.186 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s delete_pools_parallel 2024-09-11T02:51:01.186 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' delete_pools_parallel' 2024-09-11T02:51:01.187 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.189 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo delete_pools_parallel 2024-09-11T02:51:01.190 INFO:tasks.workunit.client.0.smithi190.stdout:test delete_pools_parallel on pid 28720 2024-09-11T02:51:01.190 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=delete_pools_parallel 2024-09-11T02:51:01.190 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28720 2024-09-11T02:51:01.190 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test delete_pools_parallel on pid 28720' 2024-09-11T02:51:01.191 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28720 2024-09-11T02:51:01.191 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.191 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.191 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-11T02:51:01.192 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-11T02:51:01.192 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.192 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.192 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.192 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.192 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s cls 2024-09-11T02:51:01.193 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' cls' 2024-09-11T02:51:01.193 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-11T02:51:01.193 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.193 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-11T02:51:01.200 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-11T02:51:01.200 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-11T02:51:01.206 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo cls 2024-09-11T02:51:01.207 INFO:tasks.workunit.client.0.smithi190.stdout:test cls on pid 28743 2024-09-11T02:51:01.208 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=cls 2024-09-11T02:51:01.208 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28743 2024-09-11T02:51:01.208 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test cls on pid 28743' 2024-09-11T02:51:01.208 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28743 2024-09-11T02:51:01.208 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.208 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.209 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-11T02:51:01.214 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-11T02:51:01.216 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.216 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.216 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.216 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.216 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_cls 2024-09-11T02:51:01.217 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s cmd 2024-09-11T02:51:01.218 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' cmd' 2024-09-11T02:51:01.226 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo cmd 2024-09-11T02:51:01.229 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.230 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_cls.log 2024-09-11T02:51:01.230 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ cls: /' 2024-09-11T02:51:01.234 INFO:tasks.workunit.client.0.smithi190.stdout:test cmd on pid 28799 2024-09-11T02:51:01.235 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=cmd 2024-09-11T02:51:01.235 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28799 2024-09-11T02:51:01.235 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test cmd on pid 28799' 2024-09-11T02:51:01.235 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28799 2024-09-11T02:51:01.235 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.235 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.238 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s handler_error 2024-09-11T02:51:01.239 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' handler_error' 2024-09-11T02:51:01.239 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-11T02:51:01.244 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.245 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.246 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.246 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.246 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.250 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo handler_error 2024-09-11T02:51:01.251 INFO:tasks.workunit.client.0.smithi190.stdout:test handler_error on pid 28859 2024-09-11T02:51:01.251 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=handler_error 2024-09-11T02:51:01.251 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28859 2024-09-11T02:51:01.251 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test handler_error on pid 28859' 2024-09-11T02:51:01.251 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28859 2024-09-11T02:51:01.251 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.251 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.252 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-11T02:51:01.252 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ cmd: /' 2024-09-11T02:51:01.254 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_cmd 2024-09-11T02:51:01.255 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s io 2024-09-11T02:51:01.255 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' io' 2024-09-11T02:51:01.255 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-11T02:51:01.257 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.257 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.257 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.257 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.258 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_handler_error 2024-09-11T02:51:01.259 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo io 2024-09-11T02:51:01.260 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.262 INFO:tasks.workunit.client.0.smithi190.stdout:test io on pid 28890 2024-09-11T02:51:01.262 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=io 2024-09-11T02:51:01.262 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28890 2024-09-11T02:51:01.263 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test io on pid 28890' 2024-09-11T02:51:01.263 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28890 2024-09-11T02:51:01.263 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.263 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.263 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-11T02:51:01.265 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.265 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-11T02:51:01.265 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.265 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.265 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.266 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ handler_error: /' 2024-09-11T02:51:01.266 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s ec_io 2024-09-11T02:51:01.266 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' ec_io' 2024-09-11T02:51:01.267 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ io: /' 2024-09-11T02:51:01.267 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_io.log 2024-09-11T02:51:01.267 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_io 2024-09-11T02:51:01.270 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.270 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo ec_io 2024-09-11T02:51:01.273 INFO:tasks.workunit.client.0.smithi190.stdout:test ec_io on pid 28918 2024-09-11T02:51:01.273 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=ec_io 2024-09-11T02:51:01.273 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28918 2024-09-11T02:51:01.273 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test ec_io on pid 28918' 2024-09-11T02:51:01.274 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28918 2024-09-11T02:51:01.274 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.274 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.274 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s list 2024-09-11T02:51:01.274 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' list' 2024-09-11T02:51:01.274 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.276 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo list 2024-09-11T02:51:01.277 INFO:tasks.workunit.client.0.smithi190.stdout:test list on pid 28930 2024-09-11T02:51:01.277 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=list 2024-09-11T02:51:01.278 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28930 2024-09-11T02:51:01.278 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test list on pid 28930' 2024-09-11T02:51:01.278 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28930 2024-09-11T02:51:01.278 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.278 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.278 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s ec_list 2024-09-11T02:51:01.278 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' ec_list' 2024-09-11T02:51:01.279 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-11T02:51:01.279 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.281 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.281 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.281 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.281 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.282 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_list 2024-09-11T02:51:01.282 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_list.log 2024-09-11T02:51:01.282 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ list: /' 2024-09-11T02:51:01.283 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo ec_list 2024-09-11T02:51:01.284 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=ec_list 2024-09-11T02:51:01.284 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28957 2024-09-11T02:51:01.285 INFO:tasks.workunit.client.0.smithi190.stdout:test ec_list on pid 28957 2024-09-11T02:51:01.286 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test ec_list on pid 28957' 2024-09-11T02:51:01.286 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=28957 2024-09-11T02:51:01.286 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.286 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.286 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-11T02:51:01.286 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s misc 2024-09-11T02:51:01.286 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' misc' 2024-09-11T02:51:01.287 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.287 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.287 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.287 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.287 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_ec_list 2024-09-11T02:51:01.287 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-11T02:51:01.287 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-11T02:51:01.290 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ ec_list: /' 2024-09-11T02:51:01.293 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.293 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo misc 2024-09-11T02:51:01.295 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.295 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.295 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.295 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.297 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-11T02:51:01.298 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_ec_io 2024-09-11T02:51:01.299 INFO:tasks.workunit.client.0.smithi190.stdout:test misc on pid 29021 2024-09-11T02:51:01.300 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=misc 2024-09-11T02:51:01.300 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29021 2024-09-11T02:51:01.300 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test misc on pid 29021' 2024-09-11T02:51:01.300 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29021 2024-09-11T02:51:01.300 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.300 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.300 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-11T02:51:01.301 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.301 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.301 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.301 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.302 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s pool 2024-09-11T02:51:01.302 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' pool' 2024-09-11T02:51:01.302 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ ec_io: /' 2024-09-11T02:51:01.304 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ misc: /' 2024-09-11T02:51:01.304 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_misc.log 2024-09-11T02:51:01.304 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_misc 2024-09-11T02:51:01.317 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.322 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo pool 2024-09-11T02:51:01.323 INFO:tasks.workunit.client.0.smithi190.stdout:test pool on pid 29087 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=pool 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29087 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test pool on pid 29087' 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29087 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s read_operations 2024-09-11T02:51:01.324 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' read_operations' 2024-09-11T02:51:01.325 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-11T02:51:01.327 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.327 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.327 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.327 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.328 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_pool.log 2024-09-11T02:51:01.328 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_pool 2024-09-11T02:51:01.328 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ pool: /' 2024-09-11T02:51:01.330 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo read_operations 2024-09-11T02:51:01.330 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.332 INFO:tasks.workunit.client.0.smithi190.stdout:test read_operations on pid 29118 2024-09-11T02:51:01.333 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=read_operations 2024-09-11T02:51:01.333 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29118 2024-09-11T02:51:01.333 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test read_operations on pid 29118' 2024-09-11T02:51:01.333 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29118 2024-09-11T02:51:01.333 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.333 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.333 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-11T02:51:01.334 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s snapshots 2024-09-11T02:51:01.335 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.335 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.335 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.335 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.335 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' snapshots' 2024-09-11T02:51:01.335 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_read_operations 2024-09-11T02:51:01.335 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ read_operations: /' 2024-09-11T02:51:01.336 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo snapshots 2024-09-11T02:51:01.336 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.337 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=snapshots 2024-09-11T02:51:01.337 INFO:tasks.workunit.client.0.smithi190.stdout:test snapshots on pid 29137 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29137 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test snapshots on pid 29137' 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29137 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s watch_notify 2024-09-11T02:51:01.338 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' watch_notify' 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_snapshots 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-11T02:51:01.340 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ snapshots: /' 2024-09-11T02:51:01.341 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.344 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo watch_notify 2024-09-11T02:51:01.344 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=watch_notify 2024-09-11T02:51:01.345 INFO:tasks.workunit.client.0.smithi190.stdout:test watch_notify on pid 29158 2024-09-11T02:51:01.345 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29158 2024-09-11T02:51:01.345 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test watch_notify on pid 29158' 2024-09-11T02:51:01.345 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29158 2024-09-11T02:51:01.345 INFO:tasks.workunit.client.0.smithi190.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-11T02:51:01.346 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.346 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-11T02:51:01.346 INFO:tasks.workunit.client.0.smithi190.stderr:++ printf %25s write_operations 2024-09-11T02:51:01.346 INFO:tasks.workunit.client.0.smithi190.stderr:+ r=' write_operations' 2024-09-11T02:51:01.347 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.347 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.347 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.347 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.347 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_watch_notify 2024-09-11T02:51:01.348 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-11T02:51:01.348 INFO:tasks.workunit.client.0.smithi190.stderr:++ awk '{print $1}' 2024-09-11T02:51:01.354 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ watch_notify: /' 2024-09-11T02:51:01.355 INFO:tasks.workunit.client.0.smithi190.stderr:++ echo write_operations 2024-09-11T02:51:01.356 INFO:tasks.workunit.client.0.smithi190.stderr:+ ff=write_operations 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stdout:test write_operations on pid 29190 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29190 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ echo 'test write_operations on pid 29190' 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ pids[$f]=29190 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ ret=0 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' 1 -eq 1 ']' 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28327 2024-09-11T02:51:01.357 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28327 2024-09-11T02:51:01.358 INFO:tasks.workunit.client.0.smithi190.stderr:+ '[' -z '' ']' 2024-09-11T02:51:01.358 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.358 INFO:tasks.workunit.client.0.smithi190.stderr:+ case $- in 2024-09-11T02:51:01.358 INFO:tasks.workunit.client.0.smithi190.stderr:+ return 2024-09-11T02:51:01.359 INFO:tasks.workunit.client.0.smithi190.stderr:+ ceph_test_neorados_write_operations 2024-09-11T02:51:01.359 INFO:tasks.workunit.client.0.smithi190.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-11T02:51:01.373 INFO:tasks.workunit.client.0.smithi190.stderr:+ sed 's/^/ write_operations: /' 2024-09-11T02:51:01.425 INFO:tasks.workunit.client.0.smithi190.stdout:51:01.053+0000 7f06a77fe640 10 monclient: get_auth_request con 0x7f069003c4d0 auth_method 0 2024-09-11T02:51:01.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a77fe640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f069003c4d0 0x7f069003e990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f069c000c10 tx=0x7f069c002000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:51:01.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 10 client.4571.objecter ms_handle_connect 0x7f069003c4d0 2024-09-11T02:51:01.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f06a004da40 con 0x563eb4de5ed0 2024-09-11T02:51:01.425 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_map(20..20 src has 1..20) 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 3 client.4571.objecter handle_osd_map got epochs [20,20] > 0 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 3 client.4571.objecter handle_osd_map decoding full epoch 20 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06ae12b880 10 librados: wait_for_osdmap done waiting 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06ae12b880 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f069c0123e0 con 0x563eb4de5ed0 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f06a001ab70 con 0x563eb4de5ed0 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a4ff9640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06a5ffb640 10 client.4571.objecter _get_latest_version latest 20, have it 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06ae12b880 10 client.4571.objecter create_pool name=ceph_test_rados_api_asio 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06ae12b880 10 client.4571.objecter pool_op_submit 1 2024-09-11T02:51:01.426 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06ae12b880 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.057+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x563eb4cde880 con 0x563eb4de5ed0 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 7 ==== pool_op_reply(tid 1 (0) Success v21) ==== 43+0+0 (secure 0 0 0) 0x7f06a0018360 con 0x563eb4de5ed0 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 pool_op_reply(tid 1 (0) Success v21) 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 10 client.4571.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v21) 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 10 client.4571.objecter have request 1 at 0x563eb4de5a80 Op: create 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 20 client.4571.objecter waiting for client to reach epoch 21 before calling back 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:51:01.427 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 10 monclient: _renew_subs 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=21}) -- 0x7f0690041ca0 con 0x563eb4de5ed0 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.425+0000 7f06a4ff9640 10 client.4571.objecter done 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== mon.8 v2:172.21.15.98:3304/0 8 ==== osd_map(21..21 src has 1..21) ==== 12748+0+0 (secure 0 0 0) 0x7f06a0019de0 con 0x563eb4de5ed0 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06a4ff9640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_map(21..21 src has 1..21) 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06a4ff9640 3 client.4571.objecter handle_osd_map got epochs [21,21] > 20 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06a4ff9640 3 client.4571.objecter handle_osd_map decoding incremental epoch 21 2024-09-11T02:51:01.436 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06a4ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 10 librados: set snap read head -> 1 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 10 librados: writefull oid=exist nspace= 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4de8560 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 21 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x563eb4e8ffc0 0x563eb4e923b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 2 osd.1 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 2 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 15 client.4571.objecter _send_op 2 to 6.1 on osd.1 2024-09-11T02:51:01.437 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:2 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x563eb4e928f0 con 0x563eb4e8ffc0 2024-09-11T02:51:01.438 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:01.438 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.429+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:01.438 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.433+0000 7f06a6ffd640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x563eb4e8ffc0 0x563eb4e923b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:51:01.438 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.433+0000 7f06a6ffd640 10 monclient: get_auth_request con 0x563eb4e8ffc0 auth_method 0 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.437+0000 7f06a6ffd640 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x563eb4e8ffc0 0x563eb4e923b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:01.437+0000 7f06a4ff9640 10 client.4571.objecter ms_handle_connect 0x563eb4e8ffc0 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 1 ==== osd_map(22..22 src has 1..22) ==== 9130+0+0 (crc 0 0 0) 0x7f06980069c0 con 0x563eb4e8ffc0 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_map(22..22 src has 1..22) 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 3 client.4571.objecter handle_osd_map got epochs [22,22] > 21 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 3 client.4571.objecter handle_osd_map decoding incremental epoch 22 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 10 client.4571.objecter checking op 2 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:02.774 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.649+0000 7f06a4ff9640 20 client.4571.objecter 2 6.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (3 ms) 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (2 ms) 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2 ms) 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (19 ms) 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (7 ms) 2024-09-11T02:51:02.775 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3 ms) 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (3 ms) 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (2 ms) 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (21 ms) 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (4 ms) 2024-09-11T02:51:02.776 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (11 ms) 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] 12 tests from AsioRados (78 ms total) 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [----------] Global test environment tear-down 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 2 ==== osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 2 ondisk uv 1 in 6.1 attempt 0 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 2 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 2 2024-09-11T02:51:02.777 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 2 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 librados: Objecter returned from writefull r=0 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4de8560 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.778 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 3 osd.1 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 3 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _send_op 3 to 6.1 on osd.1 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:3 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4e8efb0 con 0x563eb4e8ffc0 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4e8f350 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 4 osd.1 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.779 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 4 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _send_op 4 to 6.1 on osd.1 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:4 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4e8f800 con 0x563eb4e8ffc0 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 5 client.4571.objecter 2 in flight 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 3 ondisk uv 1 in 6.1 attempt 0 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 5 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 3 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 3 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 3 2024-09-11T02:51:02.780 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 4 ondisk uv 0 in 6.1 attempt 0 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 4 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 4 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 4 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4e8f350 2024-09-11T02:51:02.781 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 5 osd.1 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 5 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _send_op 5 to 6.1 on osd.1 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:5 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4e8fba0 con 0x563eb4e8ffc0 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4de8560 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.782 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 6 osd.1 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 6 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 15 client.4571.objecter _send_op 6 to 6.1 on osd.1 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:6 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4ded200 con 0x563eb4e8ffc0 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.689+0000 7f06ae12b880 5 client.4571.objecter 2 in flight 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 5 ondisk uv 1 in 6.1 attempt 0 2024-09-11T02:51:02.783 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 5 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 5 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 5 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 5 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 6 ondisk uv 0 in 6.1 attempt 0 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:51:02.784 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 6 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 6 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 6 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4de8560 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [read 0~256] tid 7 osd.1 2024-09-11T02:51:02.785 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 7 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 15 client.4571.objecter _send_op 7 to 6.1 on osd.1 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:7 6.1 6:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4e8f350 con 0x563eb4e8ffc0 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T02:51:02.786 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid noexist '@6' '@6' [read 0~256] tid 8 osd.1 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 8 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 15 client.4571.objecter _send_op 8 to 6.1 on osd.1 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:8 6.1 6:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dedaa0 con 0x563eb4e8ffc0 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 5 client.4571.objecter 2 in flight 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 7 ondisk uv 1 in 6.1 attempt 0 2024-09-11T02:51:02.787 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 5 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 7 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 7 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 7 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 8 ondisk uv 0 in 6.1 attempt 0 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.788 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 8 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 8 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 8 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 librados: queue_aio_write 0x563eb4ca3530 completion 0x563eb4ddf120 write_seq 1 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 9 osd.1 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.789 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.693+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 9 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.697+0000 7f06ae12b880 15 client.4571.objecter _send_op 9 to 6.1 on osd.1 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.697+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:9 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x563eb4de8560 con 0x563eb4e8ffc0 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.697+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.697+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.697+0000 7f06ae12b880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 9 ==== osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 9 ondisk uv 2 in 6.1 attempt 0 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 9 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 9 2024-09-11T02:51:02.790 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 9 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06a6ffd640 20 librados: complete_aio_write 0x563eb4ddf120 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 librados: queue_aio_write 0x563eb4ca3530 completion 0x563eb4debcf0 write_seq 2 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 10 osd.1 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.791 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 10 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 15 client.4571.objecter _send_op 10 to 6.1 on osd.1 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:10 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x563eb4de8560 con 0x563eb4e8ffc0 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.713+0000 7f06ae12b880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 10 ==== osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 10 ondisk uv 3 in 6.1 attempt 0 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 10 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 10 2024-09-11T02:51:02.792 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 10 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06a6ffd640 20 librados: complete_aio_write 0x563eb4debcf0 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 librados: queue_aio_write 0x563eb4ca3530 completion 0x563eb4de0310 write_seq 3 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [write 0~5 in=5b] tid 11 osd.1 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.793 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 11 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 15 client.4571.objecter _send_op 11 to 6.1 on osd.1 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:11 6.1 6:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x563eb4de8560 con 0x563eb4e8ffc0 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.721+0000 7f06ae12b880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 11 ==== osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 11 ondisk uv 4 in 6.1 attempt 0 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 11 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 11 2024-09-11T02:51:02.794 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 11 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 20 librados: complete_aio_write 0x563eb4de0310 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 12 osd.1 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 12 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _send_op 12 to 6.1 on osd.1 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:12 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4de8560 con 0x563eb4e8ffc0 2024-09-11T02:51:02.795 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4dedfa0 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 13 osd.1 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 13 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _send_op 13 to 6.1 on osd.1 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:13 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dee450 con 0x563eb4e8ffc0 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.796 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 5 client.4571.objecter 2 in flight 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 12 ondisk uv 4 in 6.1 attempt 0 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 5 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 12 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 12 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 12 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:51:02.797 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 13 ondisk uv 0 in 6.1 attempt 0 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 13 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 13 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 13 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.798 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 14 osd.1 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 14 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _send_op 14 to 6.1 on osd.1 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:14 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dee840 con 0x563eb4e8ffc0 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4dedfa0 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 15 osd.1 2024-09-11T02:51:02.799 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 15 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 15 client.4571.objecter _send_op 15 to 6.1 on osd.1 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:15 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dee450 con 0x563eb4e8ffc0 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.725+0000 7f06ae12b880 5 client.4571.objecter 2 in flight 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 14 ondisk uv 4 in 6.1 attempt 0 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 5 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 14 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 14 2024-09-11T02:51:02.800 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 14 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 15 ondisk uv 0 in 6.1 attempt 0 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 15 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 15 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 15 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.801 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [read 0~0] tid 16 osd.1 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 16 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 15 client.4571.objecter _send_op 16 to 6.1 on osd.1 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:16 6.1 6:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dee840 con 0x563eb4e8ffc0 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4dedfa0 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base noexist @6 precalc_pgid 0 pgid 0.0 is_read 2024-09-11T02:51:02.802 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _calc_target target noexist @6 -> pgid 6.a7bbaa1 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.a7bbaa1 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid noexist '@6' '@6' [read 0~0] tid 17 osd.1 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 17 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 15 client.4571.objecter _send_op 17 to 6.1 on osd.1 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:17 6.1 6:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dee450 con 0x563eb4e8ffc0 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 5 client.4571.objecter 2 in flight 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 16 ondisk uv 4 in 6.1 attempt 0 2024-09-11T02:51:02.803 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 5 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 16 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 16 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 5 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 16 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 17 ondisk uv 0 in 6.1 attempt 0 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.804 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 17 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 17 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 17 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 librados: queue_aio_write 0x563eb4ca3530 completion 0x563eb4de8780 write_seq 4 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 18 osd.1 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 18 2024-09-11T02:51:02.805 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 15 client.4571.objecter _send_op 18 to 6.1 on osd.1 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:18 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dee840 con 0x563eb4e8ffc0 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.729+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 18 ==== osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 18 ondisk uv 5 in 6.1 attempt 0 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 18 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 18 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.806 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 18 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06a6ffd640 20 librados: complete_aio_write 0x563eb4de8780 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 20 librados: queue_aio_write 0x563eb4ca3530 completion 0x563eb4dedfa0 write_seq 5 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4ded5f0 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 19 osd.1 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 19 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 15 client.4571.objecter _send_op 19 to 6.1 on osd.1 2024-09-11T02:51:02.807 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:19 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x563eb4dee0b0 con 0x563eb4e8ffc0 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.749+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 19 ==== osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 19 ondisk uv 6 in 6.1 attempt 0 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 19 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 19 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 19 2024-09-11T02:51:02.808 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06a6ffd640 20 librados: complete_aio_write 0x563eb4dedfa0 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 20 librados: queue_aio_write 0x563eb4ca3530 completion 0x563eb4dee550 write_seq 6 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 10 client.4571.objecter _op_submit op 0x563eb4dedfa0 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 20 client.4571.objecter _calc_target epoch 22 base exist @6 precalc_pgid 0 pgid 0.0 is_write 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 20 client.4571.objecter _calc_target target exist @6 -> pgid 6.d3a59201 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 10 client.4571.objecter _calc_target raw pgid 6.d3a59201 -> actual 6.1 acting [1,3] primary 1 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 20 client.4571.objecter _get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 10 client.4571.objecter _op_submit oid exist '@6' '@6' [writefull 0~5 in=5b] tid 20 osd.1 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 20 client.4571.objecter get_session s=0x563eb4ddefb0 osd=1 3 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 15 client.4571.objecter _session_op_assign 1 20 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 15 client.4571.objecter _send_op 20 to 6.1 on osd.1 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- osd_op(unknown.0.0:20 6.1 6:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x563eb4ded5f0 con 0x563eb4e8ffc0 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.809 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.753+0000 7f06ae12b880 5 client.4571.objecter 1 in flight 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 1 -- 172.21.15.190:0/4261071606 <== osd.1 v2:172.21.15.98:6808/2313933591 20 ==== osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f06980a8070 con 0x563eb4e8ffc0 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 10 client.4571.objecter ms_dispatch 0x563eb4cadf00 osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 10 client.4571.objecter in handle_osd_op_reply 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 7 client.4571.objecter handle_osd_op_reply 20 ondisk uv 7 in 6.1 attempt 0 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 10 client.4571.objecter op 0 rval 0 len 0 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 15 client.4571.objecter handle_osd_op_reply completed tid 20 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 15 client.4571.objecter _finish_op 20 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 4 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 15 client.4571.objecter _session_op_remove 1 20 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 5 client.4571.objecter 0 in flight 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a6ffd640 20 librados: complete_aio_write 0x563eb4dee550 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 10 librados: watch_flush enter 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 10 librados: watch_flush exit 2024-09-11T02:51:02.810 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 10 client.4571.objecter close_session for osd.1 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x563eb4e8ffc0 msgr2=0x563eb4e923b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x563eb4e8ffc0 0x563eb4e923b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 20 client.4571.objecter put_session s=0x563eb4ddefb0 osd=1 1 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 20 client.4571.objecter shutdown clearing up homeless session... 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 10 client.4571.objecter successfully canceled tick 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f069003c4d0 msgr2=0x7f069003e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f069003c4d0 0x7f069003e990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f069c000c10 tx=0x7f069c002000 comp rx=0 tx=0).stop 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 10 monclient: shutdown 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:51:02.811 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4de5ed0 msgr2=0x563eb4de8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 --2- 172.21.15.190:0/4261071606 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x563eb4de5ed0 0x563eb4de8020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f06a0005e40 tx=0x7f06a0008430 comp rx=0 tx=0).stop 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06a7fff640 1 -- 172.21.15.190:0/4261071606 reap_dead start 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 shutdown_connections 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 >> 172.21.15.190:0/4261071606 conn(0x563eb4d2fd50 msgr2=0x563eb4d5b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 shutdown_connections 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 -- 172.21.15.190:0/4261071606 wait complete. 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: 2024-09-11T02:51:02.765+0000 7f06ae12b880 1 librados: shutdown 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1725 ms total) 2024-09-11T02:51:02.812 INFO:tasks.workunit.client.0.smithi190.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] Global test environment set-up. 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjects (353 ms) 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (36 ms) 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo2,foo3 2024-09-11T02:51:03.212 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo2 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo3 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo4,foo5 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo4 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo5 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo6,foo7 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo7 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo6 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo4 2024-09-11T02:51:03.213 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo5 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo7 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo6 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo1 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo1 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo2 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo3 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (108 ms) 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 1 0 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 10 0 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 13 0 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 7 0 2024-09-11T02:51:03.214 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 14 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 0 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 15 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 11 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 5 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 8 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 6 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 3 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 4 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 12 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 9 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2 0 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (123 ms) 2024-09-11T02:51:03.215 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: x cursor=MIN 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=1 cursor=10:02547ec2:::1:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=10 cursor=10:52ea6a34:::10:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=13 cursor=10:566253c9:::13:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=7 cursor=10:5c6b0b28:::7:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=14 cursor=10:62a1935d:::14:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=0 cursor=10:6cac518f:::0:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=15 cursor=10:863748b0:::15:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=11 cursor=10:89d3ae78:::11:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=5 cursor=10:b29083e3:::5:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=8 cursor=10:bd63b0f1:::8:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=6 cursor=10:c4fdafeb:::6:head 2024-09-11T02:51:03.216 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=3 cursor=10:cfc208b3:::3:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=4 cursor=10:d83876eb:::4:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=12 cursor=10:de5d7c5f:::12:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=9 cursor=10:e960b815:::9:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > oid=2 cursor=10:f905c69b:::2:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=1 cursor=10:02547ec2:::1:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=10 cursor=10:52ea6a34:::10:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=13 cursor=10:566253c9:::13:head 2024-09-11T02:51:03.217 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:566253c9:::13:head 2024-09-11T02:51:03.218 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-09-11T02:51:03.218 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=7 cursor=10:5c6b0b28:::7:head 2024-09-11T02:51:03.218 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-09-11T02:51:03.218 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=14 cursor=10:62a1935d:::14:head 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=0 cursor=10:6cac518f:::0:head 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=15 cursor=10:863748b0:::15:head 2024-09-11T02:51:03.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:863748b0:::15:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=11 cursor=10:89d3ae78:::11:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=5 cursor=10:b29083e3:::5:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=8 cursor=10:bd63b0f1:::8:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=6 cursor=10:c4fdafeb:::6:head 2024-09-11T02:51:03.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=3 cursor=10:cfc208b3:::3:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=4 cursor=10:d83876eb:::4:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=12 cursor=10:de5d7c5f:::12:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=9 cursor=10:e960b815:::9:head 2024-09-11T02:51:03.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:e960b815:::9:head 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : oid=2 cursor=10:f905c69b:::2:head 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:863748b0:::15:head 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:863748b0:::15:head expected=10:863748b0:::15:head 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=15 expected=15 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:6cac518f:::0:head expected=10:6cac518f:::0:head 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=0 expected=0 2024-09-11T02:51:03.313 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:62a1935d:::14:head expected=10:62a1935d:::14:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=14 expected=14 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:5c6b0b28:::7:head expected=10:5c6b0b28:::7:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=7 expected=7 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:566253c9:::13:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:566253c9:::13:head expected=10:566253c9:::13:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=13 expected=13 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:52ea6a34:::10:head expected=10:52ea6a34:::10:head 2024-09-11T02:51:03.314 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=10 expected=10 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:f905c69b:::2:head expected=10:f905c69b:::2:head 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=2 expected=2 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:e960b815:::9:head 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:e960b815:::9:head expected=10:e960b815:::9:head 2024-09-11T02:51:03.315 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: api_ handler_error: Running main() from gmock_main.cc 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] Global test environment set-up. 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2183 ms) 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] 1 test from neocls_handler_error (2183 ms total) 2024-09-11T02:51:03.458 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: 2024-09-11T02:51:03.459 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [----------] Global test environment tear-down 2024-09-11T02:51:03.459 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2183 ms total) 2024-09-11T02:51:03.459 INFO:tasks.workunit.client.0.smithi190.stdout: handler_error: [ PASSED ] 1 test. 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (302 ms) 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-11T02:51:03.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (17 ms) 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (16 ms) 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (23 ms) 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (18 ms) 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (14 ms) 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (12 ms) 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (18 ms) 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-11T02:51:03.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (8 ms) 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (13 ms) 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (18 ms) 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (27 ms) 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (22 ms) 2024-09-11T02:51:03.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-11T02:51:03.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (8 ms) 2024-09-11T02:51:03.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (526 ms total) 2024-09-11T02:51:03.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: 2024-09-11T02:51:03.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-11T02:51:03.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2277 ms total) 2024-09-11T02:51:03.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (79 ms) 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (59 ms) 2024-09-11T02:51:03.474 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-11T02:51:03.475 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2172 ms) 2024-09-11T02:51:03.475 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2310 ms total) 2024-09-11T02:51:03.475 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: 2024-09-11T02:51:03.475 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-11T02:51:03.475 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2310 ms total) 2024-09-11T02:51:03.475 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-11T02:51:03.488 INFO:tasks.workunit.client.0.smithi190.stdout: cls: Running main() from gmock_main.cc 2024-09-11T02:51:03.488 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-09-11T02:51:03.488 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] Global test environment set-up. 2024-09-11T02:51:03.488 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] 1 test from NeoRadosCls 2024-09-11T02:51:03.488 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-11T02:51:03.488 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [ OK ] NeoRadosCls.DNE (2255 ms) 2024-09-11T02:51:03.489 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] 1 test from NeoRadosCls (2255 ms total) 2024-09-11T02:51:03.489 INFO:tasks.workunit.client.0.smithi190.stdout: cls: 2024-09-11T02:51:03.489 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [----------] Global test environment tear-down 2024-09-11T02:51:03.489 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [==========] 1 test from 1 test suite ran. (2255 ms total) 2024-09-11T02:51:03.489 INFO:tasks.workunit.client.0.smithi190.stdout: cls: [ PASSED ] 1 test. 2024-09-11T02:51:03.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (65 ms) 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (58 ms) 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2205 ms) 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.411940+0000 mon.f [WRN] Health check failed: 31 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438075+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28557-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438236+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi190-28185-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438351+0000 mon.f [INF] from='client.? 172.21.15.190:0/990162768' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi190-28177-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438446+0000 mon.f [INF] from='client.? 172.21.15.190:0/1284239316' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi190-28475-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438549+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi190-28169-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438657+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi190-28350-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438782+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi190-28383-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.438958+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.439133+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi190-28434-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.439333+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi190-28228-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.439526+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi190-28161-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:03.570 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.439705+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28497-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.071 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd list_parallel: process_1_[28740]: starting. 2024-09-11T02:51:04.071 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28740]: creating pool ceph_test_rados_list_parallel.smithi190-28705 2024-09-11T02:51:04.071 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28740]: created object 0... 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28740]: created object 25... 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28740]: created object 49... 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28740]: finishing. 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_1_[28740]: shutting down. 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28741]: starting. 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28741]: listing objects. 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28741]: listed object 0... 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28741]: listed object 25... 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28741]: saw 50 objects 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_2_[28741]: shutting down. 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29395]: starting. 2024-09-11T02:51:04.072 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29395]: creating pool ceph_test_rados_list_parallel.smithi190-28705 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29395]: created object 0... 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29395]: created object 25... 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29395]: created object 49... 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29395]: finishing. 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_3_[29395]: shutting down. 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29396]: starting. 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29396]: listing objects. 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29396]: listed object 0... 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29396]: listed object 25... 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29396]: saw 46 objects 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_4_[29396]: shutting down. 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.073 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29397]: starting. 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29397]: removed 25 objects... 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29397]: removed half of the objects 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29397]: removed 50 objects... 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29397]: removed 50 objects 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_5_[29397]: shutting down. 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29446]: starting. 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29446]: creating pool ceph_test_rados_list_parallel.smithi190-28705 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29446]: created object 0... 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29446]: created object 25... 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29446]: created object 49... 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29446]: finishing. 2024-09-11T02:51:04.074 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_6_[29446]: shutting down. 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29447]: starting. 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29447]: listing objects. 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29447]: listed object 0... 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29447]: listed object 25... 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29447]: listed object 50... 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29447]: saw 53 objects 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_7_[29447]: shutting down. 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29448]: starting. 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29448]: added 25 objects... 2024-09-11T02:51:04.075 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29448]: added half of the objects 2024-09-11T02:51:04.076 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29448]: added 50 objects... 2024-09-11T02:51:04.076 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29448]: added 50 objects 2024-09-11T02:51:04.076 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_8_[29448]: shutting down. 2024-09-11T02:51:04.076 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.076 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.437 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29752]: starting. 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29752]: creating pool ceph_test_rados_list_parallel.smithi190-28705 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29752]: created object 0... 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29752]: created object 25... 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29752]: created object 49... 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29752]: finishing. 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_9_[29752]: shutting down. 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.438 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: starting. 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: listing objects. 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: listed object 0... 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: listed object 25... 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: listed object 50... 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: listed object 75... 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: saw 99 objects 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_10_[29753]: shutting down. 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29756]: starting. 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29756]: removed 25 objects... 2024-09-11T02:51:04.439 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29756]: removed half of the objects 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29756]: removed 50 objects... 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29756]: removed 50 objects 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_13_[29756]: shutting down. 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29754]: starting. 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29754]: added 25 objects... 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29754]: added half of the objects 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29754]: added 50 objects... 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29754]: added 50 objects 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_11_[29754]: shutting down. 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.440 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29755]: starting. 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29755]: added 25 objects... 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29755]: added half of the objects 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29755]: added 50 objects... 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29755]: added 50 objects 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_12_[29755]: shutting down. 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29837]: starting. 2024-09-11T02:51:04.441 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29837]: creating pool ceph_test_rados_list_parallel.smithi190-28705 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29837]: created object 0... 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29837]: created object 25... 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29837]: created object 49... 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29837]: finishing. 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_14_[29837]: shutting down. 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: starting. 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: listing objects. 2024-09-11T02:51:04.442 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: listed object 0... 2024-09-11T02:51:04.466 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: listed object 2: got: 2024-09-11T02:51:02.439894+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi190-28680-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.466 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.440103+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi190-28209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.466 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.440285+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi190-28420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.466 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.440468+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi190-28403-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.466 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.440644+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28676-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.466 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.440825+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi190-28198-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:02.441013+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi190-28279-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.451446+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi190-29012-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi190-29012-1"}]': finished 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.451560+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi190-28960-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi190-28960-1"}]': finished 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.451652+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi190-28465-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi190-28465-1"}]': finished 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.465556+0000 mon.f [INF] from='client.? 172.21.15.190:0/1952286386' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28350-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.569735+0000 mon.h [INF] from='client.? 172.21.15.190:0/1958773476' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.571796+0000 client.admin [INF] onexx 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.572225+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.603862+0000 mon.g [INF] from='client.? 172.21.15.190:0/870338235' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi190-28420-7"} : dispatch 2024-09-11T02:51:04.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.605231+0000 mon.b [INF] from='client.? 172.21.15.190:0/3950920622' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi190-28434-7"} : dispatch 2024-09-11T02:51:05.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.609853+0000 m5... 2024-09-11T02:51:05.543 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: listed object 50... 2024-09-11T02:51:05.543 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: listed object 75... 2024-09-11T02:51:05.543 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: listed object 100... 2024-09-11T02:51:05.543 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: listed object 125... 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: saw 150 objects 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_15_[29838]: shutting down. 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29839]: starting. 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29839]: added 25 objects... 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29839]: added half of the objects 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29839]: added 50 objects... 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29839]: added 50 objects 2024-09-11T02:51:05.544 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: process_16_[29839]: shutting down. 2024-09-11T02:51:05.545 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.545 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.545 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.545 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.545 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******************************* 2024-09-11T02:51:05.545 INFO:tasks.workunit.client.0.smithi190.stdout: list_parallel: ******* SUCCESS ********** 2024-09-11T02:51:06.513 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28744]: starting. 2024-09-11T02:51:06.513 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28744]: creating pool ceph_test_rados_delete_pools_parallel.smithi190-28728 2024-09-11T02:51:06.513 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28744]: created object 0... 2024-09-11T02:51:06.513 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28744]: created object 25... 2024-09-11T02:51:06.513 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28744]: created object 49... 2024-09-11T02:51:06.513 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28744]: finishing. 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_1_[28744]: shutting down. 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_2_[28746]: starting. 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_2_[28746]: deleting pool ceph_test_rados_delete_pools_parallel.smithi190-28728 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_2_[28746]: shutting down. 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29497]: starting. 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29497]: creating pool ceph_test_rados_delete_pools_parallel.smithi190-28728 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29497]: created object 0... 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29497]: created object 25... 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29497]: created object 49... 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29497]: finishing. 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_3_[29497]: shutting down. 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T02:51:06.514 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29499]: starting. 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29499]: listing objects. 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29499]: listed object 0... 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29499]: listed object 25... 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29499]: saw 50 objects 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_5_[29499]: shutting down. 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_4_[29498]: starting. 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_4_[29498]: deleting pool ceph_test_rados_delete_pools_parallel.smithi190-28728 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: process_4_[29498]: shutting down. 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******************************* 2024-09-11T02:51:06.515 INFO:tasks.workunit.client.0.smithi190.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-11T02:51:06.524 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28755]: starting. 2024-09-11T02:51:06.524 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28755]: creating pool ceph_test_rados_open_pools_parallel.smithi190-28713 2024-09-11T02:51:06.524 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28755]: created object 0... 2024-09-11T02:51:06.524 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28755]: created object 25... 2024-09-11T02:51:06.524 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28755]: created object 49... 2024-09-11T02:51:06.524 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28755]: finishing. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_1_[28755]: shutting down. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28756]: starting. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28756]: rados_pool_create. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28756]: rados_ioctx_create. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_2_[28756]: shutting down. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29500]: starting. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29500]: creating pool ceph_test_rados_open_pools_parallel.smithi190-28713 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29500]: created object 0... 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29500]: created object 25... 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29500]: created object 49... 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29500]: finishing. 2024-09-11T02:51:06.525 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_3_[29500]: shutting down. 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29501]: starting. 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29501]: rados_pool_create. 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29501]: rados_ioctx_create. 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: process_4_[29501]: shutting down. 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******************************* 2024-09-11T02:51:06.526 INFO:tasks.workunit.client.0.smithi190.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: Running main() from gmock_main.cc 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] Global test environment set-up. 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1205 ms) 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-11T02:51:07.555 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (1995 ms) 2024-09-11T02:51:07.556 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-11T02:51:07.556 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3076 ms) 2024-09-11T02:51:07.556 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6276 ms total) 2024-09-11T02:51:07.556 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: 2024-09-11T02:51:07.556 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [----------] Global test environment tear-down 2024-09-11T02:51:07.556 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6277 ms total) 2024-09-11T02:51:07.556 INFO:tasks.workunit.client.0.smithi190.stdout: cmd: [ PASSED ] 3 tests. 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout:on.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi190-28434-7"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.612816+0000 mon.b [INF] from='client.? 172.21.15.190:0/3950920622' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi190-28434-7"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.613580+0000 mon.c [INF] from='client.? 172.21.15.190:0/3406615483' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi190-28177-16"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.631060+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi190-28434-7"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.631207+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi190-28420-7"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.631338+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi190-28177-16"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.631930+0000 mon.c [INF] from='client.? 172.21.15.190:0/3406615483' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi190-28177-16"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.634140+0000 mon.g [INF] from='client.? 172.21.15.190:0/870338235' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi190-28420-7"} : dispatch 2024-09-11T02:51:07.565 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.635371+0000 mon.b [INF] from='client.? 172.21.15.190:0/3950920622' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi190-28434-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.636552+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi190-28177-16"} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.638672+0000 mon.c [INF] from='client.? 172.21.15.190:0/3406615483' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi190-28177-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.643953+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi190-28420-7"} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.658311+0000 mon.g [INF] from='client.? 172.21.15.190:0/870338235' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi190-28420-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.661279+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi190-28177-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.701566+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi190-28434-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:03.704331+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi190-28420-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.479029+0000 mon.f [INF] from='client.? 172.21.15.190:0/1953733000' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi190-28169-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:07.566 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.479332+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi190-29012-1"}]': finished 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.479632+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28676-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.479864+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi190-28161-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.480089+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28363-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.480291+0000 mon.f [INF] from='client.? 172.21.15.190:0/72973087' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi190-28185-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.510046+0000 mon.c [INF] from='client.? 172.21.15.190:0/1506507037' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi190-29012-1"} : dispatch 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.536950+0000 mon.e [INF] from='client.? 172.21.15.190:0/3168454963' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi190-28465-1"} : dispatch 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.543533+0000 mon.f [INF] from='client.? 172.21.15.190:0/72973087' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi190-28185-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi190-28185-24"} : dispatch 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.544367+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi190-29012-1"} : dispatch 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.544744+0000 mon.i [INF] from='client.? 172.21.15.190:0/1189949083' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi190-28960-1"} : dispatch 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.545349+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi190-28465-1"} : dispatch 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.562849+0000 mon.f [INF] from='client.? 172.21.15.190:0/1952286386' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28350-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.563861+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-11T02:51:07.567 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.577168+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi190-28960-1"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.613948+0000 mon.h [INF] from='client.? 172.21.15.190:0/2406466134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi190-28228-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.615994+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi190-28228-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.618997+0000 mon.h [INF] from='client.? 172.21.15.190:0/2406466134' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi190-28228-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.624731+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi190-28228-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.627816+0000 mon.h [INF] from='client.? 172.21.15.190:0/2406466134' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi190-28228-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.628678+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi190-28228-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.630259+0000 mon.g [INF] from='client.? 172.21.15.190:0/4081407849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi190-28209-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.631239+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi190-28209-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.634125+0000 mon.g [INF] from='client.? 172.21.15.190:0/4081407849' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi190-28209-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.634900+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi190-28209-10"} : dispatch 2024-09-11T02:51:07.568 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.636144+0000 mon.g [INF] from='client.? 172.21.15.190:0/4081407849' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi190-28209-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:05.636446+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi190-28209-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486186+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi190-28434-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi190-28434-7"}]': finished 2024-09-11T02:51:07.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486283+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi190-28420-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi190-28420-7"}]': finished 2024-09-11T02:51:07.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486362+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi190-28177-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi190-28177-16"}]': finished 2024-09-11T02:51:07.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486433+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi190-29012-1"}]': finished 2024-09-11T02:51:07.569 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486500+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi190-28465-1"}]': finished 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486573+0000 mon.f [INF] from='client.? 172.21.15.190:0/1952286386' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28350-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486634+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi190-28960-1"}]': finished 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486708+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi190-28228-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.486811+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi190-28209-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.494916+0000 mon.g [INF] from='client.? 172.21.15.190:0/4081407849' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi190-28209-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi190-28209-10"} : dispatch 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.497536+0000 mon.i [INF] from='client.? 172.21.15.190:0/1189949083' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi190-28960-1"} : dispatch 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.499913+0000 mon.h [INF] from='client.? 172.21.15.190:0/2406466134' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi190-28228-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi190-28228-10"} : dispatch 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.502972+0000 mon.e [INF] from='client.? 172.21.15.190:0/3168454963' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi190-28465-1"} : dispatch 2024-09-11T02:51:07.656 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.508490+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi190-28209-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi190-28209-10"} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.509067+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi190-28228-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi190-28228-10"} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.509402+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi190-28960-1"} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.509632+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi190-28465-1"} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.521611+0000 mon.f [INF] from='client.? 172.21.15.190:0/1952286386' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi190-28350-4", "tierpool": "test-rados-api-smithi190-28350-4-cache"} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.521949+0000 mon.f [INF] from='client.? 172.21.15.190:0/1953733000' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi190-28169-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.596193+0000 mon.i [INF] from='client.? 172.21.15.190:0/1514635652' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi190-29012-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:06.649971+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi190-29012-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.499497+0000 mon.f [INF] from='client.? 172.21.15.190:0/72973087' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi190-28185-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi190-28185-24"}]': finished 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.499753+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi190-28960-1"}]': finished 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.500003+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi190-28465-1"}]': finished 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.500244+0000 mon.f [INF] from='client.? 172.21.15.190:0/1952286386' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi190-28350-4", "tierpool": "test-rados-api-smithi190-28350-4-cache"}]': finished 2024-09-11T02:51:07.657 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.500420+0000 mon.f [INF] from='client.? 172.21.15.190:0/1953733000' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi190-28169-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.500615+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi190-29012-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.521432+0000 mon.i [INF] from='client.? 172.21.15.190:0/1514635652' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi190-29012-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi190-29012-2"} : dispatch 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.539528+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi190-29012-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi190-29012-2"} : dispatch 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.540251+0000 mon.f [INF] from='client.? 172.21.15.190:0/1952286386' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi190-28350-4", "overlaypool": "test-rados-api-smithi190-28350-4-cache"} : dispatch 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.541599+0000 mon.b [INF] from='client.? 172.21.15.190:0/737975291' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28676-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.546052+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28676-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.549265+0000 mon.h [INF] from='client.? 172.21.15.190:0/3848466600' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi190-28161-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:07.658 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.549594+0000 mon.f [INF] from='client.? 172.21.15.190:0/1953733000' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi190-28169-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-11T02:51:08.686 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.550783+0000 mon.e [INF] from='cl api_io_pp: Running main() from gmock_main.cc 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: seed 28185 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (276 ms) 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (18 ms) 2024-09-11T02:51:08.687 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (993 ms) 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (4 ms) 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-11T02:51:08.688 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (8 ms) 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (13 ms) 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5 ms) 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8 ms) 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (6 ms) 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (7 ms) 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-11T02:51:08.689 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (26 ms) 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (4 ms) 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (1402 ms total) 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-09-11T02:51:08.690 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-11T02:51:08.691 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1130 ms) 2024-09-11T02:51:08.691 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-11T02:51:08.691 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (12 ms) 2024-09-11T02:51:08.691 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-11T02:51:08.691 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2024-09-11T02:51:10.541 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: Running main() from gmock_main.cc 2024-09-11T02:51:10.541 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-11T02:51:10.541 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] Global test environment set-up. 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (300 ms) 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: no timeout :/ 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (56 ms) 2024-09-11T02:51:10.542 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (14 ms) 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.Checksum (10 ms) 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4 ms) 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (4 ms) 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (10 ms) 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (6 ms) 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.TruncTest (12 ms) 2024-09-11T02:51:10.543 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (12 ms) 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (10 ms) 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.RmXattr (22 ms) 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIo.XattrIter (13 ms) 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 14 tests from LibRadosIo (473 ms total) 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-11T02:51:10.544 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1188 ms) 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (16 ms) 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (9 ms) 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (14 ms) 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (12 ms) 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (6 ms) 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (4 ms) 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-11T02:51:10.545 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-09-11T02:51:10.546 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-11T02:51:10.546 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (57 ms) 2024-09-11T02:51:10.546 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-11T02:51:10.546 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (9 ms) 2024-09-11T02:51:10.546 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1320 ms total) 2024-09-11T02:51:10.546 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: 2024-09-11T02:51:10.554 INFO:tasks.workunit.client.0.smithi190.stdout: api_i api_stat: Running main() from gmock_main.cc 2024-09-11T02:51:10.554 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] Global test environment set-up. 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.Stat (308 ms) 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (5 ms) 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.StatNS (14 ms) 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2024-09-11T02:51:10.555 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 5 tests from LibRadosStat (330 ms total) 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1181 ms) 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (20 ms) 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (5 ms) 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1207 ms total) 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: 2024-09-11T02:51:10.556 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [----------] Global test environment tear-down 2024-09-11T02:51:10.557 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9443 ms total) 2024-09-11T02:51:10.557 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-11T02:51:10.560 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-11T02:51:10.560 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-11T02:51:10.560 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-11T02:51:10.560 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: seed 28434 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (303 ms) 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (10 ms) 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (7 ms) 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (27 ms) 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (347 ms total) 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: 2024-09-11T02:51:10.561 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1178 ms) 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (4 ms) 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (22 ms) 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1205 ms total) 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9437 ms total) 2024-09-11T02:51:10.562 INFO:tasks.workunit.client.0.smithi190.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-11T02:51:10.563 INFO:tasks.workunit.client.0.smithi190.stdout:o: [----------] Global test environment tear-down 2024-09-11T02:51:10.563 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9551 ms total) 2024-09-11T02:51:10.563 INFO:tasks.workunit.client.0.smithi190.stdout: api_io: [ PASSED ] 24 tests. 2024-09-11T02:51:10.569 INFO:tasks.workunit.client.0.smithi190.stdout: list: Running main() from gmock_main.cc 2024-09-11T02:51:10.569 INFO:tasks.workunit.client.0.smithi190.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-11T02:51:10.569 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] Global test environment set-up. 2024-09-11T02:51:10.569 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] 3 tests from NeoradosList 2024-09-11T02:51:10.569 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-11T02:51:10.569 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ OK ] NeoradosList.ListObjects (2156 ms) 2024-09-11T02:51:10.569 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3043 ms) 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4051 ms) 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] 3 tests from NeoradosList (9250 ms total) 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: [----------] Global test environment tear-down 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: [==========] 3 tests from 1 test suite ran. (9250 ms total) 2024-09-11T02:51:10.570 INFO:tasks.workunit.client.0.smithi190.stdout: list: [ PASSED ] 3 tests. 2024-09-11T02:51:10.678 INFO:tasks.workunit.client.0.smithi190.stdout:ient.? 172.21.15.190:0/2172620575' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28363-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:10.678 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.551350+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi190-28161-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:10.678 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.553566+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi190-28363-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-11T02:51:10.678 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.569784+0000 mon.h [INF] from='client.? 172.21.15.190:0/1958773476' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-11T02:51:10.678 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.570481+0000 client.admin [INF] threexx 2024-09-11T02:51:10.678 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.570744+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.623593+0000 mon.c [INF] from='client.? 172.21.15.190:0/1442641142' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi190-28960-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: got: 2024-09-11T02:51:07.633716+0000 mon.f [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi190-28960-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7211 ms) 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (9539 ms total) 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (9539 ms total) 2024-09-11T02:51:10.679 INFO:tasks.workunit.client.0.smithi190.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-11T02:51:11.562 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-11T02:51:11.562 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-11T02:51:11.562 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: handle_notify cookie 94580649893408 notify_id 98784247809 notifier_gid 4756 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (3112 ms) 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Trying... 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: handle_notify cookie 94580663021568 notify_id 111669149700 notifier_gid 4917 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Waiting for 3.000000000s 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Timed out. 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Flushing... 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: Flushed... 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7081 ms) 2024-09-11T02:51:11.563 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (10193 ms total) 2024-09-11T02:51:11.564 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: 2024-09-11T02:51:11.564 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-11T02:51:11.564 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (10193 ms total) 2024-09-11T02:51:11.564 INFO:tasks.workunit.client.0.smithi190.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-11T02:51:13.573 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: Running main() from gmock_main.cc 2024-09-11T02:51:13.573 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-11T02:51:13.573 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] Global test environment set-up. 2024-09-11T02:51:13.573 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5175 ms) 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (7064 ms) 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12239 ms total) 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [----------] Global test environment tear-down 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12239 ms total) 2024-09-11T02:51:13.574 INFO:tasks.workunit.client.0.smithi190.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-11T02:51:14.280 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: Running main() from gmock_main.cc 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] Global test environment set-up. 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (315 ms) 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockShared (11 ms) 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1019 ms) 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1008 ms) 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-11T02:51:14.281 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (15 ms) 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.Unlock (6 ms) 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (11 ms) 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (6 ms) 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLock (2391 ms total) 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1162 ms) 2024-09-11T02:51:14.282 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (16 ms) 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1059 ms) 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1023 ms) 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (20 ms) 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (22 ms) 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (20 ms) 2024-09-11T02:51:14.283 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-11T02:51:14.284 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (214 ms) 2024-09-11T02:51:14.284 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3536 ms total) 2024-09-11T02:51:14.284 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: 2024-09-11T02:51:14.284 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [----------] Global test environment tear-down 2024-09-11T02:51:14.284 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13264 ms total) 2024-09-11T02:51:14.284 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: Running main() from gmock_main.cc 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] Global test environment set-up. 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolList (1153 ms) 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-11T02:51:14.308 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (1991 ms) 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2022 ms) 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4038 ms) 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2011 ms) 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (1743 ms) 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] 6 tests from NeoRadosPools (12958 ms total) 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [----------] Global test environment tear-down 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [==========] 6 tests from 1 test suite ran. (12958 ms total) 2024-09-11T02:51:14.309 INFO:tasks.workunit.client.0.smithi190.stdout: pool: [ PASSED ] 6 tests. 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: seed 28228 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (259 ms) 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (31 ms) 2024-09-11T02:51:14.321 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1019 ms) 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1010 ms) 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (8 ms) 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (5 ms) 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (7 ms) 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-11T02:51:14.322 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (6 ms) 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2345 ms total) 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1170 ms) 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (14 ms) 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1048 ms) 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1031 ms) 2024-09-11T02:51:14.323 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (15 ms) 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (20 ms) 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (21 ms) 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (8 ms) 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3327 ms total) 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13288 ms total) 2024-09-11T02:51:14.324 INFO:tasks.workunit.client.0.smithi190.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-11T02:51:16.768 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-11T02:51:16.768 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-11T02:51:16.768 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-11T02:51:16.768 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-11T02:51:16.768 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-11T02:51:16.768 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1168 ms) 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (8 ms) 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1176 ms total) 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (155 ms) 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4003 ms) 2024-09-11T02:51:16.769 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (11 ms) 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (8 ms) 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 141733920770 notifier_gid 4991 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (10 ms) 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 141733920771 notifier_gid 4991 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (7 ms) 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 141733920769 notifier_gid 4991 2024-09-11T02:51:16.770 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (8 ms) 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 141733920772 notifier_gid 4991 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (9 ms) 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 141733920775 notifier_gid 4991 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (11 ms) 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 141733920770 notifier_gid 4991 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (6 ms) 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: trying... 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 141733920771 notifier_gid 4991 2024-09-11T02:51:16.771 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: timed out 2024-09-11T02:51:16.772 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T02:51:19.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_no api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-11T02:51:19.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (7 ms) 2024-09-11T02:51:19.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (6 ms) 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (11 ms) 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (7 ms) 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (13 ms) 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (6 ms) 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (6 ms) 2024-09-11T02:51:19.330 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (9 ms) 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (24 ms) 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (6513 ms) 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1125 ms) 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (13 ms) 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (7 ms) 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-11T02:51:19.331 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-09-11T02:51:19.332 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (8904 ms total) 2024-09-11T02:51:19.332 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: 2024-09-11T02:51:19.332 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-11T02:51:19.332 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (18330 ms total) 2024-09-11T02:51:19.332 INFO:tasks.workunit.client.0.smithi190.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify_test_cb 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (292 ms) 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009719636672 err -107 2024-09-11T02:51:19.334 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (17 ms) 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009719651600 err -107 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (15 ms) 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 94489280512 cookie 94009718976720 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (20 ms) 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 94489280513 cookie 94009718976720 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (16 ms) 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 94489280512 cookie 94009718976720 2024-09-11T02:51:19.335 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (19 ms) 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 94489280513 cookie 94009718976640 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 94489280513 cookie 94009718981184 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (20 ms) 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 94489280514 cookie 94009718976640 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 98784247811 cookie 94009718976640 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3009 ms) 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-11T02:51:19.336 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009718981184 err -107 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_cb from 4663 notify_id 128849018881 cookie 94009718981184 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5018 ms) 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94009718981184 err -107 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1013 ms) 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9439 ms total) 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: watch_notify_test_cb 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1130 ms) 2024-09-11T02:51:19.337 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1130 ms total) 2024-09-11T02:51:19.338 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: 2024-09-11T02:51:19.338 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-11T02:51:19.338 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (18198 ms total) 2024-09-11T02:51:19.338 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-11T02:51:23.433 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: Running main() from gmock_main.cc 2024-09-11T02:51:23.433 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-11T02:51:23.433 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] Global test environment set-up. 2024-09-11T02:51:23.433 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-11T02:51:23.433 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6212 ms) 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6813 ms) 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (9097 ms) 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] 3 tests from NeoradosECList (22122 ms total) 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [----------] Global test environment tear-down 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (22122 ms total) 2024-09-11T02:51:23.434 INFO:tasks.workunit.client.0.smithi190.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: 2024-09-11T02:51:26.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2270 ms) 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3053 ms) 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3061 ms) 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2980 ms) 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2736 ms) 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-11T02:51:26.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3034 ms) 2024-09-11T02:51:26.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-11T02:51:26.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8020 ms) 2024-09-11T02:51:26.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (25154 ms total) 2024-09-11T02:51:26.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: 2024-09-11T02:51:26.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-11T02:51:26.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (25154 ms total) 2024-09-11T02:51:26.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: Running main() from gmock_main.cc 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] Global test environment set-up. 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (3105 ms) 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3064 ms) 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3001 ms) 2024-09-11T02:51:27.398 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3052 ms) 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (2705 ms) 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3017 ms) 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (8072 ms) 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (26016 ms total) 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [----------] Global test environment tear-down 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (26016 ms total) 2024-09-11T02:51:27.399 INFO:tasks.workunit.client.0.smithi190.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] Global test environment set-up. 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5056 ms) 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.RegisterLate (53 ms) 2024-09-11T02:51:29.056 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: cluster: 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: id: 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: health: HEALTH_WARN 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 12 pool(s) do not have an application enabled 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: services: 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mon: 9 daemons, quorum f,b,g,c,h,i,e (age 38s), out of quorum: a, d 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mgr: x(active, since 77s) 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: osd: 6 osds: 6 up (since 71s), 6 in (since 80s) 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: data: 2024-09-11T02:51:29.057 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pools: 25 pools, 745 pgs 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: objects: 16 objects, 577 KiB 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: usage: 165 MiB used, 540 GiB / 540 GiB avail 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pgs: 88.054% pgs unknown 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 1.074% pgs not active 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 656 unknown 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 81 active+clean 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 8 creating+peering 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: io: 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: client: 1.5 KiB/s rd, 6.2 KiB/s wr, 1 op/s rd, 8 op/s wr 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.058 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: cluster: 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: id: 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: health: HEALTH_WARN 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 15 pool(s) do not have an application enabled 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: services: 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mon: 9 daemons, quorum f,b,g,c,h,i,e (age 40s), out of quorum: a, d 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: mgr: x(active, since 79s) 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: osd: 6 osds: 6 up (since 73s), 6 in (since 82s) 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.059 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: data: 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pools: 27 pools, 713 pgs 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: objects: 51 objects, 581 KiB 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: usage: 175 MiB used, 540 GiB / 540 GiB avail 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: pgs: 54.698% pgs unknown 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 5.891% pgs not active 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 390 unknown 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 281 active+clean 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 42 creating+peering 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: io: 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: client: 1.7 KiB/s rd, 23 KiB/s wr, 175 op/s rd, 209 op/s wr 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.060 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2469 ms) 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ OK ] LibRadosService.Status (20277 ms) 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] 4 tests from LibRadosService (27855 ms total) 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [----------] Global test environment tear-down 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27855 ms total) 2024-09-11T02:51:29.061 INFO:tasks.workunit.client.0.smithi190.stdout: api_service: [ PASSED ] 4 tests. 2024-09-11T02:51:29.358 INFO:tasks.workunit.client.0.smithi190.stdout:tify_pp: flushed 2024-09-11T02:51:29.358 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3005 ms) 2024-09-11T02:51:29.358 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: trying... 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 154618822665 notifier_gid 4991 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: timed out 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushed 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3011 ms) 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: List watches 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 167503724554 notifier_gid 4991 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 done 2024-09-11T02:51:29.359 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: watch_check 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: unwatch2 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: done 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3032 ms) 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: List watches 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: handle_notify cookie 94796004985216 notify_id 180388626443 notifier_gid 4991 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: notify2 done 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: watch_check 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: unwatch2 2024-09-11T02:51:29.360 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: flushing 2024-09-11T02:51:29.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: done 2024-09-11T02:51:29.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3023 ms) 2024-09-11T02:51:29.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16299 ms total) 2024-09-11T02:51:29.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: 2024-09-11T02:51:29.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-11T02:51:29.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28223 ms total) 2024-09-11T02:51:29.361 INFO:tasks.workunit.client.0.smithi190.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-11T02:51:31.718 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-11T02:51:31.718 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5075 ms) 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (56 ms) 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20058 ms) 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: attempt 0 of 20 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5324 ms) 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30513 ms total) 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: 2024-09-11T02:51:31.719 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-11T02:51:31.720 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30513 ms total) 2024-09-11T02:51:31.720 INFO:tasks.workunit.client.0.smithi190.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-11T02:51:33.336 INFO:tasks.workunit.client.0.smithi190.stdout: misc: Running main() from gmock_main.cc 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] Global test environment set-up. 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Version (1155 ms) 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2005 ms) 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongName (3067 ms) 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2990 ms) 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3062 ms) 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-11T02:51:33.337 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2717 ms) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Exec (3015 ms) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3043 ms) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2969 ms) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3035 ms) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1947 ms) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3005 ms) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] 12 tests from NeoRadosMisc (32010 ms total) 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: 2024-09-11T02:51:33.338 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [----------] Global test environment tear-down 2024-09-11T02:51:33.339 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [==========] 12 tests from 1 test suite ran. (32010 ms total) 2024-09-11T02:51:33.339 INFO:tasks.workunit.client.0.smithi190.stdout: misc: [ PASSED ] 12 tests. 2024-09-11T02:51:41.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: Running main() from gmock_main.cc 2024-09-11T02:51:41.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-11T02:51:41.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] Global test environment set-up. 2024-09-11T02:51:41.350 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2380 ms) 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3070 ms) 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3038 ms) 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (5743 ms) 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3016 ms) 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5069 ms) 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-11T02:51:41.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4982 ms) 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4965 ms) 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8003 ms) 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] 9 tests from LibRadosPools (40267 ms total) 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [----------] Global test environment tear-down 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (40268 ms total) 2024-09-11T02:51:41.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-11T02:51:42.358 INFO:tasks.workunit.client.0.smithi190.stdout: io: Running main() from gmock_main.cc 2024-09-11T02:51:42.358 INFO:tasks.workunit.client.0.smithi190.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-11T02:51:42.358 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] Global test environment set-up. 2024-09-11T02:51:42.358 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.Limits (2202 ms) 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3023 ms) 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.ReadOp (3052 ms) 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.SparseRead (2999 ms) 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2729 ms) 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-11T02:51:42.359 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3016 ms) 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3076 ms) 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2898 ms) 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3069 ms) 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.Trunc (2987 ms) 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.Remove (2996 ms) 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-11T02:51:42.360 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3014 ms) 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.RmXattr (2968 ms) 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3026 ms) 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] 14 tests from NeoRadosIo (41056 ms total) 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [----------] Global test environment tear-down 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [==========] 14 tests from 1 test suite ran. (41056 ms total) 2024-09-11T02:51:42.361 INFO:tasks.workunit.client.0.smithi190.stdout: io: [ PASSED ] 14 tests. 2024-09-11T02:51:43.385 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: Running main() from gmock_main.cc 2024-09-11T02:51:43.385 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] Global test environment set-up. 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (3129 ms) 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3063 ms) 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3006 ms) 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3035 ms) 2024-09-11T02:51:43.386 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2723 ms) 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3025 ms) 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3044 ms) 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2934 ms) 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3052 ms) 2024-09-11T02:51:43.387 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2954 ms) 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3010 ms) 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3027 ms) 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2984 ms) 2024-09-11T02:51:43.388 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-11T02:51:43.389 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3022 ms) 2024-09-11T02:51:43.389 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (42008 ms total) 2024-09-11T02:51:43.389 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: 2024-09-11T02:51:43.389 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [----------] Global test environment tear-down 2024-09-11T02:51:43.389 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (42008 ms total) 2024-09-11T02:51:43.389 INFO:tasks.workunit.client.0.smithi190.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] Global test environment set-up. 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-11T02:51:45.368 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T02:51:01.117+0000 7faa3e410880 -1 failed for service _ceph-mon._tcp 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T02:51:01.117+0000 7faa3e410880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (63 ms) 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5028 ms) 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5091 ms total) 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: started 0x5626666f4eb0 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: started 0x5626670f60e0 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: started 2 aios 2024-09-11T02:51:45.369 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: waiting 0x5626666f4eb0 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: waiting 0x5626670f60e0 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: done. 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5422 ms) 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5422 ms total) 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.Exec (100 ms) 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (9 ms) 2024-09-11T02:51:45.370 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (8 ms) 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4603 ms) 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (26655 ms) 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] 8 tests from LibRadosMisc (31375 ms total) 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [----------] Global test environment tear-down 2024-09-11T02:51:45.371 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (44260 ms total) 2024-09-11T02:51:45.372 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-11T02:51:56.351 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2015 ms) 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2030 ms) 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2031 ms) 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1995 ms) 2024-09-11T02:51:56.352 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8071 ms total) 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (3790 ms) 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4103 ms) 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5064 ms) 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (12958 ms total) 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-11T02:51:56.353 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3014 ms) 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2008 ms) 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2016 ms) 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1962 ms) 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9000 ms total) 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-11T02:51:56.354 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4056 ms) 2024-09-11T02:51:56.355 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-11T02:51:56.355 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4071 ms) 2024-09-11T02:51:56.355 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8127 ms total) 2024-09-11T02:51:56.355 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: 2024-09-11T02:51:56.355 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-11T02:51:56.355 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (55262 ms total) 2024-09-11T02:51:56.355 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-11T02:51:59.860 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T02:51:59.861 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T02:51:59.861 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T02:51:59.864 INFO:tasks.ceph.mon.a:Started 2024-09-11T02:51:59.864 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-09-11T02:51:59.864 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-11T02:51:59.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-11T02:51:59.867 INFO:tasks.ceph.mon.d:Started 2024-09-11T02:51:59.868 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:51:59.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:51:59.952 INFO:tasks.ceph.mon.d.smithi098.stderr:2024-09-11T02:51:59.951+0000 7f8c9fc49e00 -1 mon.d@-1(???).log v0 log_external_backlog rewinding external_log_to from 28 -> 0 (sync_force? mon rebuild?) 2024-09-11T02:51:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient: get_monmap_and_config 2024-09-11T02:51:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient: build_initial_monmap 2024-09-11T02:51:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient: monmap: 2024-09-11T02:51:59.988 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:51:59.990381+0000 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:51:59.990381+0000 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:51:59.989 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:51:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:51:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:51:59.990 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:51:59.990 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:51:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 Processor -- start 2024-09-11T02:51:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 -- start start 2024-09-11T02:51:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient: init 2024-09-11T02:51:59.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient: _add_conns ranks=[6,2,3,9,8,4,10,1,7,0,5] 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 0x7f3f1c0a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): picked mon.noname-d con 0x7f3f1c0a4820 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f1c077720 0x7f3f1c0a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): picked mon.noname-b con 0x7f3f1c077720 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c1399d0 0x7f3f1c139e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): picked mon.noname-g con 0x7f3f1c1399d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1c0a5680 con 0x7f3f1c077720 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:00.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1c0a0cd0 con 0x7f3f1c1399d0 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3f1c0a0fb0 con 0x7f3f1c0a4820 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f247de640 10 monclient(hunting): authenticate will time out at 1073.870356s 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 0x7f3f1c0a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f1c077720 0x7f3f1c0a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 0x7f3f1c0a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:40330/0 (socket says 172.21.15.98:40330) 2024-09-11T02:52:00.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 1 -- 172.21.15.98:0/1367963677 learned_addr learned my addr 172.21.15.98:0/1367963677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 10 monclient(hunting): get_auth_request con 0x7f3f1c0a4820 auth_method 0 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): get_auth_request con 0x7f3f1c077720 auth_method 0 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:00.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 1 -- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 msgr2=0x7f3f1c0a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 1 -- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 msgr2=0x7f3f1c0a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 0x7f3f1c0a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f2257a640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 0x7f3f1c0a4c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f22d7b640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c1399d0 0x7f3f1c139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f22d7b640 10 monclient(hunting): get_auth_request con 0x7f3f1c1399d0 auth_method 0 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f22d7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f22d7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f22d7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:00.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient(hunting): handle_auth_done global_id 22907 payload 274 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 1 -- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 msgr2=0x7f3f1c0a4c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 0x7f3f1c0a4c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 1 -- 172.21.15.98:0/1367963677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c1399d0 msgr2=0x7f3f1c139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c1399d0 0x7f3f1c139e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient: _finish_hunting 0 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 1 monclient: found mon.noname-b 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.987+0000 7f3f21d79640 1 -- 172.21.15.98:0/1367963677 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1c0a1290 con 0x7f3f1c077720 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f1c077720 0x7f3f1c0a5140 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f3f180097f0 tx=0x7f3f1801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7dede1c9217831f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 1 -- 172.21.15.98:0/1367963677 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f180b0070 con 0x7f3f1c077720 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:00.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: dump: 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:00.005 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: _finish_auth 0 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: _check_auth_tickets 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 1 -- 172.21.15.98:0/1367963677 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f180aa810 con 0x7f3f1c077720 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 5 monclient: authenticate success, global_id 22907 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:00.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 1 -- 172.21.15.98:0/1367963677 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f180a9300 con 0x7f3f1c077720 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 4 set_mon_vals no callback set 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21578640 10 monclient: dump: 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:00.007 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient: shutdown 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f1c077720 msgr2=0x7f3f1c0a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f1c077720 0x7f3f1c0a5140 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f3f180097f0 tx=0x7f3f1801c360 comp rx=0 tx=0).stop 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- 172.21.15.98:0/1367963677 shutdown_connections 2024-09-11T02:52:00.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3f1c077720 0x7f3f1c0a5140 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c1399d0 0x7f3f1c139e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 --2- 172.21.15.98:0/1367963677 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3f1c0a4820 0x7f3f1c0a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- 172.21.15.98:0/1367963677 >> 172.21.15.98:0/1367963677 conn(0x7f3f1c06ed00 msgr2=0x7f3f1c06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- 172.21.15.98:0/1367963677 shutdown_connections 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- 172.21.15.98:0/1367963677 wait complete. 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient: build_initial_monmap 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient: monmap: 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:00.009 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 librados: starting msgr at 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 librados: starting objecter 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 Processor -- start 2024-09-11T02:52:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- start start 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 librados: setting wanted keys 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 librados: calling monclient init 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient: init 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient: _add_conns ranks=[3,0,5,1,4,2,6,8,7] 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c077720 0x7f3f1c1d1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): picked mon.noname-g con 0x7f3f1c077720 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 0x7f3f1c1389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): picked mon.noname-a con 0x7f3f1c0a4820 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f1c1399d0 0x7f3f1c13cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): picked mon.noname-h con 0x7f3f1c1399d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1c0a2040 con 0x7f3f1c0a4820 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1c0a1ec0 con 0x7f3f1c077720 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3f1c143440 con 0x7f3f1c1399d0 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f247de640 10 monclient(hunting): authenticate will time out at 1073.875318s 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 0x7f3f1c1389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 0x7f3f1c1389a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44772/0 (socket says 172.21.15.98:44772) 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 -- 172.21.15.98:0/3471926254 learned_addr learned my addr 172.21.15.98:0/3471926254 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 10 monclient(hunting): get_auth_request con 0x7f3f1c0a4820 auth_method 0 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 -- 172.21.15.98:0/3471926254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 msgr2=0x7f3f1c1389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 -- 172.21.15.98:0/3471926254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 msgr2=0x7f3f1c1389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 0x7f3f1c1389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f21d79640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 0x7f3f1c1389a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c077720 0x7f3f1c1d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f1c1399d0 0x7f3f1c13cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 10 monclient(hunting): get_auth_request con 0x7f3f1c077720 auth_method 0 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 10 monclient(hunting): get_auth_request con 0x7f3f1c1399d0 auth_method 0 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f2257a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.991+0000 7f3f22d7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 10 monclient(hunting): handle_auth_done global_id 19344 payload 824 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 1 -- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f1c1399d0 msgr2=0x7f3f1c13cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f1c1399d0 0x7f3f1c13cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 1 -- 172.21.15.98:0/3471926254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 msgr2=0x7f3f1c1389a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 0x7f3f1c1389a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 10 monclient: _finish_hunting 0 2024-09-11T02:52:00.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 1 monclient: found mon.noname-g 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 1 -- 172.21.15.98:0/3471926254 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f1c124080 con 0x7f3f1c077720 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f22d7b640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f1c1399d0 0x7f3f1c13cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f2257a640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c077720 0x7f3f1c1d1440 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00e810 tx=0x7f3f0c00ecd0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 client.?.objecter ms_handle_connect 0x7f3f1c077720 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 1 -- 172.21.15.98:0/3471926254 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f0c017070 con 0x7f3f1c077720 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: dump: 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:00.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:00.016 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: _finish_auth 0 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 1 -- 172.21.15.98:0/3471926254 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f0c00ee00 con 0x7f3f1c077720 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 5 monclient: authenticate success, global_id 19344 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 monclient: _renew_subs 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f1c124360 con 0x7f3f1c077720 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 1 -- 172.21.15.98:0/3471926254 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3f0c00f040 con 0x7f3f1c077720 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:00.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 monclient: dump: 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:00.018 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f20d77640 4 set_mon_vals no callback set 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 client.19344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 monclient: _renew_subs 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f1c13fcd0 con 0x7f3f1c077720 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 1 librados: init done 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f1c0a6c10 con 0x7f3f1c077720 2024-09-11T02:52:00.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 1 -- 172.21.15.98:0/3471926254 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3f0c012050 con 0x7f3f1c077720 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f0003df40 0x7f3f00040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 1 -- 172.21.15.98:0/3471926254 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(79..79 src has 1..79) ==== 8295+0+0 (secure 0 0 0) 0x7f3f0c04fc50 con 0x7f3f1c077720 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 client.19344.objecter ms_dispatch 0x7f3f1c000ba0 osd_map(79..79 src has 1..79) 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 3 client.19344.objecter handle_osd_map got epochs [79,79] > 0 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 3 client.19344.objecter handle_osd_map decoding full epoch 79 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 20 client.19344.objecter dump_active .. 0 homeless 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f21d79640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f0003df40 0x7f3f00040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f21d79640 10 monclient: get_auth_request con 0x7f3f0003df40 auth_method 0 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f12ffd640 10 client.19344.objecter ms_handle_connect 0x7f3f0003df40 2024-09-11T02:52:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:51:59.995+0000 7f3f21d79640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f0003df40 0x7f3f00040400 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f3f1801c190 tx=0x7f3f180a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:00.007+0000 7f3f12ffd640 1 -- 172.21.15.98:0/3471926254 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f3f0c020030 con 0x7f3f1c077720 2024-09-11T02:52:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:00.007+0000 7f3f12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:00.007+0000 7f3f12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:00.279+0000 7f3f247de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:52:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:00.279+0000 7f3f247de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:00.279+0000 7f3f247de640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:00.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:00.279+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3f1c139e30 con 0x7f3f1c077720 2024-09-11T02:52:02.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:02.991+0000 7f3f11ffb640 10 monclient: tick 2024-09-11T02:52:02.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:02.991+0000 7f3f11ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:02.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:02.991+0000 7f3f11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:02.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:02.991+0000 7f3f11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:03.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:03.991+0000 7f3f11ffb640 10 monclient: tick 2024-09-11T02:52:03.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:03.991+0000 7f3f11ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:03.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:03.991+0000 7f3f11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:04.991+0000 7f3f11ffb640 10 monclient: tick 2024-09-11T02:52:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:04.991+0000 7f3f11ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:04.991+0000 7f3f11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:04.999+0000 7f3f137fe640 10 client.19344.objecter tick 2024-09-11T02:52:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.287+0000 7f3f12ffd640 1 -- 172.21.15.98:0/3471926254 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7f3f0c01b5c0 con 0x7f3f1c077720 2024-09-11T02:52:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.287+0000 7f3f12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:05.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.287+0000 7f3f12ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:05.290 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":24,"quorum":[1,2,3,4,5,6,7,8],"quorum_names":["f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:52:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 10 librados: watch_flush enter 2024-09-11T02:52:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 10 librados: watch_flush exit 2024-09-11T02:52:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 20 client.19344.objecter shutdown clearing up homeless session... 2024-09-11T02:52:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 10 client.19344.objecter successfully canceled tick 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f0003df40 msgr2=0x7f3f00040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f0003df40 0x7f3f00040400 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f3f1801c190 tx=0x7f3f180a6000 comp rx=0 tx=0).stop 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 10 monclient: shutdown 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c077720 msgr2=0x7f3f1c1d1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c077720 0x7f3f1c1d1440 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00e810 tx=0x7f3f0c00ecd0 comp rx=0 tx=0).stop 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 shutdown_connections 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3f0003df40 0x7f3f00040400 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3f1c1399d0 0x7f3f1c13cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3f1c0a4820 0x7f3f1c1389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 --2- 172.21.15.98:0/3471926254 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3f1c077720 0x7f3f1c1d1440 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 >> 172.21.15.98:0/3471926254 conn(0x7f3f1c06ed00 msgr2=0x7f3f1c13fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 shutdown_connections 2024-09-11T02:52:05.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 -- 172.21.15.98:0/3471926254 wait complete. 2024-09-11T02:52:05.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:05.291+0000 7f3f247de640 1 librados: shutdown 2024-09-11T02:52:08.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient: get_monmap_and_config 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient: build_initial_monmap 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient: monmap: 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:08.446566+0000 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:08.446566+0000 2024-09-11T02:52:08.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:08.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 1 Processor -- start 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 1 -- start start 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient: init 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient: _add_conns ranks=[1,8,3,10,0,2,7,6,9,5,4] 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6dc406f650 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc40a8380 0x7f6dc4072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient(hunting): picked mon.noname-e con 0x7f6dc40a8380 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc4070140 0x7f6dc4140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient(hunting): picked mon.noname-g con 0x7f6dc4070140 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dc4071a80 con 0x7f6dc406f650 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc40a8800 con 0x7f6dc4070140 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6dc4071d60 con 0x7f6dc40a8380 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dcbf76640 10 monclient(hunting): authenticate will time out at 1082.326857s 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc40a8380 0x7f6dc4072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc40a8380 0x7f6dc4072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56786/0 (socket says 172.21.15.98:56786) 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 -- 172.21.15.98:0/2153278244 learned_addr learned my addr 172.21.15.98:0/2153278244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): get_auth_request con 0x7f6dc40a8380 auth_method 0 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dca513640 1 --2- 172.21.15.98:0/2153278244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc4070140 0x7f6dc4140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dca513640 10 monclient(hunting): get_auth_request con 0x7f6dc4070140 auth_method 0 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dca513640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dca513640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dca513640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient(hunting): handle_auth_done global_id 23345 payload 274 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 -- 172.21.15.98:0/2153278244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc4070140 msgr2=0x7f6dc4140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 --2- 172.21.15.98:0/2153278244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc4070140 0x7f6dc4140c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 -- 172.21.15.98:0/2153278244 >> v1:172.21.15.190:6789/0 conn(0x7f6dc406f650 legacy=0x7f6dc406fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient: _finish_hunting 0 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 monclient: found mon.noname-e 2024-09-11T02:52:08.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.443+0000 7f6dc9511640 1 -- 172.21.15.98:0/2153278244 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc40714c0 con 0x7f6dc40a8380 2024-09-11T02:52:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc9511640 1 --2- 172.21.15.98:0/2153278244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc40a8380 0x7f6dc4072640 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6db40098a0 tx=0x7f6db401c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=16aa4d4932c74ca6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 1 -- 172.21.15.98:0/2153278244 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6db40ab890 con 0x7f6dc40a8380 2024-09-11T02:52:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: dump: 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: _finish_auth 0 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: _check_auth_tickets 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 1 -- 172.21.15.98:0/2153278244 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db40b0070 con 0x7f6dc40a8380 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 5 monclient: authenticate success, global_id 23345 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 1 -- 172.21.15.98:0/2153278244 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6db40aa380 con 0x7f6dc40a8380 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 4 set_mon_vals no callback set 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dc8d10640 10 monclient: dump: 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient: shutdown 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- 172.21.15.98:0/2153278244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc40a8380 msgr2=0x7f6dc4072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/2153278244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc40a8380 0x7f6dc4072640 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6db40098a0 tx=0x7f6db401c440 comp rx=0 tx=0).stop 2024-09-11T02:52:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- 172.21.15.98:0/2153278244 shutdown_connections 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/2153278244 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc4070140 0x7f6dc4140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/2153278244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc40a8380 0x7f6dc4072640 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- 172.21.15.98:0/2153278244 >> 172.21.15.98:0/2153278244 conn(0x7f6dc40ac8e0 msgr2=0x7f6dc40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- 172.21.15.98:0/2153278244 shutdown_connections 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- 172.21.15.98:0/2153278244 wait complete. 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient: build_initial_monmap 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient: monmap: 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 librados: starting msgr at 2024-09-11T02:52:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 librados: starting objecter 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 Processor -- start 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- start start 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 librados: setting wanted keys 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 librados: calling monclient init 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient: init 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient: _add_conns ranks=[3,8,2,1,7,5,6,4,0] 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc406f650 0x7f6dc41d1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): picked mon.noname-g con 0x7f6dc406f650 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc4070140 0x7f6dc41d1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): picked mon.noname-e con 0x7f6dc4070140 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6dc40a8380 0x7f6dc41d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): picked mon.noname-b con 0x7f6dc40a8380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc4143d10 con 0x7f6dc40a8380 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc4143b90 con 0x7f6dc406f650 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6dc4143e90 con 0x7f6dc4070140 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.447+0000 7f6dcbf76640 10 monclient(hunting): authenticate will time out at 1082.331969s 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc4070140 0x7f6dc41d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6dc40a8380 0x7f6dc41d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc4070140 0x7f6dc41d1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56790/0 (socket says 172.21.15.98:56790) 2024-09-11T02:52:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 -- 172.21.15.98:0/953693812 learned_addr learned my addr 172.21.15.98:0/953693812 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): get_auth_request con 0x7f6dc4070140 auth_method 0 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 10 monclient(hunting): get_auth_request con 0x7f6dc40a8380 auth_method 0 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dca513640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc406f650 0x7f6dc41d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 10 monclient(hunting): get_auth_request con 0x7f6dc406f650 auth_method 0 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient(hunting): handle_auth_done global_id 23354 payload 824 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 -- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc406f650 msgr2=0x7f6dc41d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc406f650 0x7f6dc41d1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 -- 172.21.15.98:0/953693812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6dc40a8380 msgr2=0x7f6dc41d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6dc40a8380 0x7f6dc41d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient: _finish_hunting 0 2024-09-11T02:52:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 monclient: found mon.noname-e 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 -- 172.21.15.98:0/953693812 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc413fb10 con 0x7f6dc4070140 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9511640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc4070140 0x7f6dc41d1560 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6db401aaf0 tx=0x7f6dc40a7f40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 client.?.objecter ms_handle_connect 0x7f6dc4070140 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 1 -- 172.21.15.98:0/953693812 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6db4003e20 con 0x7f6dc4070140 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: dump: 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.470 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 1 -- 172.21.15.98:0/953693812 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6db40ab420 con 0x7f6dc4070140 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 1 -- 172.21.15.98:0/953693812 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6db40aba40 con 0x7f6dc4070140 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 monclient: dump: 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dbbfff640 4 set_mon_vals no callback set 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 5 monclient: authenticate success, global_id 23354 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 monclient: _renew_subs 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc40a5170 con 0x7f6dc4070140 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 client.23354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 monclient: _renew_subs 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc413e310 con 0x7f6dc4070140 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 1 librados: init done 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc4072640 con 0x7f6dc4070140 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 1 -- 172.21.15.98:0/953693812 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6db40abbe0 con 0x7f6dc4070140 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6da803df40 0x7f6da8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 1 -- 172.21.15.98:0/953693812 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(83..83 src has 1..83) ==== 7206+0+0 (secure 0 0 0) 0x7f6db40ea380 con 0x7f6dc4070140 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 client.23354.objecter ms_dispatch 0x7f6dc4000ba0 osd_map(83..83 src has 1..83) 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 3 client.23354.objecter handle_osd_map got epochs [83,83] > 0 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 3 client.23354.objecter handle_osd_map decoding full epoch 83 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 20 client.23354.objecter dump_active .. 0 homeless 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6da803df40 0x7f6da8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 10 monclient: get_auth_request con 0x7f6da803df40 auth_method 0 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dc9d12640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6da803df40 0x7f6da8040400 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00016f0 tx=0x7f6dc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.451+0000 7f6dba7fc640 10 client.23354.objecter ms_handle_connect 0x7f6da803df40 2024-09-11T02:52:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.459+0000 7f6dba7fc640 1 -- 172.21.15.98:0/953693812 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6db4016150 con 0x7f6dc4070140 2024-09-11T02:52:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.459+0000 7f6dba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.459+0000 7f6dba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dcbf76640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:52:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dcbf76640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dcbf76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:08.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6dc40a7580 con 0x7f6dc4070140 2024-09-11T02:52:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dba7fc640 1 -- 172.21.15.98:0/953693812 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f6db40039b0 con 0x7f6dc4070140 2024-09-11T02:52:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:08.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:08.726 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":26,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.723+0000 7f6dcbf76640 10 librados: watch_flush enter 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 10 librados: watch_flush exit 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 20 client.23354.objecter shutdown clearing up homeless session... 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 10 client.23354.objecter successfully canceled tick 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6da803df40 msgr2=0x7f6da8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6da803df40 0x7f6da8040400 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7f6dc00016f0 tx=0x7f6dc0009040 comp rx=0 tx=0).stop 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 10 monclient: shutdown 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc4070140 msgr2=0x7f6dc41d1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc4070140 0x7f6dc41d1560 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6db401aaf0 tx=0x7f6dc40a7f40 comp rx=0 tx=0).stop 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 shutdown_connections 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6da803df40 0x7f6da8040400 unknown :-1 s=CLOSED pgs=3353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6dc40a8380 0x7f6dc41d1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6dc4070140 0x7f6dc41d1560 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 --2- 172.21.15.98:0/953693812 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6dc406f650 0x7f6dc41d1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 >> 172.21.15.98:0/953693812 conn(0x7f6dc40ac8e0 msgr2=0x7f6dc40a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 shutdown_connections 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 -- 172.21.15.98:0/953693812 wait complete. 2024-09-11T02:52:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.727+0000 7f6dcbf76640 1 librados: shutdown 2024-09-11T02:52:08.743 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:52:08.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:52:08.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient: get_monmap_and_config 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient: build_initial_monmap 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient: monmap: 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:08.865538+0000 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:08.865538+0000 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:08.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 Processor -- start 2024-09-11T02:52:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 -- start start 2024-09-11T02:52:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient: init 2024-09-11T02:52:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient: _add_conns ranks=[10,2,7,9,4,0,5,8,1,6,3] 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f4880139bd0 0x7f48800a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): picked mon.noname-f con 0x7f4880139bd0 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4880139fb0 0x7f48800a4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): picked mon.noname-b con 0x7f4880139fb0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 0x7f48800a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): picked mon.noname-i con 0x7f488013a3b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f48800777d0 con 0x7f4880139bd0 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f48800a0f60 con 0x7f4880139fb0 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4880070050 con 0x7f488013a3b0 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4887099640 10 monclient(hunting): authenticate will time out at 1082.745990s 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4880139fb0 0x7f48800a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4880139fb0 0x7f48800a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56506/0 (socket says 172.21.15.98:56506) 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 -- 172.21.15.98:0/3039620744 learned_addr learned my addr 172.21.15.98:0/3039620744 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): get_auth_request con 0x7f4880139fb0 auth_method 0 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4885636640 1 --2- 172.21.15.98:0/3039620744 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 0x7f48800a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4884e35640 1 --2- 172.21.15.98:0/3039620744 >> v2:172.21.15.190:3300/0 conn(0x7f4880139bd0 0x7f48800a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4885636640 10 monclient(hunting): get_auth_request con 0x7f488013a3b0 auth_method 0 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4885636640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4885636640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4885636640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient(hunting): handle_auth_done global_id 22961 payload 274 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 -- 172.21.15.98:0/3039620744 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 msgr2=0x7f48800a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 --2- 172.21.15.98:0/3039620744 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 0x7f48800a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 -- 172.21.15.98:0/3039620744 >> v2:172.21.15.190:3300/0 conn(0x7f4880139bd0 msgr2=0x7f48800a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 --2- 172.21.15.98:0/3039620744 >> v2:172.21.15.190:3300/0 conn(0x7f4880139bd0 0x7f48800a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 monclient: found mon.noname-b 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f487ffff640 1 -- 172.21.15.98:0/3039620744 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48800a1ac0 con 0x7f4880139fb0 2024-09-11T02:52:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4884e35640 10 monclient: get_auth_request con 0x7f4880139bd0 auth_method 0 2024-09-11T02:52:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.863+0000 7f4884e35640 1 --2- 172.21.15.98:0/3039620744 >> v2:172.21.15.190:3300/0 conn(0x7f4880139bd0 0x7f48800a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 1 --2- 172.21.15.98:0/3039620744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4880139fb0 0x7f48800a4610 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f48680098a0 tx=0x7f486801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3726c26846b06794 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 1 -- 172.21.15.98:0/3039620744 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48680b0070 con 0x7f4880139fb0 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: dump: 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 1 -- 172.21.15.98:0/3039620744 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48680aa810 con 0x7f4880139fb0 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 5 monclient: authenticate success, global_id 22961 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 4 set_mon_vals no callback set 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 1 -- 172.21.15.98:0/3039620744 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48680a9300 con 0x7f4880139fb0 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487f7fe640 10 monclient: dump: 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.881 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient: shutdown 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- 172.21.15.98:0/3039620744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4880139fb0 msgr2=0x7f48800a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 --2- 172.21.15.98:0/3039620744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4880139fb0 0x7f48800a4610 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f48680098a0 tx=0x7f486801c400 comp rx=0 tx=0).stop 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- 172.21.15.98:0/3039620744 shutdown_connections 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 --2- 172.21.15.98:0/3039620744 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 0x7f48800a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 --2- 172.21.15.98:0/3039620744 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4880139fb0 0x7f48800a4610 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 --2- 172.21.15.98:0/3039620744 >> v2:172.21.15.190:3300/0 conn(0x7f4880139bd0 0x7f48800a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- 172.21.15.98:0/3039620744 >> 172.21.15.98:0/3039620744 conn(0x7f488006ed00 msgr2=0x7f488006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- 172.21.15.98:0/3039620744 shutdown_connections 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- 172.21.15.98:0/3039620744 wait complete. 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient: build_initial_monmap 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient: monmap: 2024-09-11T02:52:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 librados: starting msgr at 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 librados: starting objecter 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 Processor -- start 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- start start 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 librados: setting wanted keys 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 librados: calling monclient init 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient: init 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient: _add_conns ranks=[3,5,7,4,8,0,1,2,6] 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4880139bd0 0x7f4880132380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): picked mon.noname-g con 0x7f4880139bd0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4880139fb0 0x7f48801328c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): picked mon.noname-h con 0x7f4880139fb0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 0x7f4880132e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): picked mon.noname-i con 0x7f488013a3b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4880143440 con 0x7f4880139bd0 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f48800a54a0 con 0x7f4880139fb0 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f48801435c0 con 0x7f488013a3b0 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4887099640 10 monclient(hunting): authenticate will time out at 1082.751128s 2024-09-11T02:52:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 0x7f4880132e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4880139fb0 0x7f48801328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4880139bd0 0x7f4880132380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4880139fb0 0x7f48801328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:44000/0 (socket says 172.21.15.98:44000) 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 10 monclient(hunting): get_auth_request con 0x7f4880139bd0 auth_method 0 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 1 -- 172.21.15.98:0/3934687791 learned_addr learned my addr 172.21.15.98:0/3934687791 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 10 monclient(hunting): get_auth_request con 0x7f4880139fb0 auth_method 0 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 10 monclient(hunting): get_auth_request con 0x7f488013a3b0 auth_method 0 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f487ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.867+0000 7f4885636640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 10 monclient(hunting): handle_auth_done global_id 19094 payload 824 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 msgr2=0x7f4880132e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f488013a3b0 0x7f4880132e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4880139bd0 msgr2=0x7f4880132380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4880139bd0 0x7f4880132380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 1 monclient: found mon.noname-h 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 1 -- 172.21.15.98:0/3934687791 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488013d130 con 0x7f4880139fb0 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487ffff640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4880139fb0 0x7f48801328c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f48680097c0 tx=0x7f48680191a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 client.?.objecter ms_handle_connect 0x7f4880139fb0 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48680b9070 con 0x7f4880139fb0 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: dump: 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: _finish_auth 0 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:52:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48680044b0 con 0x7f4880139fb0 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48680b0040 con 0x7f4880139fb0 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487effd640 4 set_mon_vals no callback set 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: dump: 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 5 monclient: authenticate success, global_id 19094 2024-09-11T02:52:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 10 monclient: _renew_subs 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4880138710 con 0x7f4880139fb0 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 10 client.19094.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 10 monclient: _renew_subs 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f488013eed0 con 0x7f4880139fb0 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 1 librados: init done 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488013dc90 0x7f48801356d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f488006ca10 con 0x7f488013dc90 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488013dc90 0x7f48801356d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient: get_auth_request con 0x7f488013dc90 auth_method 0 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 10 monclient(hunting): handle_auth_done global_id 14112 payload 824 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 1 -- 172.21.15.98:0/3934687791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4880069c80 con 0x7f488013dc90 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4884e35640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488013dc90 0x7f48801356d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4878007fb0 tx=0x7f487800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 client.19094.objecter ms_handle_connect 0x7f488013dc90 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 client.19094.objecter resend_mon_ops 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f487800bb50 con 0x7f488013dc90 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: dump: 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4868004040 con 0x7f4880139fb0 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f486003e1b0 0x7f4860040670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(83..83 src has 1..83) ==== 7206+0+0 (secure 0 0 0) 0x7f48680e8e30 con 0x7f4880139fb0 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 client.19094.objecter ms_dispatch 0x7f4880000ba0 osd_map(83..83 src has 1..83) 2024-09-11T02:52:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 3 client.19094.objecter handle_osd_map got epochs [83,83] > 0 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 3 client.19094.objecter handle_osd_map decoding full epoch 83 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 20 client.19094.objecter dump_active .. 0 homeless 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f487800bcf0 con 0x7f488013dc90 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488013dc90 msgr2=0x7f48801356d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488013dc90 0x7f48801356d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4878007fb0 tx=0x7f487800c8b0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488013dc90 msgr2=0x7f48801356d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f487d7fa640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f488013dc90 0x7f48801356d0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4878007fb0 tx=0x7f487800c8b0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4885636640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f486003e1b0 0x7f4860040670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.871+0000 7f4885636640 10 monclient: get_auth_request con 0x7f486003e1b0 auth_method 0 2024-09-11T02:52:08.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.875+0000 7f4885636640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f486003e1b0 0x7f4860040670 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f4870007950 tx=0x7f4870008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.875+0000 7f487d7fa640 10 client.19094.objecter ms_handle_connect 0x7f486003e1b0 2024-09-11T02:52:08.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:08.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4880135c10 0x7f4880136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:08.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:08.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f488006e870 con 0x7f4880135c10 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4880135c10 0x7f4880136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient: get_auth_request con 0x7f4880135c10 auth_method 0 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 10 monclient(hunting): handle_auth_done global_id 14121 payload 824 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 1 -- 172.21.15.98:0/3934687791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48800a2cd0 con 0x7f4880135c10 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 10 client.19094.objecter ms_handle_connect 0x7f4880135c10 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4884e35640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4880135c10 0x7f4880136020 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f487800ce50 tx=0x7f4878004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:08.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 10 client.19094.objecter resend_mon_ops 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f487801ddf0 con 0x7f4880135c10 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 10 monclient: dump: 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:08.901 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9265 (secure 0 0 0) 0x7f4878017420 con 0x7f4880135c10 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4880135c10 msgr2=0x7f4880136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4880135c10 0x7f4880136020 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f487800ce50 tx=0x7f4878004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4880135c10 msgr2=0x7f4880136020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:08.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f487d7fa640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4880135c10 0x7f4880136020 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f487800ce50 tx=0x7f4878004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 8931, 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:08.903 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:08.904 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:08.907 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:08.908 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:08.910 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:08.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:08.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.915 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.919 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:08.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.922 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:08.924 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.925 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:08.926 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 10 librados: watch_flush enter 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 10 librados: watch_flush exit 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 20 client.19094.objecter shutdown clearing up homeless session... 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 10 client.19094.objecter successfully canceled tick 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f486003e1b0 msgr2=0x7f4860040670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f486003e1b0 0x7f4860040670 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f4870007950 tx=0x7f4870008040 comp rx=0 tx=0).stop 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 10 monclient: shutdown 2024-09-11T02:52:08.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4880139fb0 msgr2=0x7f48801328c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 --2- 172.21.15.98:0/3934687791 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4880139fb0 0x7f48801328c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f48680097c0 tx=0x7f48680191a0 comp rx=0 tx=0).stop 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4885636640 1 -- 172.21.15.98:0/3934687791 reap_dead start 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4885636640 1 -- 172.21.15.98:0/3934687791 reap_dead start 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 shutdown_connections 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 >> 172.21.15.98:0/3934687791 conn(0x7f488006ed00 msgr2=0x7f4880070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 shutdown_connections 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 -- 172.21.15.98:0/3934687791 wait complete. 2024-09-11T02:52:08.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:08.879+0000 7f4887099640 1 librados: shutdown 2024-09-11T02:52:08.929 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:08.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:52:08.930 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:52:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient: get_monmap_and_config 2024-09-11T02:52:09.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient: monmap: 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:09.046136+0000 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:09.046136+0000 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.045 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:09.046 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:09.046 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 Processor -- start 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 -- start start 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient: init 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient: _add_conns ranks=[6,7,10,4,2,1,5,3,8,0,9] 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb1000a7aa0 0x7fb1000a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): picked mon.noname-d con 0x7fb1000a7aa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb1000a12a0 0x7fb1000a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): picked mon.noname-i con 0x7fb1000a12a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb100139850 0x7fb100139c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): picked mon.noname-f con 0x7fb100139850 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb1000a2b30 con 0x7fb100139850 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb1000a8900 con 0x7fb1000a7aa0 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb1000a1720 con 0x7fb1000a12a0 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb1058de640 10 monclient(hunting): authenticate will time out at 1082.926385s 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb1000a7aa0 0x7fb1000a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb1000a7aa0 0x7fb1000a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58316/0 (socket says 172.21.15.98:58316) 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 -- 172.21.15.98:0/2421694448 learned_addr learned my addr 172.21.15.98:0/2421694448 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): get_auth_request con 0x7fb1000a7aa0 auth_method 0 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0fe7fc640 1 --2- 172.21.15.98:0/2421694448 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb1000a12a0 0x7fb1000a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0fe7fc640 10 monclient(hunting): get_auth_request con 0x7fb1000a12a0 auth_method 0 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0ff7fe640 1 --2- 172.21.15.98:0/2421694448 >> v2:172.21.15.190:3300/0 conn(0x7fb100139850 0x7fb100139c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient(hunting): handle_auth_done global_id 14145 payload 274 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 -- 172.21.15.98:0/2421694448 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb1000a12a0 msgr2=0x7fb1000a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 --2- 172.21.15.98:0/2421694448 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb1000a12a0 0x7fb1000a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 -- 172.21.15.98:0/2421694448 >> v2:172.21.15.190:3300/0 conn(0x7fb100139850 msgr2=0x7fb100139c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 --2- 172.21.15.98:0/2421694448 >> v2:172.21.15.190:3300/0 conn(0x7fb100139850 0x7fb100139c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 monclient: found mon.noname-d 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0feffd640 1 -- 172.21.15.98:0/2421694448 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1000a27c0 con 0x7fb1000a7aa0 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0ff7fe640 10 monclient: get_auth_request con 0x7fb100139850 auth_method 0 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.043+0000 7fb0ff7fe640 1 --2- 172.21.15.98:0/2421694448 >> v2:172.21.15.190:3300/0 conn(0x7fb100139850 0x7fb100139c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0feffd640 1 --2- 172.21.15.98:0/2421694448 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb1000a7aa0 0x7fb1000a7e80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009870 tx=0x7fb0f001c420 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1534a6f792a94172 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 1 -- 172.21.15.98:0/2421694448 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0f00b0070 con 0x7fb1000a7aa0 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:09.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: dump: 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.053 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 1 -- 172.21.15.98:0/2421694448 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f00aa810 con 0x7fb1000a7aa0 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 5 monclient: authenticate success, global_id 14145 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 1 -- 172.21.15.98:0/2421694448 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0f00a9300 con 0x7fb1000a7aa0 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 4 set_mon_vals no callback set 2024-09-11T02:52:09.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fdffb640 10 monclient: dump: 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.055 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.056 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.056 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.056 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.056 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient: shutdown 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- 172.21.15.98:0/2421694448 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb1000a7aa0 msgr2=0x7fb1000a7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 --2- 172.21.15.98:0/2421694448 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb1000a7aa0 0x7fb1000a7e80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0009870 tx=0x7fb0f001c420 comp rx=0 tx=0).stop 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- 172.21.15.98:0/2421694448 shutdown_connections 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 --2- 172.21.15.98:0/2421694448 >> v2:172.21.15.190:3300/0 conn(0x7fb100139850 0x7fb100139c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 --2- 172.21.15.98:0/2421694448 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb1000a12a0 0x7fb1000a83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 --2- 172.21.15.98:0/2421694448 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb1000a7aa0 0x7fb1000a7e80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- 172.21.15.98:0/2421694448 >> 172.21.15.98:0/2421694448 conn(0x7fb1000ac690 msgr2=0x7fb1000a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- 172.21.15.98:0/2421694448 shutdown_connections 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- 172.21.15.98:0/2421694448 wait complete. 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient: monmap: 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.065 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 librados: starting msgr at 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 librados: starting objecter 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 Processor -- start 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- start start 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 librados: setting wanted keys 2024-09-11T02:52:09.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 librados: calling monclient init 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient: init 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient: _add_conns ranks=[1,2,6,0,5,3,8,7,4] 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1000a12a0 0x7fb1001d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): picked mon.noname-f con 0x7fb1000a12a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1000a7aa0 0x7fb100138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): picked mon.noname-b con 0x7fb1000a7aa0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb100139850 0x7fb100139080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): picked mon.noname-d con 0x7fb100139850 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb1000a65d0 con 0x7fb1000a12a0 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb1000a6450 con 0x7fb1000a7aa0 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb1001432c0 con 0x7fb100139850 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb1058de640 10 monclient(hunting): authenticate will time out at 1082.931289s 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1000a7aa0 0x7fb100138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb100139850 0x7fb100139080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1000a7aa0 0x7fb100138b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56518/0 (socket says 172.21.15.98:56518) 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 1 -- 172.21.15.98:0/1275864193 learned_addr learned my addr 172.21.15.98:0/1275864193 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 10 monclient(hunting): get_auth_request con 0x7fb1000a7aa0 auth_method 0 2024-09-11T02:52:09.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): get_auth_request con 0x7fb100139850 auth_method 0 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0feffd640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1000a12a0 0x7fb1001d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0feffd640 10 monclient(hunting): get_auth_request con 0x7fb1000a12a0 auth_method 0 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient(hunting): handle_auth_done global_id 14154 payload 824 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1000a7aa0 msgr2=0x7fb100138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1000a7aa0 0x7fb100138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1000a12a0 msgr2=0x7fb1001d15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1000a12a0 0x7fb1001d15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 1 monclient: found mon.noname-d 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0ff7fe640 1 -- 172.21.15.98:0/1275864193 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10013fcc0 con 0x7fb100139850 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.047+0000 7fb0fe7fc640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb1000a7aa0 0x7fb100138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0ff7fe640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb100139850 0x7fb100139080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8007fd0 tx=0x7fb0f800a710 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 client.?.objecter ms_handle_connect 0x7fb100139850 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0f80158b0 con 0x7fb100139850 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: dump: 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.071 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f8015a90 con 0x7fb100139850 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 5 monclient: authenticate success, global_id 14154 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 10 monclient: _renew_subs 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1001245e0 con 0x7fb100139850 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0f8015c80 con 0x7fb100139850 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: dump: 2024-09-11T02:52:09.073 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.074 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0fd7fa640 4 set_mon_vals no callback set 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 10 client.14154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 10 monclient: _renew_subs 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10013f6a0 con 0x7fb100139850 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 1 librados: init done 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013dbb0 0x7fb10013df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb10006ca10 con 0x7fb10013dbb0 2024-09-11T02:52:09.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013dbb0 0x7fb10013df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient: get_auth_request con 0x7fb10013dbb0 auth_method 0 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb0f8015440 con 0x7fb100139850 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 10 monclient(hunting): handle_auth_done global_id 22979 payload 824 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 1 -- 172.21.15.98:0/1275864193 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1000ab520 con 0x7fb10013dbb0 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0ec03df40 0x7fb0ec040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(83..83 src has 1..83) ==== 7206+0+0 (secure 0 0 0) 0x7fb0f8050460 con 0x7fb100139850 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 client.14154.objecter ms_dispatch 0x7fb100000ba0 osd_map(83..83 src has 1..83) 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 3 client.14154.objecter handle_osd_map got epochs [83,83] > 0 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 3 client.14154.objecter handle_osd_map decoding full epoch 83 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0feffd640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013dbb0 0x7fb10013df90 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fb1000acf10 tx=0x7fb0f00191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 20 client.14154.objecter dump_active .. 0 homeless 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 client.14154.objecter ms_handle_connect 0x7fb10013dbb0 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 client.14154.objecter resend_mon_ops 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0f00b9030 con 0x7fb10013dbb0 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: dump: 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0fe7fc640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0ec03df40 0x7fb0ec040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0fe7fc640 10 monclient: get_auth_request con 0x7fb0ec03df40 auth_method 0 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0f00b1040 con 0x7fb10013dbb0 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013dbb0 msgr2=0x7fb10013df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013dbb0 0x7fb10013df90 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fb1000acf10 tx=0x7fb0f00191c0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013dbb0 msgr2=0x7fb10013df90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013dbb0 0x7fb10013df90 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0e7fff640 10 client.14154.objecter ms_handle_connect 0x7fb0ec03df40 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.051+0000 7fb0fe7fc640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0ec03df40 0x7fb0ec040400 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fb10013f310 tx=0x7fb0f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013e4d0 0x7fb100071710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb10006e870 con 0x7fb10013e4d0 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013e4d0 0x7fb100071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient: get_auth_request con 0x7fb10013e4d0 auth_method 0 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 10 monclient(hunting): handle_auth_done global_id 22988 payload 824 2024-09-11T02:52:09.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 1 -- 172.21.15.98:0/1275864193 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb100139c70 con 0x7fb10013e4d0 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 10 client.14154.objecter ms_handle_connect 0x7fb10013e4d0 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 10 client.14154.objecter resend_mon_ops 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0feffd640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013e4d0 0x7fb100071710 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001c250 tx=0x7fb0f00a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0f00b9040 con 0x7fb10013e4d0 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 10 monclient: dump: 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fb0f0003d90 con 0x7fb10013e4d0 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:09.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013e4d0 msgr2=0x7fb100071710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013e4d0 0x7fb100071710 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001c250 tx=0x7fb0f00a8000 comp rx=0 tx=0).stop 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013e4d0 msgr2=0x7fb100071710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0e7fff640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb10013e4d0 0x7fb100071710 secure :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb0f001c250 tx=0x7fb0f00a8000 comp rx=0 tx=0).stop 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:09.083 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 144843, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:52:09.084 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.085 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.086 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.087 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:09.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:09.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:09.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.099 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.101 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.102 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:09.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.106 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.107 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:09.107 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:09.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 10 librados: watch_flush enter 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 10 librados: watch_flush exit 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 20 client.14154.objecter shutdown clearing up homeless session... 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 10 client.14154.objecter successfully canceled tick 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0ec03df40 msgr2=0x7fb0ec040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0ec03df40 0x7fb0ec040400 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fb10013f310 tx=0x7fb0f4008040 comp rx=0 tx=0).stop 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 10 monclient: shutdown 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb0ff7fe640 1 -- 172.21.15.98:0/1275864193 reap_dead start 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb100139850 msgr2=0x7fb100139080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb100139850 0x7fb100139080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8007fd0 tx=0x7fb0f800a710 comp rx=0 tx=0).stop 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 shutdown_connections 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 --2- 172.21.15.98:0/1275864193 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb100139850 0x7fb100139080 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 >> 172.21.15.98:0/1275864193 conn(0x7fb1000ac690 msgr2=0x7fb1000ab770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 shutdown_connections 2024-09-11T02:52:09.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 -- 172.21.15.98:0/1275864193 wait complete. 2024-09-11T02:52:09.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.059+0000 7fb1058de640 1 librados: shutdown 2024-09-11T02:52:09.110 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:09.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:52:09.110 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient: get_monmap_and_config 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient: monmap: 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:09.233429+0000 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:09.233429+0000 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.232 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:09.233 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 Processor -- start 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 -- start start 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient: init 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient: _add_conns ranks=[8,9,2,10,3,1,7,6,5,0,4] 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f8139b10 0x7f72f8072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): picked mon.noname-e con 0x7f72f8139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72f8139ef0 0x7f72f80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): picked mon.noname-a con 0x7f72f8139ef0 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72f813a2d0 0x7f72f80a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): picked mon.noname-b con 0x7f72f813a2d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f72f80702b0 con 0x7f72f8139ef0 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f72f8071bf0 con 0x7f72f813a2d0 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f72f8071350 con 0x7f72f8139b10 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72fcd27640 10 monclient(hunting): authenticate will time out at 1083.113689s 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f8139b10 0x7f72f8072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72f813a2d0 0x7f72f80a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72f8139ef0 0x7f72f80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f8139b10 0x7f72f8072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56800/0 (socket says 172.21.15.98:56800) 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f72f8139ef0 0x7f72f80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59458/0 (socket says 172.21.15.98:59458) 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 1 -- 172.21.15.98:0/2019883701 learned_addr learned my addr 172.21.15.98:0/2019883701 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72f813a2d0 0x7f72f80a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56544/0 (socket says 172.21.15.98:56544) 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 10 monclient(hunting): get_auth_request con 0x7f72f8139b10 auth_method 0 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 10 monclient(hunting): get_auth_request con 0x7f72f813a2d0 auth_method 0 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): get_auth_request con 0x7f72f8139ef0 auth_method 0 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient(hunting): handle_auth_done global_id 14130 payload 274 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 -- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f8139b10 msgr2=0x7f72f8072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f8139b10 0x7f72f8072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 -- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72f813a2d0 msgr2=0x7f72f80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72f813a2d0 0x7f72f80a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 monclient: found mon.noname-a 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72edd9b640 1 -- 172.21.15.98:0/2019883701 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f80a9620 con 0x7f72f8139ef0 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f6d9d640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72f813a2d0 0x7f72f80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.231+0000 7f72f659c640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f8139b10 0x7f72f8072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f8139ef0 0x7f72f80a7a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f72e00097f0 tx=0x7f72e001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa2a09b953248edd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 1 -- 172.21.15.98:0/2019883701 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72e00b0070 con 0x7f72f8139ef0 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: dump: 2024-09-11T02:52:09.242 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.243 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 1 -- 172.21.15.98:0/2019883701 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72e00aa810 con 0x7f72f8139ef0 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 5 monclient: authenticate success, global_id 14130 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 1 -- 172.21.15.98:0/2019883701 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72e00a9300 con 0x7f72f8139ef0 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 4 set_mon_vals no callback set 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f5d9b640 10 monclient: dump: 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: shutdown 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f8139ef0 msgr2=0x7f72f80a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f8139ef0 0x7f72f80a7a10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f72e00097f0 tx=0x7f72e001c3f0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- 172.21.15.98:0/2019883701 shutdown_connections 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72f813a2d0 0x7f72f80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72f8139ef0 0x7f72f80a7a10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 --2- 172.21.15.98:0/2019883701 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f8139b10 0x7f72f8072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- 172.21.15.98:0/2019883701 >> 172.21.15.98:0/2019883701 conn(0x7f72f80ac790 msgr2=0x7f72f80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- 172.21.15.98:0/2019883701 shutdown_connections 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- 172.21.15.98:0/2019883701 wait complete. 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: monmap: 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 librados: starting msgr at 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 librados: starting objecter 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 Processor -- start 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- start start 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 librados: setting wanted keys 2024-09-11T02:52:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 librados: calling monclient init 2024-09-11T02:52:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: init 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: _add_conns ranks=[1,6,8,7,2,3,0,4,5] 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f8139b10 0x7f72f80a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): picked mon.noname-f con 0x7f72f8139b10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72f8139ef0 0x7f72f813f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): picked mon.noname-d con 0x7f72f8139ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f813a2d0 0x7f72f813fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): picked mon.noname-e con 0x7f72f813a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f72f8143460 con 0x7f72f8139b10 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f72f80a88a0 con 0x7f72f8139ef0 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f72f81435e0 con 0x7f72f813a2d0 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient(hunting): authenticate will time out at 1083.119090s 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f813a2d0 0x7f72f813fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72f8139ef0 0x7f72f813f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72f8139ef0 0x7f72f813f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58338/0 (socket says 172.21.15.98:58338) 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f813a2d0 0x7f72f813fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56802/0 (socket says 172.21.15.98:56802) 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 -- 172.21.15.98:0/813773082 learned_addr learned my addr 172.21.15.98:0/813773082 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): get_auth_request con 0x7f72f8139ef0 auth_method 0 2024-09-11T02:52:09.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 10 monclient(hunting): get_auth_request con 0x7f72f813a2d0 auth_method 0 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f659c640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f8139b10 0x7f72f80a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient(hunting): handle_auth_done global_id 14163 payload 824 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f813a2d0 msgr2=0x7f72f813fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f72f813a2d0 0x7f72f813fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f8139b10 msgr2=0x7f72f80a5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f8139b10 0x7f72f80a5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 monclient: found mon.noname-d 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 -- 172.21.15.98:0/813773082 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f80a9f70 con 0x7f72f8139ef0 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f659c640 10 monclient: get_auth_request con 0x7f72f8139b10 auth_method 0 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f659c640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72f8139b10 0x7f72f80a5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72edd9b640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72f8139ef0 0x7f72f813f930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f72e001cc90 tx=0x7f72e0019190 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 client.?.objecter ms_handle_connect 0x7f72f8139ef0 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72e00b0040 con 0x7f72f8139ef0 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 monclient: dump: 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 5 monclient: authenticate success, global_id 14163 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72e00b9070 con 0x7f72f8139ef0 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: _renew_subs 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72e00044b0 con 0x7f72f8139ef0 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f80aa4a0 con 0x7f72f8139ef0 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.235+0000 7f72f559a640 4 set_mon_vals no callback set 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 10 client.14163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 monclient: dump: 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 10 monclient: _renew_subs 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f813d410 con 0x7f72f8139ef0 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 1 librados: init done 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1570 0x7f72f81d1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f72f806ca10 con 0x7f72f81d1570 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f72e00b3070 con 0x7f72f8139ef0 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1570 0x7f72f81d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient: get_auth_request con 0x7f72f81d1570 auth_method 0 2024-09-11T02:52:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f72d803df40 0x7f72d8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 10 monclient(hunting): handle_auth_done global_id 22954 payload 824 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 1 -- 172.21.15.98:0/813773082 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72f81403b0 con 0x7f72f81d1570 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(83..83 src has 1..83) ==== 7206+0+0 (secure 0 0 0) 0x7f72e00a4e00 con 0x7f72f8139ef0 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 client.14163.objecter ms_dispatch 0x7f72f8000ba0 osd_map(83..83 src has 1..83) 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 3 client.14163.objecter handle_osd_map got epochs [83,83] > 0 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 3 client.14163.objecter handle_osd_map decoding full epoch 83 2024-09-11T02:52:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 20 client.14163.objecter dump_active .. 0 homeless 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 client.14163.objecter ms_handle_connect 0x7f72f81d1570 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 client.14163.objecter resend_mon_ops 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f659c640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1570 0x7f72f81d1950 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f72e8010790 tx=0x7f72e8010c50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f6d9d640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f72d803df40 0x7f72d8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72e800f040 con 0x7f72f81d1570 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 monclient: dump: 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f6d9d640 10 monclient: get_auth_request con 0x7f72d803df40 auth_method 0 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f72e80143f0 con 0x7f72f81d1570 2024-09-11T02:52:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1570 msgr2=0x7f72f81d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1570 0x7f72f81d1950 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f72e8010790 tx=0x7f72e8010c50 comp rx=0 tx=0).stop 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1570 msgr2=0x7f72f81d1950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1570 0x7f72f81d1950 secure :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f72e8010790 tx=0x7f72e8010c50 comp rx=0 tx=0).stop 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72ef7fe640 10 client.14163.objecter ms_handle_connect 0x7f72d803df40 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.239+0000 7f72f6d9d640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f72d803df40 0x7f72d8040400 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009c10 tx=0x7f72f000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1e90 0x7f72f8137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f72f806e870 con 0x7f72f81d1e90 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1e90 0x7f72f8137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient: get_auth_request con 0x7f72f81d1e90 auth_method 0 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 10 monclient(hunting): handle_auth_done global_id 22963 payload 824 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 1 -- 172.21.15.98:0/813773082 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f72f806e680 con 0x7f72f81d1e90 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 10 client.14163.objecter ms_handle_connect 0x7f72f81d1e90 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 10 client.14163.objecter resend_mon_ops 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f659c640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1e90 0x7f72f8137ba0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f72e801b000 tx=0x7f72e80046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72e80059a0 con 0x7f72f81d1e90 2024-09-11T02:52:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 10 monclient: dump: 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f72e8014af0 con 0x7f72f81d1e90 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1e90 msgr2=0x7f72f8137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1e90 0x7f72f8137ba0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f72e801b000 tx=0x7f72e80046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1e90 msgr2=0x7f72f8137ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72ef7fe640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72f81d1e90 0x7f72f8137ba0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145031, 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:09.271 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.272 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:09.273 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:09.274 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:09.275 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:09.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:09.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:09.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.283 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:09.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:09.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.290 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.292 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.294 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:09.294 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 10 librados: watch_flush enter 2024-09-11T02:52:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 10 librados: watch_flush exit 2024-09-11T02:52:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 20 client.14163.objecter shutdown clearing up homeless session... 2024-09-11T02:52:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 10 client.14163.objecter successfully canceled tick 2024-09-11T02:52:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f72d803df40 msgr2=0x7f72d8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f72d803df40 0x7f72d8040400 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009c10 tx=0x7f72f000b040 comp rx=0 tx=0).stop 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 10 monclient: shutdown 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72f8139ef0 msgr2=0x7f72f813f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 --2- 172.21.15.98:0/813773082 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f72f8139ef0 0x7f72f813f930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f72e001cc90 tx=0x7f72e0019190 comp rx=0 tx=0).stop 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f6d9d640 1 -- 172.21.15.98:0/813773082 reap_dead start 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72f6d9d640 1 -- 172.21.15.98:0/813773082 reap_dead start 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 shutdown_connections 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 >> 172.21.15.98:0/813773082 conn(0x7f72f80ac790 msgr2=0x7f72f80a7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 shutdown_connections 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 -- 172.21.15.98:0/813773082 wait complete. 2024-09-11T02:52:09.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.247+0000 7f72fcd27640 1 librados: shutdown 2024-09-11T02:52:09.297 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:09.297 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:52:09.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:52:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient: get_monmap_and_config 2024-09-11T02:52:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient: monmap: 2024-09-11T02:52:09.416 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:09.418026+0000 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:09.418026+0000 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.417 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.418 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.418 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.418 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:09.418 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:09.418 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 Processor -- start 2024-09-11T02:52:09.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 -- start start 2024-09-11T02:52:09.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient: init 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient: _add_conns ranks=[6,7,9,10,0,1,3,8,5,2,4] 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc0a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): picked mon.noname-d con 0x7fe6fc0a3690 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc09df80 0x7fe6fc0a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): picked mon.noname-i con 0x7fe6fc09df80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6fc136450 0x7fe6fc1368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): picked mon.noname-a con 0x7fe6fc136450 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe6fc073d70 con 0x7fe6fc136450 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe6fc0a44f0 con 0x7fe6fc0a3690 2024-09-11T02:52:09.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe6fc0a0870 con 0x7fe6fc09df80 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe700897640 10 monclient(hunting): authenticate will time out at 1083.298120s 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6fc136450 0x7fe6fc1368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc0a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe6fc136450 0x7fe6fc1368b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59472/0 (socket says 172.21.15.98:59472) 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 -- 172.21.15.98:0/2452952554 learned_addr learned my addr 172.21.15.98:0/2452952554 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): get_auth_request con 0x7fe6fc136450 auth_method 0 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 10 monclient(hunting): get_auth_request con 0x7fe6fc0a3690 auth_method 0 2024-09-11T02:52:09.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6f9d9b640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc09df80 0x7fe6fc0a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient(hunting): handle_auth_done global_id 14139 payload 274 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 -- 172.21.15.98:0/2452952554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc09df80 msgr2=0x7fe6fc0a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc09df80 0x7fe6fc0a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 -- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 msgr2=0x7fe6fc0a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc0a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 monclient: found mon.noname-a 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fad9d640 1 -- 172.21.15.98:0/2452952554 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6fc09fe90 con 0x7fe6fc136450 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6f9d9b640 10 monclient: get_auth_request con 0x7fe6fc09df80 auth_method 0 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6f9d9b640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc09df80 0x7fe6fc0a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.415+0000 7fe6fa59c640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc0a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fad9d640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc136450 0x7fe6fc1368b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d150 tx=0x7fe6ec01eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66dcc616b56c798 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 1 -- 172.21.15.98:0/2452952554 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6ec0a77c0 con 0x7fe6fc136450 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: dump: 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.425 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 1 -- 172.21.15.98:0/2452952554 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6ec0a7980 con 0x7fe6fc136450 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 5 monclient: authenticate success, global_id 14139 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 4 set_mon_vals no callback set 2024-09-11T02:52:09.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 1 -- 172.21.15.98:0/2452952554 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6ec009600 con 0x7fe6fc136450 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f959a640 10 monclient: dump: 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.427 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.428 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.428 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.428 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.428 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.428 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.428 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.428 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: shutdown 2024-09-11T02:52:09.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc136450 msgr2=0x7fe6fc1368b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc136450 0x7fe6fc1368b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d150 tx=0x7fe6ec01eb20 comp rx=0 tx=0).stop 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- 172.21.15.98:0/2452952554 shutdown_connections 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc136450 0x7fe6fc1368b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc09df80 0x7fe6fc0a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 --2- 172.21.15.98:0/2452952554 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc0a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- 172.21.15.98:0/2452952554 >> 172.21.15.98:0/2452952554 conn(0x7fe6fc0a4780 msgr2=0x7fe6fc0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- 172.21.15.98:0/2452952554 shutdown_connections 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- 172.21.15.98:0/2452952554 wait complete. 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: monmap: 2024-09-11T02:52:09.436 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.437 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 librados: starting msgr at 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 librados: starting objecter 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 Processor -- start 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- start start 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 librados: setting wanted keys 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 librados: calling monclient init 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: init 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: _add_conns ranks=[0,6,7,3,1,4,5,8,2] 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc09df80 0x7fe6fc13aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): picked mon.noname-a con 0x7fe6fc09df80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:09.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc13b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): picked mon.noname-d con 0x7fe6fc0a3690 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc136450 0x7fe6fc13b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): picked mon.noname-i con 0x7fe6fc136450 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6fc09d260 con 0x7fe6fc09df80 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe6fc0a5590 con 0x7fe6fc0a3690 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe6fc09d3e0 con 0x7fe6fc136450 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient(hunting): authenticate will time out at 1083.302996s 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc09df80 0x7fe6fc13aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc13b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc09df80 0x7fe6fc13aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59478/0 (socket says 172.21.15.98:59478) 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc13b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58342/0 (socket says 172.21.15.98:58342) 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 1 -- 172.21.15.98:0/1299361592 learned_addr learned my addr 172.21.15.98:0/1299361592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 10 monclient(hunting): get_auth_request con 0x7fe6fc09df80 auth_method 0 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): get_auth_request con 0x7fe6fc0a3690 auth_method 0 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient(hunting): handle_auth_done global_id 14181 payload 824 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fad9d640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc136450 0x7fe6fc13b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc136450 msgr2=0x7fe6fc13b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe6fc136450 0x7fe6fc13b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc09df80 msgr2=0x7fe6fc13aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc09df80 0x7fe6fc13aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 monclient: found mon.noname-d 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 -- 172.21.15.98:0/1299361592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6fc120740 con 0x7fe6fc0a3690 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6fa59c640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6fc09df80 0x7fe6fc13aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f9d9b640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc13b430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000c7c0 tx=0x7fe6f000cc80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 client.?.objecter ms_handle_connect 0x7fe6fc0a3690 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6f0015070 con 0x7fe6fc0a3690 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: dump: 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.442 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f0007a90 con 0x7fe6fc0a3690 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 5 monclient: authenticate success, global_id 14181 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: _renew_subs 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6fc0a73b0 con 0x7fe6fc0a3690 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6f0007c40 con 0x7fe6fc0a3690 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6eaffd640 10 monclient: dump: 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.444 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.445 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe6f8d99640 4 set_mon_vals no callback set 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 client.14181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: _renew_subs 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6fc120460 con 0x7fe6fc0a3690 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.419+0000 7fe700897640 1 librados: init done 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe700897640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe700897640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137670 0x7fe6fc137a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe6fc0a4d80 con 0x7fe6fc137670 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137670 0x7fe6fc137a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient: get_auth_request con 0x7fe6fc137670 auth_method 0 2024-09-11T02:52:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe6f0007620 con 0x7fe6fc0a3690 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_done global_id 14190 payload 824 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 1 -- 172.21.15.98:0/1299361592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6fc0a7900 con 0x7fe6fc137670 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6dc03df40 0x7fe6dc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(84..84 src has 1..84) ==== 8725+0+0 (secure 0 0 0) 0x7fe6f0050420 con 0x7fe6fc0a3690 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 client.14181.objecter ms_dispatch 0x7fe6fc000ba0 osd_map(84..84 src has 1..84) 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 3 client.14181.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T02:52:09.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 3 client.14181.objecter handle_osd_map decoding full epoch 84 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fa59c640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137670 0x7fe6fc137a50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009ef0 tx=0x7fe6f400f610 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 20 client.14181.objecter dump_active .. 0 homeless 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 client.14181.objecter ms_handle_connect 0x7fe6fc137670 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 client.14181.objecter resend_mon_ops 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6f40148b0 con 0x7fe6fc137670 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 monclient: dump: 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.448 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fad9d640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6dc03df40 0x7fe6dc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fad9d640 10 monclient: get_auth_request con 0x7fe6dc03df40 auth_method 0 2024-09-11T02:52:09.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6f4014440 con 0x7fe6fc137670 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137670 msgr2=0x7fe6fc137a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137670 0x7fe6fc137a50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009ef0 tx=0x7fe6f400f610 comp rx=0 tx=0).stop 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137670 msgr2=0x7fe6fc137a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137670 0x7fe6fc137a50 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009ef0 tx=0x7fe6f400f610 comp rx=0 tx=0).stop 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6eaffd640 10 client.14181.objecter ms_handle_connect 0x7fe6dc03df40 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.423+0000 7fe6fad9d640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6dc03df40 0x7fe6dc040400 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d150 tx=0x7fe6ec0a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe700897640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe700897640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137f90 0x7fe6fc1d76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe700897640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe6fc06ca10 con 0x7fe6fc137f90 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137f90 0x7fe6fc1d76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient: get_auth_request con 0x7fe6fc137f90 auth_method 0 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 10 monclient(hunting): handle_auth_done global_id 14199 payload 824 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.427+0000 7fe6fa59c640 1 -- 172.21.15.98:0/1299361592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6fc068b40 con 0x7fe6fc137f90 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 10 client.14181.objecter ms_handle_connect 0x7fe6fc137f90 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6fa59c640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137f90 0x7fe6fc1d76b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009ef0 tx=0x7fe6f40046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 10 client.14181.objecter resend_mon_ops 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6f4022b00 con 0x7fe6fc137f90 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 10 monclient: dump: 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9265 (secure 0 0 0) 0x7fe6f402b480 con 0x7fe6fc137f90 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137f90 msgr2=0x7fe6fc1d76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137f90 0x7fe6fc1d76b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009ef0 tx=0x7fe6f40046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137f90 msgr2=0x7fe6fc1d76b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6eaffd640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc137f90 0x7fe6fc1d76b0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4009ef0 tx=0x7fe6f40046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 9483, 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:09.454 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.455 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:09.457 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:09.458 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.459 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:09.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:09.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:09.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:09.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:09.476 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 10 librados: watch_flush enter 2024-09-11T02:52:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 10 librados: watch_flush exit 2024-09-11T02:52:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 20 client.14181.objecter shutdown clearing up homeless session... 2024-09-11T02:52:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 10 client.14181.objecter successfully canceled tick 2024-09-11T02:52:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6dc03df40 msgr2=0x7fe6dc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6dc03df40 0x7fe6dc040400 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d150 tx=0x7fe6ec0a9070 comp rx=0 tx=0).stop 2024-09-11T02:52:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 10 monclient: shutdown 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 msgr2=0x7fe6fc13b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 --2- 172.21.15.98:0/1299361592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe6fc0a3690 0x7fe6fc13b430 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000c7c0 tx=0x7fe6f000cc80 comp rx=0 tx=0).stop 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6fad9d640 1 -- 172.21.15.98:0/1299361592 reap_dead start 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe6fad9d640 1 -- 172.21.15.98:0/1299361592 reap_dead start 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 shutdown_connections 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 >> 172.21.15.98:0/1299361592 conn(0x7fe6fc0a4780 msgr2=0x7fe6fc13d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 shutdown_connections 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 -- 172.21.15.98:0/1299361592 wait complete. 2024-09-11T02:52:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.431+0000 7fe700897640 1 librados: shutdown 2024-09-11T02:52:09.479 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:09.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:52:09.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:52:09.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient: get_monmap_and_config 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient: monmap: 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:09.590134+0000 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:09.590134+0000 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.589 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.590 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.590 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.590 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.590 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.590 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:09.590 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:09.590 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 1 Processor -- start 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 1 -- start start 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient: init 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient: _add_conns ranks=[0,8,1,6,10,9,4,2,3,7,5] 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d541397d0 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5409d4c0 0x7f9d5413e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient(hunting): picked mon.noname-e con 0x7f9d5409d4c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9d5413ec90 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:09.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5409de80 con 0x7f9d541397d0 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5409e440 con 0x7f9d5413ec90 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9d5409cbe0 con 0x7f9d5409d4c0 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d5afd3640 10 monclient(hunting): authenticate will time out at 1083.470424s 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5409d4c0 0x7f9d5413e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d58d6f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9d541397d0 0x7f9d54139bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37900/0 (socket says 172.21.15.98:37900) 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): get_auth_request con 0x7f9d5409d4c0 auth_method 0 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d58d6f640 1 -- 172.21.15.98:0/3869184015 learned_addr learned my addr 172.21.15.98:0/3869184015 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient(hunting): handle_auth_done global_id 23381 payload 274 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d537fe640 1 -- 172.21.15.98:0/3869184015 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2082684924 0 0) 0x7f9d5409de80 con 0x7f9d541397d0 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 1 -- 172.21.15.98:0/3869184015 >> v1:172.21.15.190:6789/0 conn(0x7f9d5413ec90 legacy=0x7f9d54141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 1 -- 172.21.15.98:0/3869184015 >> v1:172.21.15.98:6789/0 conn(0x7f9d541397d0 legacy=0x7f9d54139bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 1 monclient: found mon.noname-e 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d53fff640 1 -- 172.21.15.98:0/3869184015 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5413a2c0 con 0x7f9d5409d4c0 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.587+0000 7f9d537fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:52:09.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 --2- 172.21.15.98:0/3869184015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5409d4c0 0x7f9d5413e750 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009dc0 tx=0x7f9d3c01c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=fb00e7a804d1c22a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 1 -- 172.21.15.98:0/3869184015 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d3c0b2070 con 0x7f9d5409d4c0 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: dump: 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 5 monclient: authenticate success, global_id 23381 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 1 -- 172.21.15.98:0/3869184015 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d3c0a5a50 con 0x7f9d5409d4c0 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:09.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 4 set_mon_vals no callback set 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 1 -- 172.21.15.98:0/3869184015 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d3c0a5c40 con 0x7f9d5409d4c0 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d537fe640 10 monclient: dump: 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient: shutdown 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- 172.21.15.98:0/3869184015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5409d4c0 msgr2=0x7f9d5413e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 --2- 172.21.15.98:0/3869184015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5409d4c0 0x7f9d5413e750 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009dc0 tx=0x7f9d3c01c850 comp rx=0 tx=0).stop 2024-09-11T02:52:09.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- 172.21.15.98:0/3869184015 shutdown_connections 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 --2- 172.21.15.98:0/3869184015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5409d4c0 0x7f9d5413e750 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- 172.21.15.98:0/3869184015 >> 172.21.15.98:0/3869184015 conn(0x7f9d540a4780 msgr2=0x7f9d540a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- 172.21.15.98:0/3869184015 shutdown_connections 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- 172.21.15.98:0/3869184015 wait complete. 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient: monmap: 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:09.601 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 librados: starting msgr at 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 librados: starting objecter 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 Processor -- start 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- start start 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 librados: setting wanted keys 2024-09-11T02:52:09.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 librados: calling monclient init 2024-09-11T02:52:09.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient: init 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient: _add_conns ranks=[6,0,5,2,1,8,3,7,4] 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9d5409d4c0 0x7f9d541205d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): picked mon.noname-d con 0x7f9d5409d4c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d541397d0 0x7f9d54138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): picked mon.noname-a con 0x7f9d541397d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9d5413ec90 0x7f9d5411e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): picked mon.noname-h con 0x7f9d5413ec90 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541451d0 con 0x7f9d541397d0 2024-09-11T02:52:09.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9d540517e0 con 0x7f9d5413ec90 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9d54145350 con 0x7f9d5409d4c0 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d5afd3640 10 monclient(hunting): authenticate will time out at 1083.475300s 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d541397d0 0x7f9d54138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9d5409d4c0 0x7f9d541205d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d541397d0 0x7f9d54138450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59482/0 (socket says 172.21.15.98:59482) 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 10 monclient(hunting): get_auth_request con 0x7f9d5409d4c0 auth_method 0 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 -- 172.21.15.98:0/2713401396 learned_addr learned my addr 172.21.15.98:0/2713401396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): get_auth_request con 0x7f9d541397d0 auth_method 0 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d59570640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9d5413ec90 0x7f9d5411e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d59570640 10 monclient(hunting): get_auth_request con 0x7f9d5413ec90 auth_method 0 2024-09-11T02:52:09.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d59570640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d59570640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d59570640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient(hunting): handle_auth_done global_id 14166 payload 824 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9d5409d4c0 msgr2=0x7f9d541205d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9d5409d4c0 0x7f9d541205d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9d5413ec90 msgr2=0x7f9d5411e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9d5413ec90 0x7f9d5411e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 monclient: found mon.noname-a 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d53fff640 1 -- 172.21.15.98:0/2713401396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d54120f60 con 0x7f9d541397d0 2024-09-11T02:52:09.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.591+0000 7f9d58d6f640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9d5409d4c0 0x7f9d541205d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d53fff640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d541397d0 0x7f9d54138450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c01d0f0 tx=0x7f9d3c0195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 client.?.objecter ms_handle_connect 0x7f9d541397d0 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d3c0b2050 con 0x7f9d541397d0 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: dump: 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 5 monclient: authenticate success, global_id 14166 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d3c0be070 con 0x7f9d541397d0 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 10 monclient: _renew_subs 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d54121dd0 con 0x7f9d541397d0 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d3c0a58b0 con 0x7f9d541397d0 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d52ffd640 4 set_mon_vals no callback set 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: dump: 2024-09-11T02:52:09.616 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.617 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 10 monclient: _renew_subs 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d54142170 con 0x7f9d541397d0 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 1 librados: init done 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5411e810 0x7f9d5411ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9d5406ca10 con 0x7f9d5411e810 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5411e810 0x7f9d5411ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9d3c003bb0 con 0x7f9d541397d0 2024-09-11T02:52:09.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient: get_auth_request con 0x7f9d5411e810 auth_method 0 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_done global_id 23390 payload 824 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 1 -- 172.21.15.98:0/2713401396 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d5411f130 con 0x7f9d5411e810 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9d3403df70 0x7f9d34040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8725+0+0 (secure 0 0 0) 0x7f9d3c0eb2d0 con 0x7f9d541397d0 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 client.14166.objecter ms_dispatch 0x7f9d54000ba0 osd_map(84..84 src has 1..84) 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 3 client.14166.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T02:52:09.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 3 client.14166.objecter handle_osd_map decoding full epoch 84 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d58d6f640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5411e810 0x7f9d5411ebf0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007fd0 tx=0x7f9d44001d70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 20 client.14166.objecter dump_active .. 0 homeless 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 client.14166.objecter ms_handle_connect 0x7f9d5411e810 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 client.14166.objecter resend_mon_ops 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d4400fe10 con 0x7f9d5411e810 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: dump: 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.620 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d59570640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9d3403df70 0x7f9d34040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d59570640 10 monclient: get_auth_request con 0x7f9d3403df70 auth_method 0 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9d44018070 con 0x7f9d5411e810 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5411e810 msgr2=0x7f9d5411ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5411e810 0x7f9d5411ebf0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007fd0 tx=0x7f9d44001d70 comp rx=0 tx=0).stop 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5411e810 msgr2=0x7f9d5411ebf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5411e810 0x7f9d5411ebf0 secure :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007fd0 tx=0x7f9d44001d70 comp rx=0 tx=0).stop 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d517fa640 10 client.14166.objecter ms_handle_connect 0x7f9d3403df70 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.595+0000 7f9d59570640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9d3403df70 0x7f9d34040430 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00b790 tx=0x7f9d4c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5413a490 0x7f9d5413a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9d5406e870 con 0x7f9d5413a490 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5413a490 0x7f9d5413a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient: get_auth_request con 0x7f9d5413a490 auth_method 0 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 10 monclient(hunting): handle_auth_done global_id 23399 payload 824 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 1 -- 172.21.15.98:0/2713401396 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d54068b40 con 0x7f9d5413a490 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d58d6f640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5413a490 0x7f9d5413a8a0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007fd0 tx=0x7f9d44000ee0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 10 client.14166.objecter ms_handle_connect 0x7f9d5413a490 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 10 client.14166.objecter resend_mon_ops 2024-09-11T02:52:09.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9d44013a70 con 0x7f9d5413a490 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 10 monclient: dump: 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.624 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f9d44013600 con 0x7f9d5413a490 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5413a490 msgr2=0x7f9d5413a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5413a490 0x7f9d5413a8a0 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007fd0 tx=0x7f9d44000ee0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5413a490 msgr2=0x7f9d5413a8a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d517fa640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9d5413a490 0x7f9d5413a8a0 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145395, 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:09.628 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:09.630 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.631 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:09.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:09.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.640 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:09.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.645 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.647 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:09.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.649 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:09.649 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 10 librados: watch_flush enter 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 10 librados: watch_flush exit 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 20 client.14166.objecter shutdown clearing up homeless session... 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 10 client.14166.objecter successfully canceled tick 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9d3403df70 msgr2=0x7f9d34040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9d3403df70 0x7f9d34040430 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c00b790 tx=0x7f9d4c008040 comp rx=0 tx=0).stop 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 10 monclient: shutdown 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d541397d0 msgr2=0x7f9d54138450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d59570640 1 -- 172.21.15.98:0/2713401396 reap_dead start 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d541397d0 0x7f9d54138450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c01d0f0 tx=0x7f9d3c0195f0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 shutdown_connections 2024-09-11T02:52:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 --2- 172.21.15.98:0/2713401396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d541397d0 0x7f9d54138450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 >> 172.21.15.98:0/2713401396 conn(0x7f9d540a4780 msgr2=0x7f9d54141080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 shutdown_connections 2024-09-11T02:52:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 -- 172.21.15.98:0/2713401396 wait complete. 2024-09-11T02:52:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.603+0000 7f9d5afd3640 1 librados: shutdown 2024-09-11T02:52:09.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:09.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:52:09.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient: get_monmap_and_config 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient: monmap: 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:09.769664+0000 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:09.769664+0000 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.768 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:09.769 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 Processor -- start 2024-09-11T02:52:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 -- start start 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient: init 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient: _add_conns ranks=[7,3,9,8,6,1,2,5,4,0,10] 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8139b80 0x7f39f8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): picked mon.noname-i con 0x7f39f8139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39f806fe50 0x7f39f80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): picked mon.noname-g con 0x7f39f806fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39f8072640 0x7f39f80a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): picked mon.noname-a con 0x7f39f8072640 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f39f80719c0 con 0x7f39f8072640 2024-09-11T02:52:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f39f8071c10 con 0x7f39f806fe50 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f39f80702d0 con 0x7f39f8139b80 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f3a00630640 10 monclient(hunting): authenticate will time out at 1083.649806s 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39f8072640 0x7f39f80a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f39f8072640 0x7f39f80a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59486/0 (socket says 172.21.15.98:59486) 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 -- 172.21.15.98:0/2303965610 learned_addr learned my addr 172.21.15.98:0/2303965610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): get_auth_request con 0x7f39f8072640 auth_method 0 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39fe3cc640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8139b80 0x7f39f8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39fdbcb640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39f806fe50 0x7f39f80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39fe3cc640 10 monclient(hunting): get_auth_request con 0x7f39f8139b80 auth_method 0 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39fe3cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39fe3cc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39fe3cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient(hunting): handle_auth_done global_id 14175 payload 274 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 -- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8139b80 msgr2=0x7f39f8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8139b80 0x7f39f8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 -- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39f806fe50 msgr2=0x7f39f80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39f806fe50 0x7f39f80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 monclient: found mon.noname-a 2024-09-11T02:52:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.767+0000 7f39febcd640 1 -- 172.21.15.98:0/2303965610 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f8071370 con 0x7f39f8072640 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8072640 0x7f39f80a7f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39f00098a0 tx=0x7f39f001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=166882bff51e6423 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 1 -- 172.21.15.98:0/2303965610 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39f00b0070 con 0x7f39f8072640 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: dump: 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 1 -- 172.21.15.98:0/2303965610 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f00aa810 con 0x7f39f8072640 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 5 monclient: authenticate success, global_id 14175 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 1 -- 172.21.15.98:0/2303965610 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39f00aa3a0 con 0x7f39f8072640 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 4 set_mon_vals no callback set 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fd3ca640 10 monclient: dump: 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient: shutdown 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- 172.21.15.98:0/2303965610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8072640 msgr2=0x7f39f80a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8072640 0x7f39f80a7f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39f00098a0 tx=0x7f39f001c400 comp rx=0 tx=0).stop 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- 172.21.15.98:0/2303965610 shutdown_connections 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8072640 0x7f39f80a7f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f39f806fe50 0x7f39f80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 --2- 172.21.15.98:0/2303965610 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8139b80 0x7f39f8139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- 172.21.15.98:0/2303965610 >> 172.21.15.98:0/2303965610 conn(0x7f39f80ac730 msgr2=0x7f39f80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- 172.21.15.98:0/2303965610 shutdown_connections 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- 172.21.15.98:0/2303965610 wait complete. 2024-09-11T02:52:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient: monmap: 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 librados: starting msgr at 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 librados: starting objecter 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 Processor -- start 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- start start 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 librados: setting wanted keys 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 librados: calling monclient init 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient: init 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient: _add_conns ranks=[6,7,0,3,2,4,1,8,5] 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f39f806fe50 0x7f39f8134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): picked mon.noname-d con 0x7f39f806fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8072640 0x7f39f8134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): picked mon.noname-i con 0x7f39f8072640 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8139b80 0x7f39f813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): picked mon.noname-a con 0x7f39f8139b80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81435c0 con 0x7f39f8139b80 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f39f8143440 con 0x7f39f806fe50 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f39f8143740 con 0x7f39f8072640 2024-09-11T02:52:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f3a00630640 10 monclient(hunting): authenticate will time out at 1083.655110s 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8139b80 0x7f39f813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f39f806fe50 0x7f39f8134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f39f806fe50 0x7f39f8134a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58388/0 (socket says 172.21.15.98:58388) 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8139b80 0x7f39f813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59498/0 (socket says 172.21.15.98:59498) 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 1 -- 172.21.15.98:0/1152542311 learned_addr learned my addr 172.21.15.98:0/1152542311 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 10 monclient(hunting): get_auth_request con 0x7f39f806fe50 auth_method 0 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): get_auth_request con 0x7f39f8139b80 auth_method 0 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fdbcb640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8072640 0x7f39f8134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fdbcb640 10 monclient(hunting): get_auth_request con 0x7f39f8072640 auth_method 0 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fdbcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fdbcb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fdbcb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient(hunting): handle_auth_done global_id 14184 payload 824 2024-09-11T02:52:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8072640 msgr2=0x7f39f8134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f39f8072640 0x7f39f8134f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f39f806fe50 msgr2=0x7f39f8134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f39f806fe50 0x7f39f8134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 monclient: found mon.noname-a 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 -- 172.21.15.98:0/1152542311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f813ddb0 con 0x7f39f8139b80 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39fe3cc640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f39f806fe50 0x7f39f8134a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39febcd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8139b80 0x7f39f813a6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f39f00ab000 tx=0x7f39f00191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 10 client.?.objecter ms_handle_connect 0x7f39f8139b80 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39f00b9070 con 0x7f39f8139b80 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 10 monclient: dump: 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.771+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39f00044b0 con 0x7f39f8139b80 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39f00b0040 con 0x7f39f8139b80 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 5 monclient: authenticate success, global_id 14184 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 10 monclient: _renew_subs 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f813c1d0 con 0x7f39f8139b80 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fcbc9640 4 set_mon_vals no callback set 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: dump: 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 10 monclient: _renew_subs 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f8135810 con 0x7f39f8139b80 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 1 librados: init done 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f39f0004040 con 0x7f39f8139b80 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813abf0 0x7f39f813afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f39f813c890 con 0x7f39f813abf0 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f39dc03df30 0x7f39dc0403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8725+0+0 (secure 0 0 0) 0x7f39f00e9340 con 0x7f39f8139b80 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 client.14184.objecter ms_dispatch 0x7f39f8000ba0 osd_map(84..84 src has 1..84) 2024-09-11T02:52:09.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 3 client.14184.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 3 client.14184.objecter handle_osd_map decoding full epoch 84 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 20 client.14184.objecter dump_active .. 0 homeless 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813abf0 0x7f39f813afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient: get_auth_request con 0x7f39f813abf0 auth_method 0 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fdbcb640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f39dc03df30 0x7f39dc0403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fdbcb640 10 monclient: get_auth_request con 0x7f39dc03df30 auth_method 0 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_done global_id 24076 payload 824 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 1 -- 172.21.15.98:0/1152542311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39f813bd80 con 0x7f39f813abf0 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 client.14184.objecter ms_handle_connect 0x7f39dc03df30 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fdbcb640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f39dc03df30 0x7f39dc0403f0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a5bc0 tx=0x7f39e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 client.14184.objecter ms_handle_connect 0x7f39f813abf0 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 client.14184.objecter resend_mon_ops 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39fe3cc640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813abf0 0x7f39f813afd0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f39f40049c0 tx=0x7f39f40077c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39f4014ac0 con 0x7f39f813abf0 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: dump: 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f39f401a070 con 0x7f39f813abf0 2024-09-11T02:52:09.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:09.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813abf0 msgr2=0x7f39f813afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813abf0 0x7f39f813afd0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f39f40049c0 tx=0x7f39f40077c0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813abf0 msgr2=0x7f39f813afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.775+0000 7f39eeffd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813abf0 0x7f39f813afd0 secure :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f39f40049c0 tx=0x7f39f40077c0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:09.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813b510 0x7f39f80a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f39f806ca10 con 0x7f39f813b510 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813b510 0x7f39f80a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient: get_auth_request con 0x7f39f813b510 auth_method 0 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 10 monclient(hunting): handle_auth_done global_id 24085 payload 824 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 1 -- 172.21.15.98:0/1152542311 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39f80aa330 con 0x7f39f813b510 2024-09-11T02:52:09.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 10 client.14184.objecter ms_handle_connect 0x7f39f813b510 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 10 client.14184.objecter resend_mon_ops 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39fe3cc640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813b510 0x7f39f80a0e60 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f39f40049c0 tx=0x7f39f4004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f39f4015420 con 0x7f39f813b510 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 10 monclient: dump: 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.804 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7f39f4014b50 con 0x7f39f813b510 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:09.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813b510 msgr2=0x7f39f80a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813b510 0x7f39f80a0e60 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f39f40049c0 tx=0x7f39f4004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813b510 msgr2=0x7f39f80a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39eeffd640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39f813b510 0x7f39f80a0e60 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145552, 2024-09-11T02:52:09.806 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:09.807 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:09.810 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.811 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.813 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:09.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.818 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.820 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.823 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.825 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.828 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:09.829 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:09.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 10 librados: watch_flush enter 2024-09-11T02:52:09.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 10 librados: watch_flush exit 2024-09-11T02:52:09.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 20 client.14184.objecter shutdown clearing up homeless session... 2024-09-11T02:52:09.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 10 client.14184.objecter successfully canceled tick 2024-09-11T02:52:09.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f39dc03df30 msgr2=0x7f39dc0403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f39dc03df30 0x7f39dc0403f0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a5bc0 tx=0x7f39e8008040 comp rx=0 tx=0).stop 2024-09-11T02:52:09.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 10 monclient: shutdown 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8139b80 msgr2=0x7f39f813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f3a00630640 1 --2- 172.21.15.98:0/1152542311 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39f8139b80 0x7f39f813a6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f39f00ab000 tx=0x7f39f00191a0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.783+0000 7f39febcd640 1 -- 172.21.15.98:0/1152542311 reap_dead start 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.787+0000 7f39febcd640 1 -- 172.21.15.98:0/1152542311 reap_dead start 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.787+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 shutdown_connections 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.787+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 >> 172.21.15.98:0/1152542311 conn(0x7f39f80ac730 msgr2=0x7f39f80ab540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.787+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 shutdown_connections 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.787+0000 7f3a00630640 1 -- 172.21.15.98:0/1152542311 wait complete. 2024-09-11T02:52:09.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.787+0000 7f3a00630640 1 librados: shutdown 2024-09-11T02:52:09.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:09.833 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:52:09.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:52:09.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.943+0000 7fb5b30a8640 10 monclient: get_monmap_and_config 2024-09-11T02:52:09.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.943+0000 7fb5b30a8640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.943+0000 7fb5b30a8640 10 monclient: monmap: 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:09.949167+0000 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:09.949167+0000 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.948 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.949 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.949 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.949 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.949 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:09.949 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:09.949 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 1 Processor -- start 2024-09-11T02:52:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 1 -- start start 2024-09-11T02:52:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient: init 2024-09-11T02:52:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient: _add_conns ranks=[7,2,1,9,8,10,0,3,5,4,6] 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139b10 0x7fb5ac072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient(hunting): picked mon.noname-i con 0x7fb5ac139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5ac139ef0 0x7fb5ac0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb5ac13a2d0 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5ac0702b0 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac071bf0 con 0x7fb5ac139ef0 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb5ac071350 con 0x7fb5ac139b10 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b30a8640 10 monclient(hunting): authenticate will time out at 1083.829442s 2024-09-11T02:52:09.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5ac139ef0 0x7fb5ac0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5ac139ef0 0x7fb5ac0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56550/0 (socket says 172.21.15.98:56550) 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 -- 172.21.15.98:0/3007028464 learned_addr learned my addr 172.21.15.98:0/3007028464 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): get_auth_request con 0x7fb5ac139ef0 auth_method 0 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 1 --2- 172.21.15.98:0/3007028464 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139b10 0x7fb5ac072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 10 monclient(hunting): get_auth_request con 0x7fb5ac139b10 auth_method 0 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient(hunting): handle_auth_done global_id 23006 payload 274 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 -- 172.21.15.98:0/3007028464 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139b10 msgr2=0x7fb5ac072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 --2- 172.21.15.98:0/3007028464 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139b10 0x7fb5ac072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 -- 172.21.15.98:0/3007028464 >> v1:172.21.15.190:6789/0 conn(0x7fb5ac13a2d0 legacy=0x7fb5ac0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 monclient: found mon.noname-b 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.947+0000 7fb5abfff640 1 -- 172.21.15.98:0/3007028464 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac0a9620 con 0x7fb5ac139ef0 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5abfff640 1 --2- 172.21.15.98:0/3007028464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5ac139ef0 0x7fb5ac0a7a10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb5940098a0 tx=0x7fb59401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3339371f95bfc572 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 1 -- 172.21.15.98:0/3007028464 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5940b1070 con 0x7fb5ac139ef0 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: dump: 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.965 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 1 -- 172.21.15.98:0/3007028464 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5940aa7b0 con 0x7fb5ac139ef0 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 5 monclient: authenticate success, global_id 23006 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 1 -- 172.21.15.98:0/3007028464 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5940aae10 con 0x7fb5ac139ef0 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 4 set_mon_vals no callback set 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:09.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5ab7fe640 10 monclient: dump: 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.968 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: shutdown 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- 172.21.15.98:0/3007028464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5ac139ef0 msgr2=0x7fb5ac0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 --2- 172.21.15.98:0/3007028464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5ac139ef0 0x7fb5ac0a7a10 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb5940098a0 tx=0x7fb59401c440 comp rx=0 tx=0).stop 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- 172.21.15.98:0/3007028464 shutdown_connections 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 --2- 172.21.15.98:0/3007028464 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5ac139ef0 0x7fb5ac0a7a10 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 --2- 172.21.15.98:0/3007028464 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139b10 0x7fb5ac072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- 172.21.15.98:0/3007028464 >> 172.21.15.98:0/3007028464 conn(0x7fb5ac0ac790 msgr2=0x7fb5ac0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:09.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- 172.21.15.98:0/3007028464 shutdown_connections 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- 172.21.15.98:0/3007028464 wait complete. 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: build_initial_monmap 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: monmap: 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:09.970 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 librados: starting msgr at 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 librados: starting objecter 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 Processor -- start 2024-09-11T02:52:09.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- start start 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 librados: setting wanted keys 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 librados: calling monclient init 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: init 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: _add_conns ranks=[5,7,4,1,6,0,3,2,8] 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5ac139b10 0x7fb5ac13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): picked mon.noname-h con 0x7fb5ac139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139ef0 0x7fb5ac13ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): picked mon.noname-i con 0x7fb5ac139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5ac13a2d0 0x7fb5ac13b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb5ac13a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb5ac144470 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb5ac0a8870 con 0x7fb5ac139b10 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb5ac1445f0 con 0x7fb5ac139ef0 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient(hunting): authenticate will time out at 1083.834529s 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5ac13a2d0 0x7fb5ac13b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5ac13a2d0 0x7fb5ac13b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45858/0 (socket says 172.21.15.98:45858) 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 -- 172.21.15.98:0/217150580 learned_addr learned my addr 172.21.15.98:0/217150580 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): get_auth_request con 0x7fb5ac13a2d0 auth_method 0 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5abfff640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139ef0 0x7fb5ac13ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b0e44640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5ac139b10 0x7fb5ac13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5abfff640 10 monclient(hunting): get_auth_request con 0x7fb5ac139ef0 auth_method 0 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5abfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b0e44640 10 monclient(hunting): get_auth_request con 0x7fb5ac139b10 auth_method 0 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b0e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b0e44640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b0e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient(hunting): handle_auth_done global_id 22972 payload 824 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139ef0 msgr2=0x7fb5ac13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5ac139ef0 0x7fb5ac13ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5ac139b10 msgr2=0x7fb5ac13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5ac139b10 0x7fb5ac13a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient: _finish_hunting 0 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 monclient: found mon.noname-c 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:09.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 -- 172.21.15.98:0/217150580 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac0a5d50 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b1645640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5ac13a2d0 0x7fb5ac13b250 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c3c0 tx=0x7fb59c00c880 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 client.?.objecter ms_handle_connect 0x7fb5ac13a2d0 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb59c017070 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: dump: 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.976 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: _finish_auth 0 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: _check_auth_tickets 2024-09-11T02:52:09.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c003600 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 5 monclient: authenticate success, global_id 22972 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: _renew_subs 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac13ce20 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb59c003c60 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5aa7fc640 4 set_mon_vals no callback set 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5a97fa640 10 monclient: dump: 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.978 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 client.22972.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: _renew_subs 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac13d100 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.951+0000 7fb5b30a8640 1 librados: init done 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b30a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b30a8640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac1337f0 0x7fb5ac133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b30a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac140fe0 con 0x7fb5ac1337f0 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb59c0037f0 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac1337f0 0x7fb5ac133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb58c03df70 0x7fb58c040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient: get_auth_request con 0x7fb5ac1337f0 auth_method 0 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(84..84 src has 1..84) ==== 8725+0+0 (secure 0 0 0) 0x7fb59c04f220 con 0x7fb5ac13a2d0 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 client.22972.objecter ms_dispatch 0x7fb5ac000ba0 osd_map(84..84 src has 1..84) 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 3 client.22972.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 3 client.22972.objecter handle_osd_map decoding full epoch 84 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 20 client.22972.objecter dump_active .. 0 homeless 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5abfff640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb58c03df70 0x7fb58c040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5abfff640 10 monclient: get_auth_request con 0x7fb58c03df70 auth_method 0 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_done global_id 19425 payload 824 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 1 -- 172.21.15.98:0/217150580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5ac074a50 con 0x7fb5ac1337f0 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 client.22972.objecter ms_handle_connect 0x7fb5ac1337f0 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5b0e44640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac1337f0 0x7fb5ac133bd0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4007f00 tx=0x7fb5a40115a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 client.22972.objecter resend_mon_ops 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a4003e10 con 0x7fb5ac1337f0 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 monclient: dump: 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 client.22972.objecter ms_handle_connect 0x7fb58c03df70 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5abfff640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb58c03df70 0x7fb58c040430 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7fb5940098a0 tx=0x7fb5940a6070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5a40039a0 con 0x7fb5ac1337f0 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac1337f0 msgr2=0x7fb5ac133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac1337f0 0x7fb5ac133bd0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4007f00 tx=0x7fb5a40115a0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac1337f0 msgr2=0x7fb5ac133bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.955+0000 7fb5a97fa640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac1337f0 0x7fb5ac133bd0 secure :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4007f00 tx=0x7fb5a40115a0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac134110 0x7fb5ac0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac06ca10 con 0x7fb5ac134110 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac134110 0x7fb5ac0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient: get_auth_request con 0x7fb5ac134110 auth_method 0 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:09.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 10 monclient(hunting): handle_auth_done global_id 19434 payload 824 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 1 -- 172.21.15.98:0/217150580 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5ac06e680 con 0x7fb5ac134110 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b0e44640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac134110 0x7fb5ac0a0e60 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4007f00 tx=0x7fb5a4004a50 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 10 client.22972.objecter ms_handle_connect 0x7fb5ac134110 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 10 client.22972.objecter resend_mon_ops 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a40045e0 con 0x7fb5ac134110 2024-09-11T02:52:09.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 10 monclient: dump: 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:09.988 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fb5a40045e0 con 0x7fb5ac134110 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:09.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac134110 msgr2=0x7fb5ac0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac134110 0x7fb5ac0a0e60 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4007f00 tx=0x7fb5a4004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac134110 msgr2=0x7fb5ac0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5a97fa640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5ac134110 0x7fb5ac0a0e60 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:09.990 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145712, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.991 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.993 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:09.994 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:09.995 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:09.995 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:09.996 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.997 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:09.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:09.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:10.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:10.006 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:10.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.008 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:10.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.010 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.012 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:10.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:10.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:10.016 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:10.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 10 librados: watch_flush enter 2024-09-11T02:52:10.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 10 librados: watch_flush exit 2024-09-11T02:52:10.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 20 client.22972.objecter shutdown clearing up homeless session... 2024-09-11T02:52:10.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 10 client.22972.objecter successfully canceled tick 2024-09-11T02:52:10.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb58c03df70 msgr2=0x7fb58c040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb58c03df70 0x7fb58c040430 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7fb5940098a0 tx=0x7fb5940a6070 comp rx=0 tx=0).stop 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 10 monclient: shutdown 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5ac13a2d0 msgr2=0x7fb5ac13b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 --2- 172.21.15.98:0/217150580 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5ac13a2d0 0x7fb5ac13b250 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00c3c0 tx=0x7fb59c00c880 comp rx=0 tx=0).stop 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b1645640 1 -- 172.21.15.98:0/217150580 reap_dead start 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b1645640 1 -- 172.21.15.98:0/217150580 reap_dead start 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 shutdown_connections 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 >> 172.21.15.98:0/217150580 conn(0x7fb5ac0ac790 msgr2=0x7fb5ac0ab270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 shutdown_connections 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 -- 172.21.15.98:0/217150580 wait complete. 2024-09-11T02:52:10.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:09.963+0000 7fb5b30a8640 1 librados: shutdown 2024-09-11T02:52:10.020 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:10.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:52:10.020 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:52:10.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient: get_monmap_and_config 2024-09-11T02:52:10.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient: build_initial_monmap 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient: monmap: 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:10.141388+0000 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:10.141388+0000 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:10.141 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 1 Processor -- start 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 1 -- start start 2024-09-11T02:52:10.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient: init 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient: _add_conns ranks=[8,0,7,3,4,5,6,1,9,10,2] 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f124c0a46a0 0x7f124c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient(hunting): picked mon.noname-e con 0x7f124c0a46a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f124c0a4fc0 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f124c13a040 0x7f124c141c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient(hunting): picked mon.noname-i con 0x7f124c13a040 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124c0740c0 con 0x7f124c0a4fc0 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f124c0a1540 con 0x7f124c13a040 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f124c0a1790 con 0x7f124c0a46a0 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f1250b27640 10 monclient(hunting): authenticate will time out at 1084.021671s 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f124c0a46a0 0x7f124c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f124c0a46a0 0x7f124c0a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56844/0 (socket says 172.21.15.98:56844) 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 -- 172.21.15.98:0/521564191 learned_addr learned my addr 172.21.15.98:0/521564191 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): get_auth_request con 0x7f124c0a46a0 auth_method 0 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124ad9d640 1 --2- 172.21.15.98:0/521564191 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f124c13a040 0x7f124c141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124ad9d640 10 monclient(hunting): get_auth_request con 0x7f124c13a040 auth_method 0 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient(hunting): handle_auth_done global_id 23408 payload 274 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 -- 172.21.15.98:0/521564191 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f124c13a040 msgr2=0x7f124c141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 --2- 172.21.15.98:0/521564191 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f124c13a040 0x7f124c141c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 -- 172.21.15.98:0/521564191 >> v1:172.21.15.98:6789/0 conn(0x7f124c0a4fc0 legacy=0x7f124c139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient: _finish_hunting 0 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 monclient: found mon.noname-e 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124a59c640 1 -- 172.21.15.98:0/521564191 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124c09fe60 con 0x7f124c0a46a0 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124959a640 1 -- 172.21.15.98:0/521564191 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160518178 0 0) 0x7f124c0740c0 con 0x7f124c0a4fc0 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.139+0000 7f124959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 --2- 172.21.15.98:0/521564191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f124c0a46a0 0x7f124c0a4a80 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f12340098a0 tx=0x7f123401c3e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b11a2ee317927978 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 1 -- 172.21.15.98:0/521564191 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12340b0070 con 0x7f124c0a46a0 2024-09-11T02:52:10.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:10.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: dump: 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.156 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: _finish_auth 0 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: _check_auth_tickets 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 5 monclient: authenticate success, global_id 23408 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 1 -- 172.21.15.98:0/521564191 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12340aa810 con 0x7f124c0a46a0 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:10.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 4 set_mon_vals no callback set 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 1 -- 172.21.15.98:0/521564191 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12340a9300 con 0x7f124c0a46a0 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124959a640 10 monclient: dump: 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: shutdown 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- 172.21.15.98:0/521564191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f124c0a46a0 msgr2=0x7f124c0a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 --2- 172.21.15.98:0/521564191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f124c0a46a0 0x7f124c0a4a80 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f12340098a0 tx=0x7f123401c3e0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- 172.21.15.98:0/521564191 shutdown_connections 2024-09-11T02:52:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 --2- 172.21.15.98:0/521564191 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f124c13a040 0x7f124c141c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 --2- 172.21.15.98:0/521564191 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f124c0a46a0 0x7f124c0a4a80 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- 172.21.15.98:0/521564191 >> 172.21.15.98:0/521564191 conn(0x7f124c06ed00 msgr2=0x7f124c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- 172.21.15.98:0/521564191 shutdown_connections 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- 172.21.15.98:0/521564191 wait complete. 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: build_initial_monmap 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: monmap: 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 librados: starting msgr at 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 librados: starting objecter 2024-09-11T02:52:10.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 Processor -- start 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- start start 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 librados: setting wanted keys 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 librados: calling monclient init 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: init 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: _add_conns ranks=[4,5,2,8,0,1,7,6,3] 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f124c0a46a0 0x7f124c1d19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): picked mon.noname-c con 0x7f124c0a46a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c0a4fc0 0x7f124c1d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): picked mon.noname-h con 0x7f124c0a4fc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f124c13a040 0x7f124c1d2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): picked mon.noname-b con 0x7f124c13a040 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f124c144960 con 0x7f124c13a040 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f124c077860 con 0x7f124c0a46a0 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f124c144ae0 con 0x7f124c0a4fc0 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient(hunting): authenticate will time out at 1084.026622s 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f124c0a46a0 0x7f124c1d19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f124c0a46a0 0x7f124c1d19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45860/0 (socket says 172.21.15.98:45860) 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f124c13a040 0x7f124c1d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 -- 172.21.15.98:0/2095041129 learned_addr learned my addr 172.21.15.98:0/2095041129 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): get_auth_request con 0x7f124c0a46a0 auth_method 0 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 10 monclient(hunting): get_auth_request con 0x7f124c13a040 auth_method 0 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1249d9b640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c0a4fc0 0x7f124c1d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1249d9b640 10 monclient(hunting): get_auth_request con 0x7f124c0a4fc0 auth_method 0 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1249d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1249d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1249d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient(hunting): handle_auth_done global_id 22981 payload 824 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c0a4fc0 msgr2=0x7f124c1d1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c0a4fc0 0x7f124c1d1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f124c13a040 msgr2=0x7f124c1d2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f124c13a040 0x7f124c1d2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient: _finish_hunting 0 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 monclient: found mon.noname-c 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:10.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 -- 172.21.15.98:0/2095041129 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f124c140f40 con 0x7f124c0a46a0 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124ad9d640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f124c13a040 0x7f124c1d2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f124a59c640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f124c0a46a0 0x7f124c1d19f0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f123401aaf0 tx=0x7f12340191e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 client.?.objecter ms_handle_connect 0x7f124c0a46a0 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12340b9070 con 0x7f124c0a46a0 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: dump: 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.166 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: _finish_auth 0 2024-09-11T02:52:10.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12340044b0 con 0x7f124c0a46a0 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 5 monclient: authenticate success, global_id 22981 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: _renew_subs 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f124c141220 con 0x7f124c0a46a0 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12340b0040 con 0x7f124c0a46a0 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1242ffd640 10 monclient: dump: 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.168 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.169 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1248d99640 4 set_mon_vals no callback set 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 client.22981.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: _renew_subs 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f124c0a6b90 con 0x7f124c0a46a0 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.143+0000 7f1250b27640 1 librados: init done 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1250b27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1250b27640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13d510 0x7f124c13d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1250b27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f124c139640 con 0x7f124c13d510 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1234004040 con 0x7f124c0a46a0 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f122c03df60 0x7f122c040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13d510 0x7f124c13d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient: get_auth_request con 0x7f124c13d510 auth_method 0 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(84..84 src has 1..84) ==== 8725+0+0 (secure 0 0 0) 0x7f12340e96e0 con 0x7f124c0a46a0 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 client.22981.objecter ms_dispatch 0x7f124c000ba0 osd_map(84..84 src has 1..84) 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 3 client.22981.objecter handle_osd_map got epochs [84,84] > 0 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 3 client.22981.objecter handle_osd_map decoding full epoch 84 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 20 client.22981.objecter dump_active .. 0 homeless 2024-09-11T02:52:10.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f124ad9d640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f122c03df60 0x7f122c040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f124ad9d640 10 monclient: get_auth_request con 0x7f122c03df60 auth_method 0 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 10 monclient(hunting): handle_auth_done global_id 19130 payload 824 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 1 -- 172.21.15.98:0/2095041129 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f124c0a4a80 con 0x7f124c13d510 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 client.22981.objecter ms_handle_connect 0x7f124c13d510 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 client.22981.objecter resend_mon_ops 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1249d9b640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13d510 0x7f124c13d8f0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f124c13d420 tx=0x7f124400ecd0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 client.22981.objecter ms_handle_connect 0x7f122c03df60 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f124ad9d640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f122c03df60 0x7f122c040420 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f123c007d40 tx=0x7f123c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12440050d0 con 0x7f124c13d510 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 monclient: dump: 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.173 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1244018070 con 0x7f124c13d510 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13d510 msgr2=0x7f124c13d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13d510 0x7f124c13d8f0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f124c13d420 tx=0x7f124400ecd0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13d510 msgr2=0x7f124c13d8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:10.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.147+0000 7f1242ffd640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13d510 0x7f124c13d8f0 secure :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f124c13d420 tx=0x7f124400ecd0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13de30 0x7f124c072620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f124c06ca10 con 0x7f124c13de30 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13de30 0x7f124c072620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient: get_auth_request con 0x7f124c13de30 auth_method 0 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 10 monclient(hunting): handle_auth_done global_id 19139 payload 824 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 1 -- 172.21.15.98:0/2095041129 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f124c13e930 con 0x7f124c13de30 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1249d9b640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13de30 0x7f124c072620 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f1244017000 tx=0x7f1244001260 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 10 client.22981.objecter ms_handle_connect 0x7f124c13de30 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 10 client.22981.objecter resend_mon_ops 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f124401de30 con 0x7f124c13de30 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 10 monclient: dump: 2024-09-11T02:52:10.177 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.178 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f12440050d0 con 0x7f124c13de30 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13de30 msgr2=0x7f124c072620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13de30 0x7f124c072620 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f1244017000 tx=0x7f1244001260 comp rx=0 tx=0).stop 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13de30 msgr2=0x7f124c072620 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1242ffd640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f124c13de30 0x7f124c072620 secure :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f1244017000 tx=0x7f1244001260 comp rx=0 tx=0).stop 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:10.179 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 145904, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:10.180 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:10.181 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:10.182 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:10.183 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:10.184 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.185 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.187 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.188 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:10.190 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.195 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:10.198 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.199 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.200 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:10.201 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.202 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:10.203 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.203 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.203 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:10.203 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 10 librados: watch_flush enter 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 10 librados: watch_flush exit 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 20 client.22981.objecter shutdown clearing up homeless session... 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 10 client.22981.objecter successfully canceled tick 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f122c03df60 msgr2=0x7f122c040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f122c03df60 0x7f122c040420 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7f123c007d40 tx=0x7f123c008040 comp rx=0 tx=0).stop 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 10 monclient: shutdown 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f124c0a46a0 msgr2=0x7f124c1d19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 --2- 172.21.15.98:0/2095041129 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f124c0a46a0 0x7f124c1d19f0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f123401aaf0 tx=0x7f12340191e0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f124ad9d640 1 -- 172.21.15.98:0/2095041129 reap_dead start 2024-09-11T02:52:10.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f124ad9d640 1 -- 172.21.15.98:0/2095041129 reap_dead start 2024-09-11T02:52:10.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 shutdown_connections 2024-09-11T02:52:10.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 >> 172.21.15.98:0/2095041129 conn(0x7f124c06ed00 msgr2=0x7f124c073960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:10.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 shutdown_connections 2024-09-11T02:52:10.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 -- 172.21.15.98:0/2095041129 wait complete. 2024-09-11T02:52:10.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.155+0000 7f1250b27640 1 librados: shutdown 2024-09-11T02:52:10.206 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:10.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:52:10.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.327+0000 7f0d483d0640 10 monclient: get_monmap_and_config 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.327+0000 7f0d483d0640 10 monclient: build_initial_monmap 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.327+0000 7f0d483d0640 10 monclient: monmap: 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:10.332420+0000 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:10.332420+0000 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:10.331 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:10.332 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 Processor -- start 2024-09-11T02:52:10.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 -- start start 2024-09-11T02:52:10.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient: init 2024-09-11T02:52:10.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient: _add_conns ranks=[3,2,7,9,6,8,0,5,1,10,4] 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d400a3de0 0x7f0d400a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): picked mon.noname-g con 0x7f0d400a3de0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d400a41c0 0x7f0d400a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): picked mon.noname-b con 0x7f0d400a41c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d4009b240 0x7f0d4009b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): picked mon.noname-i con 0x7f0d4009b240 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0d400a0170 con 0x7f0d400a41c0 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0d40068fa0 con 0x7f0d400a3de0 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0d40068b40 con 0x7f0d4009b240 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d483d0640 10 monclient(hunting): authenticate will time out at 1084.213509s 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d400a41c0 0x7f0d400a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d400a41c0 0x7f0d400a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56572/0 (socket says 172.21.15.98:56572) 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 -- 172.21.15.98:0/1700083660 learned_addr learned my addr 172.21.15.98:0/1700083660 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): get_auth_request con 0x7f0d400a41c0 auth_method 0 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient(hunting): handle_auth_done global_id 23024 payload 274 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 -- 172.21.15.98:0/1700083660 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d4009b240 msgr2=0x7f0d4009b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 --2- 172.21.15.98:0/1700083660 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d4009b240 0x7f0d4009b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 -- 172.21.15.98:0/1700083660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d400a3de0 msgr2=0x7f0d400a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 --2- 172.21.15.98:0/1700083660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d400a3de0 0x7f0d400a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient: _finish_hunting 0 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 monclient: found mon.noname-b 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:10.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.331+0000 7f0d4596b640 1 -- 172.21.15.98:0/1700083660 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d4009eaf0 con 0x7f0d400a41c0 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 --2- 172.21.15.98:0/1700083660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d400a41c0 0x7f0d400a45c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0d340098d0 tx=0x7f0d3401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d9a47aa01472258 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 1 -- 172.21.15.98:0/1700083660 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d340b0070 con 0x7f0d400a41c0 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: dump: 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.342 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.343 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: _finish_auth 0 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: _check_auth_tickets 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 1 -- 172.21.15.98:0/1700083660 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d340aa810 con 0x7f0d400a41c0 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 5 monclient: authenticate success, global_id 23024 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 1 -- 172.21.15.98:0/1700083660 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d340a9300 con 0x7f0d400a41c0 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 4 set_mon_vals no callback set 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:10.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4516a640 10 monclient: dump: 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.345 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient: shutdown 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- 172.21.15.98:0/1700083660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d400a41c0 msgr2=0x7f0d400a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 --2- 172.21.15.98:0/1700083660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d400a41c0 0x7f0d400a45c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f0d340098d0 tx=0x7f0d3401c440 comp rx=0 tx=0).stop 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- 172.21.15.98:0/1700083660 shutdown_connections 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 --2- 172.21.15.98:0/1700083660 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d400a41c0 0x7f0d400a45c0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 --2- 172.21.15.98:0/1700083660 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d4009b240 0x7f0d4009b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 --2- 172.21.15.98:0/1700083660 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d400a3de0 0x7f0d400a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- 172.21.15.98:0/1700083660 >> 172.21.15.98:0/1700083660 conn(0x7f0d400a4ed0 msgr2=0x7f0d400a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:10.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- 172.21.15.98:0/1700083660 shutdown_connections 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- 172.21.15.98:0/1700083660 wait complete. 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient: build_initial_monmap 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient: monmap: 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:10.347 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 librados: starting msgr at 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 librados: starting objecter 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 Processor -- start 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- start start 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 librados: setting wanted keys 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 librados: calling monclient init 2024-09-11T02:52:10.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient: init 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient: _add_conns ranks=[3,8,1,4,2,0,6,7,5] 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d4009b240 0x7f0d401d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): picked mon.noname-g con 0x7f0d4009b240 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d400a3de0 0x7f0d40137950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): picked mon.noname-e con 0x7f0d400a3de0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d400a41c0 0x7f0d4012e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): picked mon.noname-f con 0x7f0d400a41c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0d401413a0 con 0x7f0d400a41c0 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0d40141220 con 0x7f0d4009b240 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0d40141520 con 0x7f0d400a3de0 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d483d0640 10 monclient(hunting): authenticate will time out at 1084.219004s 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d400a3de0 0x7f0d40137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d400a3de0 0x7f0d40137950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56848/0 (socket says 172.21.15.98:56848) 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 -- 172.21.15.98:0/727420342 learned_addr learned my addr 172.21.15.98:0/727420342 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): get_auth_request con 0x7f0d400a3de0 auth_method 0 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient(hunting): handle_auth_done global_id 23417 payload 824 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d4009b240 msgr2=0x7f0d401d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d4009b240 0x7f0d401d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d400a41c0 msgr2=0x7f0d4012e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d400a41c0 0x7f0d4012e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient: _finish_hunting 0 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 monclient: found mon.noname-e 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 -- 172.21.15.98:0/727420342 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d40131910 con 0x7f0d400a3de0 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d4596b640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d400a3de0 0x7f0d40137950 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f0d3401c270 tx=0x7f0d340191e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d3effd640 10 client.?.objecter ms_handle_connect 0x7f0d400a3de0 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.335+0000 7f0d3effd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d340b9070 con 0x7f0d400a3de0 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: dump: 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: _finish_auth 0 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 5 monclient: authenticate success, global_id 23417 2024-09-11T02:52:10.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d340044a0 con 0x7f0d400a3de0 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 10 monclient: _renew_subs 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d40131bf0 con 0x7f0d400a3de0 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d340b0040 con 0x7f0d400a3de0 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 10 client.23417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d44969640 4 set_mon_vals no callback set 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d3effd640 10 monclient: dump: 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.354 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.355 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.356 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 10 monclient: _renew_subs 2024-09-11T02:52:10.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:10.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d401309b0 con 0x7f0d400a3de0 2024-09-11T02:52:10.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 1 librados: init done 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40137e90 0x7f0d4012ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0d4006ca10 con 0x7f0d40137e90 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40137e90 0x7f0d4012ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient: get_auth_request con 0x7f0d40137e90 auth_method 0 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 10 monclient(hunting): handle_auth_done global_id 18844 payload 824 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.339+0000 7f0d4616c640 1 -- 172.21.15.98:0/727420342 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d400a4670 con 0x7f0d40137e90 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d4616c640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40137e90 0x7f0d4012ebf0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000bd50 tx=0x7f0d30009ed0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 client.23417.objecter ms_handle_connect 0x7f0d40137e90 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 client.23417.objecter resend_mon_ops 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d30019070 con 0x7f0d40137e90 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 monclient: dump: 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.358 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0d340b3070 con 0x7f0d400a3de0 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d2c046a00 0x7f0d2c048ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(85..85 src has 1..85) ==== 8843+0+0 (secure 0 0 0) 0x7f0d340e90e0 con 0x7f0d400a3de0 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 client.23417.objecter ms_dispatch 0x7f0d40000ba0 osd_map(85..85 src has 1..85) 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 3 client.23417.objecter handle_osd_map got epochs [85,85] > 0 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 3 client.23417.objecter handle_osd_map decoding full epoch 85 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 20 client.23417.objecter dump_active .. 0 homeless 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d3000cbf0 con 0x7f0d40137e90 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40137e90 msgr2=0x7f0d4012ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40137e90 0x7f0d4012ebf0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000bd50 tx=0x7f0d30009ed0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40137e90 msgr2=0x7f0d4012ebf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d3effd640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40137e90 0x7f0d4012ebf0 secure :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000bd50 tx=0x7f0d30009ed0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d4696d640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d2c046a00 0x7f0d2c048ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.343+0000 7f0d4696d640 10 monclient: get_auth_request con 0x7f0d2c046a00 auth_method 0 2024-09-11T02:52:10.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.347+0000 7f0d4696d640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d2c046a00 0x7f0d2c048ec0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f0d4006d630 tx=0x7f0d38006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.347+0000 7f0d3effd640 10 client.23417.objecter ms_handle_connect 0x7f0d2c046a00 2024-09-11T02:52:10.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:10.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40131fd0 0x7f0d4012f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0d4006e870 con 0x7f0d40131fd0 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40131fd0 0x7f0d4012f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient: get_auth_request con 0x7f0d40131fd0 auth_method 0 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 10 monclient(hunting): handle_auth_done global_id 18853 payload 824 2024-09-11T02:52:10.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 1 -- 172.21.15.98:0/727420342 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d400a0570 con 0x7f0d40131fd0 2024-09-11T02:52:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4616c640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40131fd0 0x7f0d4012f130 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000bd50 tx=0x7f0d30004b60 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 10 client.23417.objecter ms_handle_connect 0x7f0d40131fd0 2024-09-11T02:52:10.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 10 client.23417.objecter resend_mon_ops 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d30014e40 con 0x7f0d40131fd0 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 10 monclient: dump: 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.364 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f0d300149d0 con 0x7f0d40131fd0 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:10.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40131fd0 msgr2=0x7f0d4012f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40131fd0 0x7f0d4012f130 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000bd50 tx=0x7f0d30004b60 comp rx=0 tx=0).stop 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40131fd0 msgr2=0x7f0d4012f130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d3effd640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d40131fd0 0x7f0d4012f130 secure :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f0d3000bd50 tx=0x7f0d30004b60 comp rx=0 tx=0).stop 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:10.366 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146128, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:10.367 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:10.368 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:10.369 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:10.370 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:10.371 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.372 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:10.375 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:10.376 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.377 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:10.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.384 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:10.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:10.389 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-09-11T02:52:10.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:10.390 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:10.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:10.390 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:10.390 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 10 librados: watch_flush enter 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 10 librados: watch_flush exit 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 20 client.23417.objecter shutdown clearing up homeless session... 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 10 client.23417.objecter successfully canceled tick 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d2c046a00 msgr2=0x7f0d2c048ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d2c046a00 0x7f0d2c048ec0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f0d4006d630 tx=0x7f0d38006ca0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 10 monclient: shutdown 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d400a3de0 msgr2=0x7f0d40137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d483d0640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d400a3de0 0x7f0d40137950 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f0d3401c270 tx=0x7f0d340191e0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.359+0000 7f0d4696d640 1 -- 172.21.15.98:0/727420342 reap_dead start 2024-09-11T02:52:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.363+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 shutdown_connections 2024-09-11T02:52:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.363+0000 7f0d483d0640 1 --2- 172.21.15.98:0/727420342 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d400a3de0 0x7f0d40137950 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.363+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 >> 172.21.15.98:0/727420342 conn(0x7f0d400a4ed0 msgr2=0x7f0d400a62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.363+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 shutdown_connections 2024-09-11T02:52:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.363+0000 7f0d483d0640 1 -- 172.21.15.98:0/727420342 wait complete. 2024-09-11T02:52:10.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.363+0000 7f0d483d0640 1 librados: shutdown 2024-09-11T02:52:10.393 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:10.393 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T02:52:10.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.503+0000 7fa9a5ef9640 10 monclient: get_monmap_and_config 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.503+0000 7fa9a5ef9640 10 monclient: build_initial_monmap 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.503+0000 7fa9a5ef9640 10 monclient: monmap: 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:10.508490+0000 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:10.508490+0000 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:10.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:10.508 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.503+0000 7fa9a5ef9640 1 Processor -- start 2024-09-11T02:52:10.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.503+0000 7fa9a5ef9640 1 -- start start 2024-09-11T02:52:10.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.503+0000 7fa9a5ef9640 10 monclient: init 2024-09-11T02:52:10.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:10.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient: _add_conns ranks=[0,6,9,10,4,5,2,8,3,7,1] 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9a00a37a0 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a00a3b80 0x7fa9a00a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient(hunting): picked mon.noname-d con 0x7fa9a00a3b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9a013c9c0 0x7fa9a013edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient(hunting): picked mon.noname-a con 0x7fa9a013c9c0 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9a009d440 con 0x7fa9a00a37a0 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa9a00a1b90 con 0x7fa9a013c9c0 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa9a009d720 con 0x7fa9a00a3b80 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient(hunting): authenticate will time out at 1084.388588s 2024-09-11T02:52:10.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa9a00a37a0 0x7fa9a009e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37918/0 (socket says 172.21.15.98:37918) 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99f7fe640 1 -- 172.21.15.98:0/709316468 learned_addr learned my addr 172.21.15.98:0/709316468 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa9a013c9c0 0x7fa9a013edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): get_auth_request con 0x7fa9a013c9c0 auth_method 0 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 1 --2- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a00a3b80 0x7fa9a00a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 10 monclient(hunting): get_auth_request con 0x7fa9a00a3b80 auth_method 0 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient(hunting): handle_auth_done global_id 14202 payload 274 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 1 -- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a00a3b80 msgr2=0x7fa9a00a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 1 --2- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a00a3b80 0x7fa9a00a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 1 -- 172.21.15.98:0/709316468 >> v1:172.21.15.98:6789/0 conn(0x7fa9a00a37a0 legacy=0x7fa9a009e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 1 monclient: found mon.noname-a 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 1 -- 172.21.15.98:0/709316468 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3396848965 0 0) 0x7fa9a009d440 con 0x7fa9a00a37a0 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:10.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:10.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 1 -- 172.21.15.98:0/709316468 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a009da00 con 0x7fa9a013c9c0 2024-09-11T02:52:10.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:52:10.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99ffff640 1 --2- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a013edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa998009d90 tx=0x7fa99801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed6953718639e5d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 1 -- 172.21.15.98:0/709316468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9980b2070 con 0x7fa9a013c9c0 2024-09-11T02:52:10.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:10.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: dump: 2024-09-11T02:52:10.519 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.520 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 1 -- 172.21.15.98:0/709316468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9980a5b00 con 0x7fa9a013c9c0 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 5 monclient: authenticate success, global_id 14202 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:10.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 4 set_mon_vals no callback set 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 1 -- 172.21.15.98:0/709316468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9980a5cf0 con 0x7fa9a013c9c0 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa99e7fc640 10 monclient: dump: 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.522 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 10 monclient: shutdown 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 msgr2=0x7fa9a013edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a013edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa998009d90 tx=0x7fa99801c8c0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/709316468 shutdown_connections 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a013edb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/709316468 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a00a3b80 0x7fa9a00a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/709316468 >> 172.21.15.98:0/709316468 conn(0x7fa9a00a4890 msgr2=0x7fa9a00a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/709316468 shutdown_connections 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.507+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/709316468 wait complete. 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: build_initial_monmap 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: monmap: 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:10.524 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 librados: starting msgr at 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 librados: starting objecter 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 Processor -- start 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 -- start start 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 librados: setting wanted keys 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 librados: calling monclient init 2024-09-11T02:52:10.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: init 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _add_conns ranks=[3,1,0,7,2,8,5,6,4] 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9a00a37a0 0x7fa9a01cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): picked mon.noname-g con 0x7fa9a00a37a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9a00a3b80 0x7fa9a01cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): picked mon.noname-f con 0x7fa9a00a3b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a01cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): picked mon.noname-a con 0x7fa9a013c9c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a0141ea0 con 0x7fa9a013c9c0 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a0141d20 con 0x7fa9a00a3b80 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a0142020 con 0x7fa9a00a37a0 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient(hunting): authenticate will time out at 1084.393640s 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a01cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a01cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59522/0 (socket says 172.21.15.98:59522) 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 -- 172.21.15.98:0/3309132880 learned_addr learned my addr 172.21.15.98:0/3309132880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): get_auth_request con 0x7fa9a013c9c0 auth_method 0 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99effd640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9a00a3b80 0x7fa9a01cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99effd640 10 monclient(hunting): get_auth_request con 0x7fa9a00a3b80 auth_method 0 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99f7fe640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9a00a37a0 0x7fa9a01cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient(hunting): handle_auth_done global_id 14220 payload 824 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 -- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9a00a37a0 msgr2=0x7fa9a01cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9a00a37a0 0x7fa9a01cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 -- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9a00a3b80 msgr2=0x7fa9a01cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9a00a3b80 0x7fa9a01cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 monclient: found mon.noname-a 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 -- 172.21.15.98:0/3309132880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a0124190 con 0x7fa9a013c9c0 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a01cfa90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b0040 tx=0x7fa998019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 client.?.objecter ms_handle_connect 0x7fa9a013c9c0 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 1 -- 172.21.15.98:0/3309132880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9980b2050 con 0x7fa9a013c9c0 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: dump: 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.529 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: _finish_auth 0 2024-09-11T02:52:10.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 1 -- 172.21.15.98:0/3309132880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9980be070 con 0x7fa9a013c9c0 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 5 monclient: authenticate success, global_id 14220 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _renew_subs 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a0125000 con 0x7fa9a013c9c0 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 1 -- 172.21.15.98:0/3309132880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9980a6e00 con 0x7fa9a013c9c0 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 monclient: dump: 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:10.531 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:10.532 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99dffb640 4 set_mon_vals no callback set 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 client.14220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _renew_subs 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a00a6ca0 con 0x7fa9a013c9c0 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 librados: init done 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 1 -- 172.21.15.98:0/3309132880 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa998003bb0 con 0x7fa9a013c9c0 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa98c03df40 0x7fa98c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:10.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a00a3f80 con 0x7fa9a013c9c0 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 1 -- 172.21.15.98:0/3309132880 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8843+0+0 (secure 0 0 0) 0x7fa9980eb260 con 0x7fa9a013c9c0 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 10 client.14220.objecter ms_dispatch 0x7fa9a0000ba0 osd_map(85..85 src has 1..85) 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 3 client.14220.objecter handle_osd_map got epochs [85,85] > 0 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 3 client.14220.objecter handle_osd_map decoding full epoch 85 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa987fff640 20 client.14220.objecter dump_active .. 0 homeless 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99f7fe640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa98c03df40 0x7fa98c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.511+0000 7fa99f7fe640 10 monclient: get_auth_request con 0x7fa98c03df40 auth_method 0 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.515+0000 7fa987fff640 10 client.14220.objecter ms_handle_connect 0x7fa98c03df40 2024-09-11T02:52:10.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.515+0000 7fa99f7fe640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa98c03df40 0x7fa98c040400 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7fa994000f70 tx=0x7fa994009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:10.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.519+0000 7fa987fff640 1 -- 172.21.15.98:0/3309132880 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fa998016540 con 0x7fa9a013c9c0 2024-09-11T02:52:10.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.519+0000 7fa987fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:10.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.519+0000 7fa987fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:10.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.751+0000 7fa9a5ef9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T02:52:10.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.751+0000 7fa9a5ef9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T02:52:10.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.751+0000 7fa9a5ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:10.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.751+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa9a0122ca0 con 0x7fa9a013c9c0 2024-09-11T02:52:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.755+0000 7fa987fff640 1 -- 172.21.15.98:0/3309132880 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa9980e8c90 con 0x7fa9a013c9c0 2024-09-11T02:52:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.755+0000 7fa987fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T02:52:10.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.755+0000 7fa987fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:10.760 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.760 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.760 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=1485586730,kv=3991532845,logm=1242856919}) 2024-09-11T02:52:10.760 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.760 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=9,kv=10,logm=70} crc {auth=3721576184,config=692619927,health=3041987372,kv=3991532845,logm=1959042311}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=8,config=2,health=12,kv=9,logm=69} crc {auth=2669926208,config=692619927,health=3643023599,kv=893004536,logm=3424362909}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=8,config=2,health=10,kv=10,logm=70} crc {auth=1376553207,config=692619927,health=1935562574,kv=3991532845,logm=1028193585}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=2671165181}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.761 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=9,config=2,health=11,kv=9,logm=69} crc {auth=1801079511,config=692619927,health=4192131285,kv=1044946065,logm=2902251688}) 2024-09-11T02:52:10.762 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.762 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.762 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=3721576184,config=692619927,health=3643023599,kv=1009120483,logm=1948774477}) 2024-09-11T02:52:10.762 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.762 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=3721576184,config=692619927,health=3643023599,kv=3991532845,logm=679853548}) 2024-09-11T02:52:10.762 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.756+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=8,config=2,health=11,kv=9,logm=70} crc {auth=2669926208,config=692619927,health=4183472653,kv=1551318347,logm=3034727254}) 2024-09-11T02:52:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 10 librados: watch_flush enter 2024-09-11T02:52:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 10 librados: watch_flush exit 2024-09-11T02:52:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 20 client.14220.objecter shutdown clearing up homeless session... 2024-09-11T02:52:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 10 client.14220.objecter successfully canceled tick 2024-09-11T02:52:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa98c03df40 msgr2=0x7fa98c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa98c03df40 0x7fa98c040400 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7fa994000f70 tx=0x7fa994009040 comp rx=0 tx=0).stop 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 10 monclient: shutdown 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 msgr2=0x7fa9a01cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a01cfa90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b0040 tx=0x7fa998019660 comp rx=0 tx=0).stop 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 shutdown_connections 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa98c03df40 0x7fa98c040400 unknown :-1 s=CLOSED pgs=3366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a013c9c0 0x7fa9a01cfa90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9a00a3b80 0x7fa9a01cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 --2- 172.21.15.98:0/3309132880 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa9a00a37a0 0x7fa9a01cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 >> 172.21.15.98:0/3309132880 conn(0x7fa9a00a4890 msgr2=0x7fa9a00a7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 shutdown_connections 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 -- 172.21.15.98:0/3309132880 wait complete. 2024-09-11T02:52:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:10.759+0000 7fa9a5ef9640 1 librados: shutdown 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2047112089}) 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4167496529}) 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412921683}) 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.767 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2826721350}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2618113264}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1376320449}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3754327864}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209235609}) 2024-09-11T02:52:10.768 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.760+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3983952204}) 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393073035}) 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3978740590}) 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2003371115}) 2024-09-11T02:52:10.771 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.764+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2936658280}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2946098027}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2092543176}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1642493962}) 2024-09-11T02:52:10.772 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.773 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405804813}) 2024-09-11T02:52:10.773 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=369818520}) 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439797547}) 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.768+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3855554951}) 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3168120933}) 2024-09-11T02:52:10.775 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=56354490}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2874249700}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2989594342}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2926204996}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177071608}) 2024-09-11T02:52:10.776 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.772+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2206752709}) 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368375840}) 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2182988125}) 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1608100519}) 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.779 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2335206694}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2684086783}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1153000812}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3545940150}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2437682422}) 2024-09-11T02:52:10.780 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.776+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3667676478}) 2024-09-11T02:52:10.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T02:52:10.785 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2557121856}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426081105}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2048190189}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1026440048}) 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.786 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3246333853}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=653088668}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3286156453}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058789475}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2965210458}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018710476}) 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.787 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1889769538}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=492459490}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.780+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=122471111}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=179558209}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.788 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3771624823}) 2024-09-11T02:52:10.789 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.789 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.789 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2470844414}) 2024-09-11T02:52:10.789 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.789 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623401223}) 2024-09-11T02:52:10.789 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3111390455}) 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1844094930}) 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=255594143}) 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.791 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337923499}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3753223269}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4231131977}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1193982908}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.792 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2485541216}) 2024-09-11T02:52:10.793 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.793 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464937485}) 2024-09-11T02:52:10.793 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.784+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3739509273}) 2024-09-11T02:52:10.795 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.795 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.795 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281468457}) 2024-09-11T02:52:10.795 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.795 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31354458}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2149624360}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3497113165}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4187397724}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.788+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2533194228}) 2024-09-11T02:52:10.796 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.797 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.797 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4247117097}) 2024-09-11T02:52:10.797 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.797 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148746532}) 2024-09-11T02:52:10.797 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=107205480}) 2024-09-11T02:52:10.798 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.798 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717500427}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804379908}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1351335591}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=932196232}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.799 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3254753231}) 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=897984226}) 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.792+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1032971685}) 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44681526}) 2024-09-11T02:52:10.800 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=189561383}) 2024-09-11T02:52:10.801 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.801 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.801 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740045386}) 2024-09-11T02:52:10.801 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.801 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.801 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124470879}) 2024-09-11T02:52:10.801 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=599121939}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1631441094}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3193665043}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2040300438}) 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.802 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.803 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1968115089}) 2024-09-11T02:52:10.803 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.803 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758725327}) 2024-09-11T02:52:10.803 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.796+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1918961954}) 2024-09-11T02:52:10.804 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.804 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.804 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1841606880}) 2024-09-11T02:52:10.804 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.804 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956940075}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1779456044}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=139365673}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3201053699}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.805 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1635652254}) 2024-09-11T02:52:10.806 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.806 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.806 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2085234111}) 2024-09-11T02:52:10.806 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.806 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590167640}) 2024-09-11T02:52:10.806 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.800+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1905217159}) 2024-09-11T02:52:10.808 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.808 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.808 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339751098}) 2024-09-11T02:52:10.808 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266457268}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4279000824}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=861509064}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=500018596}) 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.809 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.810 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=626788049}) 2024-09-11T02:52:10.810 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.810 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.810 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3784440161}) 2024-09-11T02:52:10.810 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.810 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630329352}) 2024-09-11T02:52:10.810 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.804+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1495024404}) 2024-09-11T02:52:10.812 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157326570}) 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=649264817}) 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=519653762}) 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.813 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=944325743}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=546130815}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=22086589}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2303344282}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527020945}) 2024-09-11T02:52:10.814 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.808+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3217903036}) 2024-09-11T02:52:10.815 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.815 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.815 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076690639}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1502665692}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2547058153}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=837692269}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.816 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3772599662}) 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2032393534}) 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3179440066}) 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837909464}) 2024-09-11T02:52:10.817 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.812+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3705501335}) 2024-09-11T02:52:10.819 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.819 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.819 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18,osdmap=9} crc {logm=2130085497,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2215576653,mon_config_key=1863298652,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=2187246926,osdmap=361223339}) 2024-09-11T02:52:10.819 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18,osdmap=6} crc {logm=1931565110,mdsmap=3491474967,mgr=446734576,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=634900297,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=404800797,osdmap=895271499}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=18,osdmap=8} crc {logm=3221692739,mdsmap=92030302,mgr=534862250,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=4081230052,mon_config_key=2106052021,monmap=416611756,nvmeofgw=598337686,osd_metadata=1489146596,osd_pg_creating=2632004962,osd_snap=2443446156,osdmap=338421537}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18,osdmap=3} crc {logm=3353291761,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=634900297,mon_config_key=2106052021,monmap=416611756,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=3891675909,osdmap=41283803}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=10} crc {logm=1804050682,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=634900297,mon_config_key=3492741104,monmap=2495409540,nvmeofgw=1596038039,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=2207404409,osdmap=2954916960}) 2024-09-11T02:52:10.820 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18,osdmap=6} crc {logm=2991990318,mdsmap=3140514009,mgr=534862250,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=634900297,mon_config_key=2106052021,monmap=3140514009,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=973372696,osdmap=3931948476}) 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=30,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=19,osdmap=8} crc {logm=2477497394,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2215576653,mon_config_key=992323854,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1459920058,osd_pg_creating=2632004962,osd_snap=3734949995,osdmap=338421537}) 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=30,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=1619340690,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1491352780,mon_config_key=2106052021,monmap=1094721948,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=1593228097,osdmap=905020243}) 2024-09-11T02:52:10.821 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.816+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=7} crc {logm=3871034346,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=4081230052,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2632004962,osd_snap=2384341044,osdmap=2911053458}) 2024-09-11T02:52:10.825 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.825 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.825 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=527591768}) 2024-09-11T02:52:10.825 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.825 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.825 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3652229511}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3561373402}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1875310957}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=705994267}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3835590630}) 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.826 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.827 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=793637205}) 2024-09-11T02:52:10.827 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.827 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1175248354}) 2024-09-11T02:52:10.827 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.824+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=901780071}) 2024-09-11T02:52:10.839 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.839 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.839 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=1968097597}) 2024-09-11T02:52:10.839 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.839 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=26} crc {osdmap=4176128631}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=27} crc {osdmap=58684273}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=28} crc {osdmap=3910267521}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=25} crc {osdmap=2641990403}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.840 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=27} crc {osdmap=329429942}) 2024-09-11T02:52:10.841 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.841 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.841 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=28} crc {osdmap=3910267521}) 2024-09-11T02:52:10.841 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:52:10.841 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=27} crc {osdmap=3846086652}) 2024-09-11T02:52:10.841 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:52:10.836+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=27} crc {osdmap=1111680006}) 2024-09-11T02:52:11.783 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:52:11.784 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:52:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient: get_monmap_and_config 2024-09-11T02:52:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient: build_initial_monmap 2024-09-11T02:52:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient: monmap: 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:11.914996+0000 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:11.914996+0000 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:11.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 1 Processor -- start 2024-09-11T02:52:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 1 -- start start 2024-09-11T02:52:11.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient: init 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient: _add_conns ranks=[4,9,0,10,1,7,5,2,8,3,6] 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f551006f650 0x7f5510072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient(hunting): picked mon.noname-c con 0x7f551006f650 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f55100a8380 0x7f551006fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient(hunting): picked mon.noname-a con 0x7f55100a8380 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f551006ff70 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5510071a80 con 0x7f551006ff70 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f55100a8800 con 0x7f55100a8380 2024-09-11T02:52:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5510071d60 con 0x7f551006f650 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f551770e640 10 monclient(hunting): authenticate will time out at 1085.795252s 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f551006f650 0x7f5510072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5515cab640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f551006ff70 0x7f55101397c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37924/0 (socket says 172.21.15.98:37924) 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5515cab640 1 -- 172.21.15.98:0/2525073193 learned_addr learned my addr 172.21.15.98:0/2525073193 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): get_auth_request con 0x7f551006f650 auth_method 0 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 1 --2- 172.21.15.98:0/2525073193 >> v2:172.21.15.98:3300/0 conn(0x7f55100a8380 0x7f551006fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 10 monclient(hunting): get_auth_request con 0x7f55100a8380 auth_method 0 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:11.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f5514ca9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient(hunting): handle_auth_done global_id 22990 payload 274 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 1 -- 172.21.15.98:0/2525073193 >> v2:172.21.15.98:3300/0 conn(0x7f55100a8380 msgr2=0x7f551006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 1 --2- 172.21.15.98:0/2525073193 >> v2:172.21.15.98:3300/0 conn(0x7f55100a8380 0x7f551006fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 1 -- 172.21.15.98:0/2525073193 >> v1:172.21.15.98:6789/0 conn(0x7f551006ff70 legacy=0x7f55101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient: _finish_hunting 0 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 1 monclient: found mon.noname-c 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:11.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.911+0000 7f55154aa640 1 -- 172.21.15.98:0/2525073193 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55100714c0 con 0x7f551006f650 2024-09-11T02:52:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f55154aa640 1 --2- 172.21.15.98:0/2525073193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f551006f650 0x7f5510072640 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f550c0097c0 tx=0x7f550c01cd90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=c21f6407ad461847 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 1 -- 172.21.15.98:0/2525073193 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f550c0b0070 con 0x7f551006f650 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: dump: 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: _finish_auth 0 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 1 -- 172.21.15.98:0/2525073193 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550c0aa810 con 0x7f551006f650 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 1 -- 172.21.15.98:0/2525073193 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f550c0a9300 con 0x7f551006f650 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 5 monclient: authenticate success, global_id 22990 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 4 set_mon_vals no callback set 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f5507fff640 10 monclient: dump: 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 10 monclient: shutdown 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 -- 172.21.15.98:0/2525073193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f551006f650 msgr2=0x7f5510072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 --2- 172.21.15.98:0/2525073193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f551006f650 0x7f5510072640 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f550c0097c0 tx=0x7f550c01cd90 comp rx=0 tx=0).stop 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 -- 172.21.15.98:0/2525073193 shutdown_connections 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 --2- 172.21.15.98:0/2525073193 >> v2:172.21.15.98:3300/0 conn(0x7f55100a8380 0x7f551006fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 --2- 172.21.15.98:0/2525073193 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f551006f650 0x7f5510072640 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 -- 172.21.15.98:0/2525073193 >> 172.21.15.98:0/2525073193 conn(0x7f55100ac8e0 msgr2=0x7f55100a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 -- 172.21.15.98:0/2525073193 shutdown_connections 2024-09-11T02:52:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.915+0000 7f551770e640 1 -- 172.21.15.98:0/2525073193 wait complete. 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: build_initial_monmap 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: monmap: 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 librados: starting msgr at 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 librados: starting objecter 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 Processor -- start 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 -- start start 2024-09-11T02:52:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 librados: setting wanted keys 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 librados: calling monclient init 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: init 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _add_conns ranks=[5,8,3,1,6,0,4,2,7] 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f551006f650 0x7f5510138900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): picked mon.noname-h con 0x7f551006f650 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f551006ff70 0x7f5510138e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): picked mon.noname-e con 0x7f551006ff70 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55100a8380 0x7f551013e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): picked mon.noname-g con 0x7f55100a8380 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5510143fd0 con 0x7f55100a8380 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5510143e50 con 0x7f551006f650 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5510144150 con 0x7f551006ff70 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient(hunting): authenticate will time out at 1085.800924s 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f551006ff70 0x7f5510138e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f551006ff70 0x7f5510138e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56850/0 (socket says 172.21.15.98:56850) 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 -- 172.21.15.98:0/587485098 learned_addr learned my addr 172.21.15.98:0/587485098 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): get_auth_request con 0x7f551006ff70 auth_method 0 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f55154aa640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f551006f650 0x7f5510138900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient(hunting): handle_auth_done global_id 23435 payload 824 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 -- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f551006f650 msgr2=0x7f5510138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5515cab640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55100a8380 0x7f551013e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f551006f650 0x7f5510138900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 -- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55100a8380 msgr2=0x7f551013e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55100a8380 0x7f551013e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient: _finish_hunting 0 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 monclient: found mon.noname-e 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 -- 172.21.15.98:0/587485098 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55100a56a0 con 0x7f551006ff70 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f55154aa640 10 monclient: get_auth_request con 0x7f551006f650 auth_method 0 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f55154aa640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f551006f650 0x7f5510138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5514ca9640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f551006ff70 0x7f5510138e40 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f550000e840 tx=0x7f550000ec40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 client.?.objecter ms_handle_connect 0x7f551006ff70 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 1 -- 172.21.15.98:0/587485098 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5500017070 con 0x7f551006ff70 2024-09-11T02:52:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: dump: 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: _finish_auth 0 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 1 -- 172.21.15.98:0/587485098 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550000ee90 con 0x7f551006ff70 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 5 monclient: authenticate success, global_id 23435 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _renew_subs 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55100a5150 con 0x7f551006ff70 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 1 -- 172.21.15.98:0/587485098 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f550000cdb0 con 0x7f551006ff70 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 monclient: dump: 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 client.23435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _renew_subs 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f55077fe640 4 set_mon_vals no callback set 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f55101231e0 con 0x7f551006ff70 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 librados: init done 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5510072640 con 0x7f551006ff70 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 1 -- 172.21.15.98:0/587485098 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5500024060 con 0x7f551006ff70 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f54f403df40 0x7f54f4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 1 -- 172.21.15.98:0/587485098 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(86..86 src has 1..86) ==== 8541+0+0 (secure 0 0 0) 0x7f550000f040 con 0x7f551006ff70 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 10 client.23435.objecter ms_dispatch 0x7f5510000ba0 osd_map(86..86 src has 1..86) 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 3 client.23435.objecter handle_osd_map got epochs [86,86] > 0 2024-09-11T02:52:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 3 client.23435.objecter handle_osd_map decoding full epoch 86 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f5505ffb640 20 client.23435.objecter dump_active .. 0 homeless 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f55154aa640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f54f403df40 0x7f54f4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.919+0000 7f55154aa640 10 monclient: get_auth_request con 0x7f54f403df40 auth_method 0 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.923+0000 7f5505ffb640 10 client.23435.objecter ms_handle_connect 0x7f54f403df40 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.923+0000 7f55154aa640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f54f403df40 0x7f54f4040400 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f550c0ad000 tx=0x7f550c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.923+0000 7f5505ffb640 1 -- 172.21.15.98:0/587485098 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f55100a5150 con 0x7f551006ff70 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.923+0000 7f5505ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:11.923+0000 7f5505ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:12.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.159+0000 7f551770e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:52:12.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.159+0000 7f551770e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:12.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.159+0000 7f551770e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:12.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.159+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5510068f80 con 0x7f551006ff70 2024-09-11T02:52:12.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.159+0000 7f5505ffb640 1 -- 172.21.15.98:0/587485098 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f55100a56a0 con 0x7f551006ff70 2024-09-11T02:52:12.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.159+0000 7f5505ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:12.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.159+0000 7f5505ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:12.162 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":26,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:52:12.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 10 librados: watch_flush enter 2024-09-11T02:52:12.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 10 librados: watch_flush exit 2024-09-11T02:52:12.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 20 client.23435.objecter shutdown clearing up homeless session... 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 10 client.23435.objecter successfully canceled tick 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f54f403df40 msgr2=0x7f54f4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f54f403df40 0x7f54f4040400 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f550c0ad000 tx=0x7f550c004740 comp rx=0 tx=0).stop 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 10 monclient: shutdown 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f551006ff70 msgr2=0x7f5510138e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f551006ff70 0x7f5510138e40 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f550000e840 tx=0x7f550000ec40 comp rx=0 tx=0).stop 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 shutdown_connections 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f54f403df40 0x7f54f4040400 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f55100a8380 0x7f551013e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f551006ff70 0x7f5510138e40 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 --2- 172.21.15.98:0/587485098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f551006f650 0x7f5510138900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 >> 172.21.15.98:0/587485098 conn(0x7f55100ac8e0 msgr2=0x7f55100aa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 shutdown_connections 2024-09-11T02:52:12.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 -- 172.21.15.98:0/587485098 wait complete. 2024-09-11T02:52:12.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.163+0000 7f551770e640 1 librados: shutdown 2024-09-11T02:52:12.181 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:52:12.181 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T02:52:12.181 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:52:12.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:52:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient: get_monmap_and_config 2024-09-11T02:52:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient: monmap: 2024-09-11T02:52:12.296 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:12.297968+0000 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:12.297968+0000 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.297 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:12.298 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:12.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 1 Processor -- start 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 1 -- start start 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient: init 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient: _add_conns ranks=[0,5,10,1,7,2,4,8,6,3,9] 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8e4139b10 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 0x7fc8e413ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient(hunting): picked mon.noname-h con 0x7fc8e4072600 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc8e413ef60 0x7fc8e4141350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient(hunting): picked mon.noname-f con 0x7fc8e413ef60 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e413a600 con 0x7fc8e4139b10 2024-09-11T02:52:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc8e40702b0 con 0x7fc8e413ef60 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc8e4071bf0 con 0x7fc8e4072600 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 10 monclient(hunting): authenticate will time out at 1086.178776s 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e9eaa640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc8e4139b10 0x7fc8e4139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37938/0 (socket says 172.21.15.98:37938) 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e9eaa640 1 -- 172.21.15.98:0/3243919020 learned_addr learned my addr 172.21.15.98:0/3243919020 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 0x7fc8e413ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220924349 0 0) 0x7fc8e413a600 con 0x7fc8e4139b10 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): _negotiate 2024-09-11T02:52:12.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): my global_id is 14256 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8d0003530 con 0x7fc8e4139b10 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request con 0x7fc8e4072600 auth_method 0 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ea6ab640 1 --2- 172.21.15.98:0/3243919020 >> v2:172.21.15.190:3300/0 conn(0x7fc8e413ef60 0x7fc8e4141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 897996477 0 0) 0x7fc8d0003530 con 0x7fc8e4139b10 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 msgr2=0x7fc8e413ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 0x7fc8e413ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 >> v2:172.21.15.190:3300/0 conn(0x7fc8e413ef60 msgr2=0x7fc8e4141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 --2- 172.21.15.98:0/3243919020 >> v2:172.21.15.190:3300/0 conn(0x7fc8e413ef60 0x7fc8e4141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e4071350 con 0x7fc8e4139b10 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fc8e0001970 con 0x7fc8e4139b10 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 5 monclient: authenticate success, global_id 14256 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ea6ab640 10 monclient: get_auth_request con 0x7fc8e413ef60 auth_method 0 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ea6ab640 1 --2- 172.21.15.98:0/3243919020 >> v2:172.21.15.190:3300/0 conn(0x7fc8e413ef60 0x7fc8e4141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: dump: 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.309 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 >> v1:172.21.15.98:6789/0 conn(0x7fc8e4139b10 legacy=0x7fc8e4139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient: _add_conns ranks=[3,5,7,0,8,6,1,4,2] 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8d0004210 0x7fc8d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): picked mon.g con 0x7fc8d0004210 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 0x7fc8d0004e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): picked mon.h con 0x7fc8e4072600 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc8d00099c0 0x7fc8d000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): picked mon.i con 0x7fc8d00099c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e40702b0 con 0x7fc8d0004210 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc8e4071bf0 con 0x7fc8e4072600 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc8d000c3f0 con 0x7fc8d00099c0 2024-09-11T02:52:12.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8e8ea8640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.295+0000 7fc8ec10e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 0x7fc8d0004e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8d0004210 0x7fc8d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request con 0x7fc8e4072600 auth_method 0 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc8d00099c0 0x7fc8d000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 10 monclient(hunting): get_auth_request con 0x7fc8d0004210 auth_method 0 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:12.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 10 monclient(hunting): get_auth_request con 0x7fc8d00099c0 auth_method 0 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e9eaa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ea6ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:52:12.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_done global_id 14256 payload 293 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 -- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc8d00099c0 msgr2=0x7fc8d000bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc8d00099c0 0x7fc8d000bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 -- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8d0004210 msgr2=0x7fc8d0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8d0004210 0x7fc8d0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 monclient: found mon.h 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 -- 172.21.15.98:0/3243919020 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d0003530 con 0x7fc8e4072600 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 0x7fc8d0004e00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4009ae0 tx=0x7fc8d401ca60 comp rx=0 tx=0).ready entity=mon.5 client_cookie=2cc02b09c592fe72 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8d40b2070 con 0x7fc8e4072600 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 10 monclient: dump: 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d40a5ac0 con 0x7fc8e4072600 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ec10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ec10e640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ec10e640 4 set_mon_vals no callback set 2024-09-11T02:52:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 1 -- 172.21.15.98:0/3243919020 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8d40096b0 con 0x7fc8e4072600 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8e8ea8640 10 monclient: dump: 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ec10e640 10 monclient: shutdown 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ec10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 msgr2=0x7fc8d0004e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.299+0000 7fc8ec10e640 1 --2- 172.21.15.98:0/3243919020 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8e4072600 0x7fc8d0004e00 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4009ae0 tx=0x7fc8d401ca60 comp rx=0 tx=0).stop 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 1 -- 172.21.15.98:0/3243919020 reap_dead start 2024-09-11T02:52:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3243919020 shutdown_connections 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3243919020 >> 172.21.15.98:0/3243919020 conn(0x7fc8e40ac790 msgr2=0x7fc8e40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3243919020 shutdown_connections 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3243919020 wait complete. 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: monmap: 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 librados: starting msgr at 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 librados: starting objecter 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 Processor -- start 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- start start 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 librados: setting wanted keys 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 librados: calling monclient init 2024-09-11T02:52:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: init 2024-09-11T02:52:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: _add_conns ranks=[6,3,2,8,5,4,0,7,1] 2024-09-11T02:52:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8e4072600 0x7fc8e40a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): picked mon.noname-d con 0x7fc8e4072600 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8e4123a30 0x7fc8e4138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): picked mon.noname-g con 0x7fc8e4123a30 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8e4138dc0 0x7fc8e4122650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): picked mon.noname-b con 0x7fc8e4138dc0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e41420a0 con 0x7fc8e4138dc0 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e40517e0 con 0x7fc8e4123a30 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc8e4142220 con 0x7fc8e4072600 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient(hunting): authenticate will time out at 1086.185805s 2024-09-11T02:52:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8e4072600 0x7fc8e40a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8e4072600 0x7fc8e40a4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58400/0 (socket says 172.21.15.98:58400) 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 -- 172.21.15.98:0/3802089297 learned_addr learned my addr 172.21.15.98:0/3802089297 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): get_auth_request con 0x7fc8e4072600 auth_method 0 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8e4138dc0 0x7fc8e4122650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 10 monclient(hunting): get_auth_request con 0x7fc8e4138dc0 auth_method 0 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ea6ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8e4123a30 0x7fc8e4138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request con 0x7fc8e4123a30 auth_method 0 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e96a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e96a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient(hunting): handle_auth_done global_id 14253 payload 824 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8e4123a30 msgr2=0x7fc8e4138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8e4123a30 0x7fc8e4138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8e4138dc0 msgr2=0x7fc8e4122650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8e4138dc0 0x7fc8e4122650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 monclient: found mon.noname-d 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 -- 172.21.15.98:0/3802089297 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e40a5930 con 0x7fc8e4072600 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8e9eaa640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8e4072600 0x7fc8e40a4ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00073a0 tx=0x7fc8e0007860 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 client.?.objecter ms_handle_connect 0x7fc8e4072600 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8e001a070 con 0x7fc8e4072600 2024-09-11T02:52:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: dump: 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8e00013c0 con 0x7fc8e4072600 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8e000fd80 con 0x7fc8e4072600 2024-09-11T02:52:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8da7fc640 10 monclient: dump: 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8dbfff640 4 set_mon_vals no callback set 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 5 monclient: authenticate success, global_id 14253 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: _renew_subs 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e41c7570 con 0x7fc8e4072600 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 client.14253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: _renew_subs 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e4141360 con 0x7fc8e4072600 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 1 librados: init done 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.303+0000 7fc8ec10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8ec10e640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402d450 0x7fc8e402d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8ec10e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e406ca10 con 0x7fc8e402d450 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc8e000f910 con 0x7fc8e4072600 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402d450 0x7fc8e402d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient: get_auth_request con 0x7fc8e402d450 auth_method 0 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc8c803df40 0x7fc8c8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(86..86 src has 1..86) ==== 8541+0+0 (secure 0 0 0) 0x7fc8e0010040 con 0x7fc8e4072600 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 client.14253.objecter ms_dispatch 0x7fc8e4000ba0 osd_map(86..86 src has 1..86) 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 3 client.14253.objecter handle_osd_map got epochs [86,86] > 0 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 3 client.14253.objecter handle_osd_map decoding full epoch 86 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_done global_id 14265 payload 824 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 1 -- 172.21.15.98:0/3802089297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8e40a9f00 con 0x7fc8e402d450 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 20 client.14253.objecter dump_active .. 0 homeless 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 client.14253.objecter ms_handle_connect 0x7fc8e402d450 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 client.14253.objecter resend_mon_ops 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402d450 0x7fc8e402d830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4009ab0 tx=0x7fc8d0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8ea6ab640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc8c803df40 0x7fc8c8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8ea6ab640 10 monclient: get_auth_request con 0x7fc8c803df40 auth_method 0 2024-09-11T02:52:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8d40bc070 con 0x7fc8e402d450 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 monclient: dump: 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc8d40096b0 con 0x7fc8e402d450 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402d450 msgr2=0x7fc8e402d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402d450 0x7fc8e402d830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4009ab0 tx=0x7fc8d0005840 comp rx=0 tx=0).stop 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402d450 msgr2=0x7fc8e402d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402d450 0x7fc8e402d830 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc8d4009ab0 tx=0x7fc8d0005840 comp rx=0 tx=0).stop 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8da7fc640 10 client.14253.objecter ms_handle_connect 0x7fc8c803df40 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.307+0000 7fc8ea6ab640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc8c803df40 0x7fc8c8040400 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40716a0 tx=0x7fc8dc00f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8ec10e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8ec10e640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402dd70 0x7fc8e402e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8ec10e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e406e680 con 0x7fc8e402dd70 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402dd70 0x7fc8e402e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient: get_auth_request con 0x7fc8e402dd70 auth_method 0 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 10 monclient(hunting): handle_auth_done global_id 14274 payload 824 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 1 -- 172.21.15.98:0/3802089297 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc8e4139ef0 con 0x7fc8e402dd70 2024-09-11T02:52:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8e96a9640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402dd70 0x7fc8e402e180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40b0040 tx=0x7fc8d40a7070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8da7fc640 10 client.14253.objecter ms_handle_connect 0x7fc8e402dd70 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.311+0000 7fc8da7fc640 10 client.14253.objecter resend_mon_ops 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8d40b2040 con 0x7fc8e402dd70 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 10 monclient: dump: 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7fc8d4003620 con 0x7fc8e402dd70 2024-09-11T02:52:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402dd70 msgr2=0x7fc8e402e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402dd70 0x7fc8e402e180 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40b0040 tx=0x7fc8d40a7070 comp rx=0 tx=0).stop 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402dd70 msgr2=0x7fc8e402e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8da7fc640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8e402dd70 0x7fc8e402e180 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc8d40b0040 tx=0x7fc8d40a7070 comp rx=0 tx=0).stop 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:12.343 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 12367, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:12.344 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.345 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:12.347 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:12.366 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 10 librados: watch_flush enter 2024-09-11T02:52:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 10 librados: watch_flush exit 2024-09-11T02:52:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 20 client.14253.objecter shutdown clearing up homeless session... 2024-09-11T02:52:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 10 client.14253.objecter successfully canceled tick 2024-09-11T02:52:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc8c803df40 msgr2=0x7fc8c8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc8c803df40 0x7fc8c8040400 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40716a0 tx=0x7fc8dc00f040 comp rx=0 tx=0).stop 2024-09-11T02:52:12.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 10 monclient: shutdown 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8e4072600 msgr2=0x7fc8e40a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 --2- 172.21.15.98:0/3802089297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8e4072600 0x7fc8e40a4ea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00073a0 tx=0x7fc8e0007860 comp rx=0 tx=0).stop 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ea6ab640 1 -- 172.21.15.98:0/3802089297 reap_dead start 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ea6ab640 1 -- 172.21.15.98:0/3802089297 reap_dead start 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 shutdown_connections 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 >> 172.21.15.98:0/3802089297 conn(0x7fc8e40ac790 msgr2=0x7fc8e40aa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 shutdown_connections 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 -- 172.21.15.98:0/3802089297 wait complete. 2024-09-11T02:52:12.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.315+0000 7fc8ec10e640 1 librados: shutdown 2024-09-11T02:52:12.370 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:12.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:52:12.370 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:52:12.467 INFO:tasks.workunit.client.0.smithi190.stdout:list: : entry=9 expected=9 2024-09-11T02:52:12.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-09-11T02:52:12.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:de5d7c5f:::12:head expected=10:de5d7c5f:::12:head 2024-09-11T02:52:12.467 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=12 expected=12 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:d83876eb:::4:head expected=10:d83876eb:::4:head 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=4 expected=4 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:cfc208b3:::3:head expected=10:cfc208b3:::3:head 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=3 expected=3 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:c4fdafeb:::6:head expected=10:c4fdafeb:::6:head 2024-09-11T02:52:12.468 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=6 expected=6 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:bd63b0f1:::8:head expected=10:bd63b0f1:::8:head 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=8 expected=8 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:b29083e3:::5:head expected=10:b29083e3:::5:head 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=5 expected=5 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:89d3ae78:::11:head expected=10:89d3ae78:::11:head 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-09-11T02:52:12.469 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=11 expected=11 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : cursor()=10:02547ec2:::1:head expected=10:02547ec2:::1:head 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: : entry=1 expected=1 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (324 ms) 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (51015 ms) 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 0/5 -> MIN 10:33333333::::head 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 1/5 -> 10:33333333::::head 10:66666666::::head 2024-09-11T02:52:12.470 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 2/5 -> 10:66666666::::head 10:99999999::::head 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 3/5 -> 10:99999999::::head 10:cccccccc::::head 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: split 4/5 -> 10:cccccccc::::head MAX 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (13089 ms) 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 7 tests from LibRadosList (65048 ms total) 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1027 ms) 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo2,foo3 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo2 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo3 2024-09-11T02:52:12.471 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo1,foo4,foo5 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo4 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo5 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo1 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset foo6,foo7 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo7 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: foo6 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo4 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo5 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo7 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns2:foo6 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: ns1:foo1 2024-09-11T02:52:12.472 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo1 2024-09-11T02:52:12.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo2 2024-09-11T02:52:12.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: :foo3 2024-09-11T02:52:12.473 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (83 ms) 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient: get_monmap_and_config 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient: monmap: 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:12.491493+0000 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:12.491493+0000 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:12.490 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:12.491 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 1 Processor -- start 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 1 -- start start 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient: init 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient: _add_conns ranks=[0,4,3,8,2,5,6,7,10,1,9] 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c4c0a46a0 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c4c0a5190 0x7f9c4c13ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient(hunting): picked mon.noname-c con 0x7f9c4c0a5190 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c13f120 0x7f9c4c141510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient(hunting): picked mon.noname-g con 0x7f9c4c13f120 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c4c0a17b0 con 0x7f9c4c0a46a0 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c0a0750 con 0x7f9c4c13f120 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9c4c0775d0 con 0x7f9c4c0a5190 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.487+0000 7f9c52059640 10 monclient(hunting): authenticate will time out at 1086.371986s 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4b7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9c4c0a46a0 0x7f9c4c0a4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37950/0 (socket says 172.21.15.98:37950) 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4b7fe640 1 -- 172.21.15.98:0/3811073113 learned_addr learned my addr 172.21.15.98:0/3811073113 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 1 --2- 172.21.15.98:0/3811073113 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c4c0a5190 0x7f9c4c13ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): get_auth_request con 0x7f9c4c0a5190 auth_method 0 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4bfff640 1 --2- 172.21.15.98:0/3811073113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c13f120 0x7f9c4c141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4bfff640 10 monclient(hunting): get_auth_request con 0x7f9c4c13f120 auth_method 0 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 1 -- 172.21.15.98:0/3811073113 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3542154128 0 0) 0x7f9c4c0a17b0 con 0x7f9c4c0a46a0 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient(hunting): _negotiate 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient(hunting): my global_id is 14283 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 1 -- 172.21.15.98:0/3811073113 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c38003530 con 0x7f9c4c0a46a0 2024-09-11T02:52:12.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient(hunting): handle_auth_done global_id 22999 payload 274 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 1 -- 172.21.15.98:0/3811073113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c13f120 msgr2=0x7f9c4c141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 1 --2- 172.21.15.98:0/3811073113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c13f120 0x7f9c4c141510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 1 -- 172.21.15.98:0/3811073113 >> v1:172.21.15.98:6789/0 conn(0x7f9c4c0a46a0 legacy=0x7f9c4c0a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 1 monclient: found mon.noname-c 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:12.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 1 -- 172.21.15.98:0/3811073113 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c0a0200 con 0x7f9c4c0a5190 2024-09-11T02:52:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c43fff640 1 --2- 172.21.15.98:0/3811073113 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c4c0a5190 0x7f9c4c13ebe0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f9c340097f0 tx=0x7f9c3401c3a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=8ec54146356e0e39 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 1 -- 172.21.15.98:0/3811073113 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c340b0070 con 0x7f9c4c0a5190 2024-09-11T02:52:12.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: dump: 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.504 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 1 -- 172.21.15.98:0/3811073113 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c340aa810 con 0x7f9c4c0a5190 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 1 -- 172.21.15.98:0/3811073113 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c340a9300 con 0x7f9c4c0a5190 2024-09-11T02:52:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 5 monclient: authenticate success, global_id 22999 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 4 set_mon_vals no callback set 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c4affd640 10 monclient: dump: 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 10 monclient: shutdown 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 -- 172.21.15.98:0/3811073113 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c4c0a5190 msgr2=0x7f9c4c13ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 --2- 172.21.15.98:0/3811073113 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c4c0a5190 0x7f9c4c13ebe0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f9c340097f0 tx=0x7f9c3401c3a0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 -- 172.21.15.98:0/3811073113 shutdown_connections 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 --2- 172.21.15.98:0/3811073113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c13f120 0x7f9c4c141510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 --2- 172.21.15.98:0/3811073113 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9c4c0a5190 0x7f9c4c13ebe0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 -- 172.21.15.98:0/3811073113 >> 172.21.15.98:0/3811073113 conn(0x7f9c4c06ed00 msgr2=0x7f9c4c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 -- 172.21.15.98:0/3811073113 shutdown_connections 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.491+0000 7f9c52059640 1 -- 172.21.15.98:0/3811073113 wait complete. 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: monmap: 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 librados: starting msgr at 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 librados: starting objecter 2024-09-11T02:52:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 Processor -- start 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 -- start start 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 librados: setting wanted keys 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 librados: calling monclient init 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: init 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: _add_conns ranks=[3,0,8,7,2,1,4,6,5] 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0a46a0 0x7f9c4c1d15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): picked mon.noname-g con 0x7f9c4c0a46a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c4c0a5190 0x7f9c4c1d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): picked mon.noname-a con 0x7f9c4c0a5190 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c4c13f120 0x7f9c4c13a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): picked mon.noname-e con 0x7f9c4c13f120 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c144270 con 0x7f9c4c0a5190 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c077a60 con 0x7f9c4c0a46a0 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9c4c1443f0 con 0x7f9c4c13f120 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): authenticate will time out at 1086.377269s 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c4c0a5190 0x7f9c4c1d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c4c13f120 0x7f9c4c13a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c4c0a5190 0x7f9c4c1d1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59566/0 (socket says 172.21.15.98:59566) 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 10 monclient(hunting): get_auth_request con 0x7f9c4c13f120 auth_method 0 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 -- 172.21.15.98:0/408832165 learned_addr learned my addr 172.21.15.98:0/408832165 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): get_auth_request con 0x7f9c4c0a5190 auth_method 0 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0a46a0 0x7f9c4c1d15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): get_auth_request con 0x7f9c4c0a46a0 auth_method 0 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient(hunting): handle_auth_done global_id 14292 payload 824 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c4c13f120 msgr2=0x7f9c4c13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9c4c13f120 0x7f9c4c13a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0a46a0 msgr2=0x7f9c4c1d15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9c4c0a46a0 0x7f9c4c1d15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 monclient: found mon.noname-a 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 -- 172.21.15.98:0/408832165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c13ee90 con 0x7f9c4c0a5190 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c43fff640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c4c0a5190 0x7f9c4c1d1b00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9c3401aa50 tx=0x7f9c34019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 client.?.objecter ms_handle_connect 0x7f9c4c0a5190 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c340b0040 con 0x7f9c4c0a5190 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: dump: 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c340044b0 con 0x7f9c4c0a5190 2024-09-11T02:52:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c340aaca0 con 0x7f9c4c0a5190 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 monclient: dump: 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4a7fc640 4 set_mon_vals no callback set 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 5 monclient: authenticate success, global_id 14292 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: _renew_subs 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c13dd90 con 0x7f9c4c0a5190 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 client.14292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:12.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: _renew_subs 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4c13e350 con 0x7f9c4c0a5190 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 librados: init done 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13aef0 0x7f9c4c13b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c06ca10 con 0x7f9c4c13aef0 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9c340b3070 con 0x7f9c4c0a5190 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13aef0 0x7f9c4c13b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient: get_auth_request con 0x7f9c4c13aef0 auth_method 0 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9c2c03df30 0x7f9c2c0403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9323+0+0 (secure 0 0 0) 0x7f9c340b9070 con 0x7f9c4c0a5190 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 10 client.14292.objecter ms_dispatch 0x7f9c4c000ba0 osd_map(87..87 src has 1..87) 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 3 client.14292.objecter handle_osd_map got epochs [87,87] > 0 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.495+0000 7f9c48ff9640 3 client.14292.objecter handle_osd_map decoding full epoch 87 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_done global_id 23051 payload 824 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c4b7fe640 1 -- 172.21.15.98:0/408832165 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c13e190 con 0x7f9c4c13aef0 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 20 client.14292.objecter dump_active .. 0 homeless 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c4bfff640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9c2c03df30 0x7f9c2c0403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c4bfff640 10 monclient: get_auth_request con 0x7f9c2c03df30 auth_method 0 2024-09-11T02:52:12.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 client.14292.objecter ms_handle_connect 0x7f9c4c13aef0 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 client.14292.objecter resend_mon_ops 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c4b7fe640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13aef0 0x7f9c4c13b2d0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005890 tx=0x7f9c3c00e980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c3c0070d0 con 0x7f9c4c13aef0 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 monclient: dump: 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.520 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9c3c018070 con 0x7f9c4c13aef0 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13aef0 msgr2=0x7f9c4c13b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13aef0 0x7f9c4c13b2d0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005890 tx=0x7f9c3c00e980 comp rx=0 tx=0).stop 2024-09-11T02:52:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13aef0 msgr2=0x7f9c4c13b2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13aef0 0x7f9c4c13b2d0 secure :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005890 tx=0x7f9c3c00e980 comp rx=0 tx=0).stop 2024-09-11T02:52:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c48ff9640 10 client.14292.objecter ms_handle_connect 0x7f9c2c03df30 2024-09-11T02:52:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.499+0000 7f9c4bfff640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9c2c03df30 0x7f9c2c0403f0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c09f990 tx=0x7f9c44009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13b810 0x7f9c4c0a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c06e870 con 0x7f9c4c13b810 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13b810 0x7f9c4c0a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient: get_auth_request con 0x7f9c4c13b810 auth_method 0 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 10 monclient(hunting): handle_auth_done global_id 23060 payload 824 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 1 -- 172.21.15.98:0/408832165 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c4c13c8d0 con 0x7f9c4c13b810 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4b7fe640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13b810 0x7f9c4c0a7830 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005890 tx=0x7f9c3c005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 10 client.14292.objecter ms_handle_connect 0x7f9c4c13b810 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 10 client.14292.objecter resend_mon_ops 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9c3c007700 con 0x7f9c4c13b810 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 10 monclient: dump: 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f9c3c007700 con 0x7f9c4c13b810 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13b810 msgr2=0x7f9c4c0a7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13b810 0x7f9c4c0a7830 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005890 tx=0x7f9c3c005800 comp rx=0 tx=0).stop 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13b810 msgr2=0x7f9c4c0a7830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c48ff9640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9c4c13b810 0x7f9c4c0a7830 secure :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c005890 tx=0x7f9c3c005800 comp rx=0 tx=0).stop 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:12.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 148291, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:12.528 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.529 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.530 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:12.531 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.532 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.533 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.534 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.536 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:12.537 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:12.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:12.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:12.551 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 10 librados: watch_flush enter 2024-09-11T02:52:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 10 librados: watch_flush exit 2024-09-11T02:52:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 20 client.14292.objecter shutdown clearing up homeless session... 2024-09-11T02:52:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 10 client.14292.objecter successfully canceled tick 2024-09-11T02:52:12.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9c2c03df30 msgr2=0x7f9c2c0403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9c2c03df30 0x7f9c2c0403f0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c09f990 tx=0x7f9c44009780 comp rx=0 tx=0).stop 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 10 monclient: shutdown 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c4c0a5190 msgr2=0x7f9c4c1d1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 --2- 172.21.15.98:0/408832165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c4c0a5190 0x7f9c4c1d1b00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9c3401aa50 tx=0x7f9c34019140 comp rx=0 tx=0).stop 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4bfff640 1 -- 172.21.15.98:0/408832165 reap_dead start 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c4bfff640 1 -- 172.21.15.98:0/408832165 reap_dead start 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 shutdown_connections 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 >> 172.21.15.98:0/408832165 conn(0x7f9c4c06ed00 msgr2=0x7f9c4c0739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 shutdown_connections 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 -- 172.21.15.98:0/408832165 wait complete. 2024-09-11T02:52:12.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.507+0000 7f9c52059640 1 librados: shutdown 2024-09-11T02:52:12.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:12.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:52:12.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:52:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient: get_monmap_and_config 2024-09-11T02:52:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient: monmap: 2024-09-11T02:52:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:12.669540+0000 2024-09-11T02:52:12.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:12.669540+0000 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:12.669 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 1 Processor -- start 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 1 -- start start 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient: init 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient: _add_conns ranks=[1,4,10,0,6,5,2,8,9,3,7] 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f5108139b80 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f510813ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient(hunting): picked mon.noname-c con 0x7f510806fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5108072640 0x7f5108140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient(hunting): picked mon.noname-f con 0x7f5108072640 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5108071370 con 0x7f5108139b80 2024-09-11T02:52:12.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f51080719c0 con 0x7f5108072640 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5108071c10 con 0x7f510806fe50 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f510e003640 10 monclient(hunting): authenticate will time out at 1086.549767s 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f510813ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f510813ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45886/0 (socket says 172.21.15.98:45886) 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 -- 172.21.15.98:0/3468811970 learned_addr learned my addr 172.21.15.98:0/3468811970 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): get_auth_request con 0x7f510806fe50 auth_method 0 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5107fff640 1 --2- 172.21.15.98:0/3468811970 >> v2:172.21.15.190:3300/0 conn(0x7f5108072640 0x7f5108140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5107fff640 10 monclient(hunting): get_auth_request con 0x7f5108072640 auth_method 0 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5107fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5107fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5107fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient(hunting): handle_auth_done global_id 23008 payload 274 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 -- 172.21.15.98:0/3468811970 >> v2:172.21.15.190:3300/0 conn(0x7f5108072640 msgr2=0x7f5108140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 --2- 172.21.15.98:0/3468811970 >> v2:172.21.15.190:3300/0 conn(0x7f5108072640 0x7f5108140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 -- 172.21.15.98:0/3468811970 >> v1:172.21.15.190:6789/0 conn(0x7f5108139b80 legacy=0x7f5108139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 monclient: found mon.noname-c 2024-09-11T02:52:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:12.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.667+0000 7f5106ffd640 1 -- 172.21.15.98:0/3468811970 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51080a9600 con 0x7f510806fe50 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 --2- 172.21.15.98:0/3468811970 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f510813ea00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009870 tx=0x7f50f001c420 comp rx=0 tx=0).ready entity=mon.4 client_cookie=dab970761e2a969a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 1 -- 172.21.15.98:0/3468811970 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f00b0070 con 0x7f510806fe50 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: dump: 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 1 -- 172.21.15.98:0/3468811970 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f00aa810 con 0x7f510806fe50 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 5 monclient: authenticate success, global_id 23008 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 1 -- 172.21.15.98:0/3468811970 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f00a9300 con 0x7f510806fe50 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 4 set_mon_vals no callback set 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51067fc640 10 monclient: dump: 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.683 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.684 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient: shutdown 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- 172.21.15.98:0/3468811970 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 msgr2=0x7f510813ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 --2- 172.21.15.98:0/3468811970 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f510813ea00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f50f0009870 tx=0x7f50f001c420 comp rx=0 tx=0).stop 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- 172.21.15.98:0/3468811970 shutdown_connections 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 --2- 172.21.15.98:0/3468811970 >> v2:172.21.15.190:3300/0 conn(0x7f5108072640 0x7f5108140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 --2- 172.21.15.98:0/3468811970 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f510813ea00 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- 172.21.15.98:0/3468811970 >> 172.21.15.98:0/3468811970 conn(0x7f51080ac730 msgr2=0x7f51080a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- 172.21.15.98:0/3468811970 shutdown_connections 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- 172.21.15.98:0/3468811970 wait complete. 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient: monmap: 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.686 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 librados: starting msgr at 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 librados: starting objecter 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 Processor -- start 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- start start 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 librados: setting wanted keys 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 librados: calling monclient init 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient: init 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient: _add_conns ranks=[4,0,3,6,2,7,1,5,8] 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f5108137f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): picked mon.noname-c con 0x7f510806fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5108072640 0x7f5108138470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): picked mon.noname-a con 0x7f5108072640 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5108139b80 0x7f510813d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): picked mon.noname-g con 0x7f5108139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5108144040 con 0x7f5108072640 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5108143ec0 con 0x7f5108139b80 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f51081441c0 con 0x7f510806fe50 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f510e003640 10 monclient(hunting): authenticate will time out at 1086.555489s 2024-09-11T02:52:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5108072640 0x7f5108138470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f5108137f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5108072640 0x7f5108138470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59574/0 (socket says 172.21.15.98:59574) 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 -- 172.21.15.98:0/2439869526 learned_addr learned my addr 172.21.15.98:0/2439869526 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): get_auth_request con 0x7f5108072640 auth_method 0 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 10 monclient(hunting): get_auth_request con 0x7f510806fe50 auth_method 0 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient(hunting): handle_auth_done global_id 14301 payload 824 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 msgr2=0x7f5108137f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510806fe50 0x7f5108137f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5108139b80 msgr2=0x7f510813d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5107fff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5108139b80 0x7f510813d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5108139b80 0x7f510813d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 monclient: found mon.noname-a 2024-09-11T02:52:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.671+0000 7f5106ffd640 1 -- 172.21.15.98:0/2439869526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510813f480 con 0x7f5108072640 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f5106ffd640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5108072640 0x7f5108138470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50f00ae040 tx=0x7f50f00191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 client.?.objecter ms_handle_connect 0x7f5108072640 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f00b0040 con 0x7f5108072640 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: dump: 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.691 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f0004240 con 0x7f5108072640 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f00043f0 con 0x7f5108072640 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 5 monclient: authenticate success, global_id 14301 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 10 monclient: _renew_subs 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5108122f90 con 0x7f5108072640 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f5105ffb640 4 set_mon_vals no callback set 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: dump: 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.693 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.694 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 10 client.14301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 10 monclient: _renew_subs 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51080a6e20 con 0x7f5108072640 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 1 librados: init done 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e130 0x7f510802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f510813eda0 con 0x7f510813e130 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e130 0x7f510802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient: get_auth_request con 0x7f510813e130 auth_method 0 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f50f00b3070 con 0x7f5108072640 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 10 monclient(hunting): handle_auth_done global_id 23026 payload 824 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50e803df40 0x7f50e8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 1 -- 172.21.15.98:0/2439869526 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5108139f60 con 0x7f510813e130 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9323+0+0 (secure 0 0 0) 0x7f50f00e9920 con 0x7f5108072640 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 client.14301.objecter ms_dispatch 0x7f5108000ba0 osd_map(87..87 src has 1..87) 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 3 client.14301.objecter handle_osd_map got epochs [87,87] > 0 2024-09-11T02:52:12.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 3 client.14301.objecter handle_osd_map decoding full epoch 87 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 20 client.14301.objecter dump_active .. 0 homeless 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 client.14301.objecter ms_handle_connect 0x7f510813e130 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 client.14301.objecter resend_mon_ops 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f51077fe640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e130 0x7f510802d450 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f510000bea0 tx=0x7f510000bf90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f5107fff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50e803df40 0x7f50e8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f5107fff640 10 monclient: get_auth_request con 0x7f50e803df40 auth_method 0 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5100013a00 con 0x7f510813e130 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: dump: 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.697 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.698 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5100013ba0 con 0x7f510813e130 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e130 msgr2=0x7f510802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e130 0x7f510802d450 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f510000bea0 tx=0x7f510000bf90 comp rx=0 tx=0).stop 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e130 msgr2=0x7f510802d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e130 0x7f510802d450 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f50fffff640 10 client.14301.objecter ms_handle_connect 0x7f50e803df40 2024-09-11T02:52:12.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.675+0000 7f5107fff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50e803df40 0x7f50e8040400 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f50f8009710 tx=0x7f50f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e510 0x7f510802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f510806ca10 con 0x7f510813e510 2024-09-11T02:52:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e510 0x7f510802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient: get_auth_request con 0x7f510813e510 auth_method 0 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 10 monclient(hunting): handle_auth_done global_id 23035 payload 824 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 1 -- 172.21.15.98:0/2439869526 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51080a4d40 con 0x7f510813e510 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 10 client.14301.objecter ms_handle_connect 0x7f510813e510 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f51077fe640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e510 0x7f510802d990 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f510000bea0 tx=0x7f5100004a80 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 10 client.14301.objecter resend_mon_ops 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5100014890 con 0x7f510813e510 2024-09-11T02:52:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 10 monclient: dump: 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f5100014420 con 0x7f510813e510 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e510 msgr2=0x7f510802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e510 0x7f510802d990 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f510000bea0 tx=0x7f5100004a80 comp rx=0 tx=0).stop 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e510 msgr2=0x7f510802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f50fffff640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f510813e510 0x7f510802d990 secure :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f510000bea0 tx=0x7f5100004a80 comp rx=0 tx=0).stop 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 148467, 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:12.705 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.707 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:12.708 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:12.709 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.710 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:12.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.715 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.717 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.723 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:12.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:12.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:12.728 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 10 librados: watch_flush enter 2024-09-11T02:52:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 10 librados: watch_flush exit 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 20 client.14301.objecter shutdown clearing up homeless session... 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 10 client.14301.objecter successfully canceled tick 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50e803df40 msgr2=0x7f50e8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50e803df40 0x7f50e8040400 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7f50f8009710 tx=0x7f50f8008040 comp rx=0 tx=0).stop 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 10 monclient: shutdown 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5108072640 msgr2=0x7f5108138470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 --2- 172.21.15.98:0/2439869526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5108072640 0x7f5108138470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f50f00ae040 tx=0x7f50f00191c0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f5107fff640 1 -- 172.21.15.98:0/2439869526 reap_dead start 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f5107fff640 1 -- 172.21.15.98:0/2439869526 reap_dead start 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 shutdown_connections 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 >> 172.21.15.98:0/2439869526 conn(0x7f51080ac730 msgr2=0x7f51080a7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 shutdown_connections 2024-09-11T02:52:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 -- 172.21.15.98:0/2439869526 wait complete. 2024-09-11T02:52:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.683+0000 7f510e003640 1 librados: shutdown 2024-09-11T02:52:12.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:12.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:52:12.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient: get_monmap_and_config 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient: monmap: 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:12.849561+0000 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:12.849561+0000 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.848 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:12.849 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 1 Processor -- start 2024-09-11T02:52:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 1 -- start start 2024-09-11T02:52:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient: init 2024-09-11T02:52:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient: _add_conns ranks=[0,9,5,8,10,1,4,3,2,7,6] 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7acc09b8c0 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7acc09bca0 0x7f7acc13e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient(hunting): picked mon.noname-a con 0x7f7acc09bca0 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7acc13eac0 0x7f7acc140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient(hunting): picked mon.noname-h con 0x7f7acc13eac0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc09d5a0 con 0x7f7acc09b8c0 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f7acc09d880 con 0x7f7acc09bca0 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f7acc0a5d10 con 0x7f7acc13eac0 2024-09-11T02:52:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ad1476640 10 monclient(hunting): authenticate will time out at 1086.729727s 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f7acc09bca0 0x7f7acc13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7acaffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7acc09b8c0 0x7f7acc09df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37952/0 (socket says 172.21.15.98:37952) 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): get_auth_request con 0x7f7acc09bca0 auth_method 0 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7acaffd640 1 -- 172.21.15.98:0/3567623918 learned_addr learned my addr 172.21.15.98:0/3567623918 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7acb7fe640 1 --2- 172.21.15.98:0/3567623918 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7acc13eac0 0x7f7acc140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_done global_id 14310 payload 274 2024-09-11T02:52:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 1 -- 172.21.15.98:0/3567623918 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7acc13eac0 msgr2=0x7f7acc140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 1 --2- 172.21.15.98:0/3567623918 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7acc13eac0 0x7f7acc140eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 1 -- 172.21.15.98:0/3567623918 >> v1:172.21.15.98:6789/0 conn(0x7f7acc09b8c0 legacy=0x7f7acc09df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 1 monclient: found mon.noname-a 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7aca7fc640 1 -- 172.21.15.98:0/3567623918 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc09c080 con 0x7f7acc09bca0 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7acb7fe640 10 monclient: get_auth_request con 0x7f7acc13eac0 auth_method 0 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7acb7fe640 1 --2- 172.21.15.98:0/3567623918 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7acc13eac0 0x7f7acc140eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ac9ffb640 1 -- 172.21.15.98:0/3567623918 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1440559292 0 0) 0x7f7acc09d5a0 con 0x7f7acc09b8c0 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.847+0000 7f7ac9ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:52:12.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 --2- 172.21.15.98:0/3567623918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7acc09bca0 0x7f7acc13e580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00097c0 tx=0x7f7ac001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a01d35cf3b7bdb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 1 -- 172.21.15.98:0/3567623918 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ac00b0070 con 0x7f7acc09bca0 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: dump: 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.857 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 1 -- 172.21.15.98:0/3567623918 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac00aa810 con 0x7f7acc09bca0 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 5 monclient: authenticate success, global_id 14310 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:12.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 1 -- 172.21.15.98:0/3567623918 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ac00a9300 con 0x7f7acc09bca0 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 4 set_mon_vals no callback set 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac9ffb640 10 monclient: dump: 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.859 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: shutdown 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- 172.21.15.98:0/3567623918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7acc09bca0 msgr2=0x7f7acc13e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 --2- 172.21.15.98:0/3567623918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7acc09bca0 0x7f7acc13e580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ac00097c0 tx=0x7f7ac001c360 comp rx=0 tx=0).stop 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- 172.21.15.98:0/3567623918 shutdown_connections 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 --2- 172.21.15.98:0/3567623918 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7acc13eac0 0x7f7acc140eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 --2- 172.21.15.98:0/3567623918 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7acc09bca0 0x7f7acc13e580 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- 172.21.15.98:0/3567623918 >> 172.21.15.98:0/3567623918 conn(0x7f7acc0a4ed0 msgr2=0x7f7acc0a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- 172.21.15.98:0/3567623918 shutdown_connections 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- 172.21.15.98:0/3567623918 wait complete. 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: build_initial_monmap 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: monmap: 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:12.861 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 librados: starting msgr at 2024-09-11T02:52:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 librados: starting objecter 2024-09-11T02:52:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 Processor -- start 2024-09-11T02:52:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- start start 2024-09-11T02:52:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 librados: setting wanted keys 2024-09-11T02:52:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 librados: calling monclient init 2024-09-11T02:52:12.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: init 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: _add_conns ranks=[8,6,3,7,0,4,1,5,2] 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7acc09b8c0 0x7f7acc13a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): picked mon.noname-e con 0x7f7acc09b8c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc09bca0 0x7f7acc13aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): picked mon.noname-d con 0x7f7acc09bca0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7acc13eac0 0x7f7acc13afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): picked mon.noname-g con 0x7f7acc13eac0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7acc143fa0 con 0x7f7acc13eac0 2024-09-11T02:52:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7acc143e20 con 0x7f7acc09bca0 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7acc144120 con 0x7f7acc09b8c0 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient(hunting): authenticate will time out at 1086.734457s 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7acc09b8c0 0x7f7acc13a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc09bca0 0x7f7acc13aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc09bca0 0x7f7acc13aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58412/0 (socket says 172.21.15.98:58412) 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7acc09b8c0 0x7f7acc13a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56868/0 (socket says 172.21.15.98:56868) 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 -- 172.21.15.98:0/1641280602 learned_addr learned my addr 172.21.15.98:0/1641280602 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): get_auth_request con 0x7f7acc09bca0 auth_method 0 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 10 monclient(hunting): get_auth_request con 0x7f7acc09b8c0 auth_method 0 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acb7fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7acc13eac0 0x7f7acc13afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acb7fe640 10 monclient(hunting): get_auth_request con 0x7f7acc13eac0 auth_method 0 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient(hunting): handle_auth_done global_id 14262 payload 824 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7acc09b8c0 msgr2=0x7f7acc13a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7acc09b8c0 0x7f7acc13a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7acc13eac0 msgr2=0x7f7acc13afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7acc13eac0 0x7f7acc13afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 monclient: found mon.noname-d 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 -- 172.21.15.98:0/1641280602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc0739c0 con 0x7f7acc09bca0 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7acaffd640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7acc09b8c0 0x7f7acc13a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7aca7fc640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc09bca0 0x7f7acc13aaa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0019f90 tx=0x7f7ac0019100 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 client.?.objecter ms_handle_connect 0x7f7acc09bca0 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ac00b0040 con 0x7f7acc09bca0 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: dump: 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac00044b0 con 0x7f7acc09bca0 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 5 monclient: authenticate success, global_id 14262 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: _renew_subs 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc1404f0 con 0x7f7acc09bca0 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ac0003a90 con 0x7f7acc09bca0 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ab37fe640 10 monclient: dump: 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 client.14262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: _renew_subs 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ac97fa640 4 set_mon_vals no callback set 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc092c90 con 0x7f7acc09bca0 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.851+0000 7f7ad1476640 1 librados: init done 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ad1476640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ad1476640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13b780 0x7f7acc13bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ad1476640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7acc06ca10 con 0x7f7acc13b780 2024-09-11T02:52:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13b780 0x7f7acc13bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient: get_auth_request con 0x7f7acc13b780 auth_method 0 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f7ac00b3070 con 0x7f7acc09bca0 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 10 monclient(hunting): handle_auth_done global_id 14271 payload 824 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 1 -- 172.21.15.98:0/1641280602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7acc0745d0 con 0x7f7acc13b780 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7ab803df40 0x7f7ab8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(87..87 src has 1..87) ==== 9323+0+0 (secure 0 0 0) 0x7f7ac00e9e60 con 0x7f7acc09bca0 2024-09-11T02:52:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acaffd640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13b780 0x7f7acc13bb60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7abc007fd0 tx=0x7f7abc001d70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 client.14262.objecter ms_dispatch 0x7f7acc000ba0 osd_map(87..87 src has 1..87) 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 3 client.14262.objecter handle_osd_map got epochs [87,87] > 0 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 3 client.14262.objecter handle_osd_map decoding full epoch 87 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 20 client.14262.objecter dump_active .. 0 homeless 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 client.14262.objecter ms_handle_connect 0x7f7acc13b780 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 client.14262.objecter resend_mon_ops 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7abc00fe10 con 0x7f7acc13b780 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 monclient: dump: 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.880 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acb7fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7ab803df40 0x7f7ab8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acb7fe640 10 monclient: get_auth_request con 0x7f7ab803df40 auth_method 0 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7abc018070 con 0x7f7acc13b780 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13b780 msgr2=0x7f7acc13bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13b780 0x7f7acc13bb60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7abc007fd0 tx=0x7f7abc001d70 comp rx=0 tx=0).stop 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13b780 msgr2=0x7f7acc13bb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13b780 0x7f7acc13bb60 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7abc007fd0 tx=0x7f7abc001d70 comp rx=0 tx=0).stop 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7acb7fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7ab803df40 0x7f7ab8040400 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f7acc092b90 tx=0x7f7ac40074a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.855+0000 7f7ab37fe640 10 client.14262.objecter ms_handle_connect 0x7f7ab803df40 2024-09-11T02:52:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ad1476640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ad1476640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13c0a0 0x7f7acc071800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ad1476640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f7acc06e870 con 0x7f7acc13c0a0 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13c0a0 0x7f7acc071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient: get_auth_request con 0x7f7acc13c0a0 auth_method 0 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 10 monclient(hunting): handle_auth_done global_id 14280 payload 824 2024-09-11T02:52:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 1 -- 172.21.15.98:0/1641280602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7acc09df80 con 0x7f7acc13c0a0 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ab37fe640 10 client.14262.objecter ms_handle_connect 0x7f7acc13c0a0 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ab37fe640 10 client.14262.objecter resend_mon_ops 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7acaffd640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13c0a0 0x7f7acc071800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7abc007fd0 tx=0x7f7abc000ee0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7abc013a70 con 0x7f7acc13c0a0 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ab37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ab37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.859+0000 7f7ab37fe640 10 monclient: dump: 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f7abc013600 con 0x7f7acc13c0a0 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ab37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ab37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13c0a0 msgr2=0x7f7acc071800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ab37fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13c0a0 0x7f7acc071800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7abc007fd0 tx=0x7f7abc000ee0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ab37fe640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13c0a0 msgr2=0x7f7acc071800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ab37fe640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc13c0a0 0x7f7acc071800 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7abc007fd0 tx=0x7f7abc000ee0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:12.885 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 12911, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:12.886 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.888 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.889 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:12.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.891 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.894 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:12.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:12.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.901 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:12.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:12.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:12.908 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 10 librados: watch_flush enter 2024-09-11T02:52:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 10 librados: watch_flush exit 2024-09-11T02:52:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 20 client.14262.objecter shutdown clearing up homeless session... 2024-09-11T02:52:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 10 client.14262.objecter successfully canceled tick 2024-09-11T02:52:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7ab803df40 msgr2=0x7f7ab8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7ab803df40 0x7f7ab8040400 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f7acc092b90 tx=0x7f7ac40074a0 comp rx=0 tx=0).stop 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 10 monclient: shutdown 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc09bca0 msgr2=0x7f7acc13aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 --2- 172.21.15.98:0/1641280602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f7acc09bca0 0x7f7acc13aaa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0019f90 tx=0x7f7ac0019100 comp rx=0 tx=0).stop 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7acb7fe640 1 -- 172.21.15.98:0/1641280602 reap_dead start 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7acb7fe640 1 -- 172.21.15.98:0/1641280602 reap_dead start 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 shutdown_connections 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 >> 172.21.15.98:0/1641280602 conn(0x7f7acc0a4ed0 msgr2=0x7f7acc0a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 shutdown_connections 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 -- 172.21.15.98:0/1641280602 wait complete. 2024-09-11T02:52:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:12.863+0000 7f7ad1476640 1 librados: shutdown 2024-09-11T02:52:12.911 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:12.912 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:52:12.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:52:13.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.015+0000 7f6e538c6640 10 monclient: get_monmap_and_config 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.015+0000 7f6e538c6640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient: monmap: 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:13.021225+0000 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:13.021225+0000 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.020 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:13.021 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 Processor -- start 2024-09-11T02:52:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 -- start start 2024-09-11T02:52:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient: init 2024-09-11T02:52:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient: _add_conns ranks=[10,5,6,2,7,8,0,9,4,1,3] 2024-09-11T02:52:13.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6e4c0a58c0 0x7f6e4c0a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): picked mon.noname-f con 0x7f6e4c0a58c0 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6e4c06f9d0 0x7f6e4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): picked mon.noname-h con 0x7f6e4c06f9d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6e4c139690 0x7f6e4c139b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): picked mon.noname-d con 0x7f6e4c139690 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6e4c0a6720 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f6e4c0703e0 con 0x7f6e4c06f9d0 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6e4c0706c0 con 0x7f6e4c139690 2024-09-11T02:52:13.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e538c6640 10 monclient(hunting): authenticate will time out at 1086.901413s 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6e4c139690 0x7f6e4c139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6e4c139690 0x7f6e4c139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58456/0 (socket says 172.21.15.98:58456) 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 -- 172.21.15.98:0/4168736996 learned_addr learned my addr 172.21.15.98:0/4168736996 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): get_auth_request con 0x7f6e4c139690 auth_method 0 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e50e61640 1 --2- 172.21.15.98:0/4168736996 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6e4c06f9d0 0x7f6e4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51662640 1 --2- 172.21.15.98:0/4168736996 >> v2:172.21.15.190:3300/0 conn(0x7f6e4c0a58c0 0x7f6e4c0a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e50e61640 10 monclient(hunting): get_auth_request con 0x7f6e4c06f9d0 auth_method 0 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e50e61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e50e61640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e50e61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51662640 10 monclient(hunting): get_auth_request con 0x7f6e4c0a58c0 auth_method 0 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51662640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient(hunting): handle_auth_done global_id 14289 payload 274 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 -- 172.21.15.98:0/4168736996 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6e4c06f9d0 msgr2=0x7f6e4c0a61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 --2- 172.21.15.98:0/4168736996 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6e4c06f9d0 0x7f6e4c0a61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 -- 172.21.15.98:0/4168736996 >> v2:172.21.15.190:3300/0 conn(0x7f6e4c0a58c0 msgr2=0x7f6e4c0a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 --2- 172.21.15.98:0/4168736996 >> v2:172.21.15.190:3300/0 conn(0x7f6e4c0a58c0 0x7f6e4c0a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 monclient: found mon.noname-d 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.019+0000 7f6e51e63640 1 -- 172.21.15.98:0/4168736996 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4c0709a0 con 0x7f6e4c139690 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 1 --2- 172.21.15.98:0/4168736996 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6e4c139690 0x7f6e4c139b10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009e80 tx=0x7f6e4401c8d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b384386d0d1adbd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 1 -- 172.21.15.98:0/4168736996 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e440090d0 con 0x7f6e4c139690 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: dump: 2024-09-11T02:52:13.029 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.030 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 1 -- 172.21.15.98:0/4168736996 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e440a5a70 con 0x7f6e4c139690 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 5 monclient: authenticate success, global_id 14289 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 1 -- 172.21.15.98:0/4168736996 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e440a5c60 con 0x7f6e4c139690 2024-09-11T02:52:13.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 4 set_mon_vals no callback set 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e43fff640 10 monclient: dump: 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.032 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: shutdown 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/4168736996 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6e4c139690 msgr2=0x7f6e4c139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- 172.21.15.98:0/4168736996 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6e4c139690 0x7f6e4c139b10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e44009e80 tx=0x7f6e4401c8d0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/4168736996 shutdown_connections 2024-09-11T02:52:13.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- 172.21.15.98:0/4168736996 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6e4c139690 0x7f6e4c139b10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- 172.21.15.98:0/4168736996 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6e4c06f9d0 0x7f6e4c0a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- 172.21.15.98:0/4168736996 >> v2:172.21.15.190:3300/0 conn(0x7f6e4c0a58c0 0x7f6e4c0a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/4168736996 >> 172.21.15.98:0/4168736996 conn(0x7f6e4c0aa4b0 msgr2=0x7f6e4c0a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/4168736996 shutdown_connections 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/4168736996 wait complete. 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: monmap: 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.034 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 librados: starting msgr at 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 librados: starting objecter 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 Processor -- start 2024-09-11T02:52:13.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- start start 2024-09-11T02:52:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 librados: setting wanted keys 2024-09-11T02:52:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 librados: calling monclient init 2024-09-11T02:52:13.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: init 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: _add_conns ranks=[2,0,8,4,3,6,7,1,5] 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e4c06f9d0 0x7f6e4c1d12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4c06f9d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0a58c0 0x7f6e4c138690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): picked mon.noname-a con 0x7f6e4c0a58c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c139690 0x7f6e4c120500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): picked mon.noname-e con 0x7f6e4c139690 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4c06ef60 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f6e4c06ede0 con 0x7f6e4c06f9d0 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6e4c143310 con 0x7f6e4c139690 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): authenticate will time out at 1086.906291s 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0a58c0 0x7f6e4c138690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e4c06f9d0 0x7f6e4c1d12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c139690 0x7f6e4c120500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0a58c0 0x7f6e4c138690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59590/0 (socket says 172.21.15.98:59590) 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 -- 172.21.15.98:0/2022223237 learned_addr learned my addr 172.21.15.98:0/2022223237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): get_auth_request con 0x7f6e4c0a58c0 auth_method 0 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient(hunting): get_auth_request con 0x7f6e4c06f9d0 auth_method 0 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 10 monclient(hunting): get_auth_request con 0x7f6e4c139690 auth_method 0 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient(hunting): handle_auth_done global_id 14328 payload 824 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c139690 msgr2=0x7f6e4c120500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c139690 0x7f6e4c120500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e4c06f9d0 msgr2=0x7f6e4c1d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e4c06f9d0 0x7f6e4c1d12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 monclient: found mon.noname-a 2024-09-11T02:52:13.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 -- 172.21.15.98:0/2022223237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4c124030 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6e4c06f9d0 0x7f6e4c1d12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51e63640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c139690 0x7f6e4c120500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e50e61640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0a58c0 0x7f6e4c138690 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00d800 tx=0x7f6e3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 client.?.objecter ms_handle_connect 0x7f6e4c0a58c0 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e3c00bb30 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: dump: 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.047 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 5 monclient: authenticate success, global_id 14328 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3c019070 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: _renew_subs 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e3c00bd10 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4c0a63f0 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e437fe640 4 set_mon_vals no callback set 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 client.14328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e41ffb640 10 monclient: dump: 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.049 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.050 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: _renew_subs 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4c123d50 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 librados: init done 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c138bd0 0x7f6e4c120a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6e4c06ca10 con 0x7f6e4c138bd0 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c138bd0 0x7f6e4c120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.023+0000 7f6e51662640 10 monclient: get_auth_request con 0x7f6e4c138bd0 auth_method 0 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6e3c013070 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 10 monclient(hunting): handle_auth_done global_id 23471 payload 824 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 1 -- 172.21.15.98:0/2022223237 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e4c13faa0 con 0x7f6e4c138bd0 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6e3003df40 0x7f6e30040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9323+0+0 (secure 0 0 0) 0x7f6e3c00f040 con 0x7f6e4c0a58c0 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 client.14328.objecter ms_dispatch 0x7f6e4c000ba0 osd_map(87..87 src has 1..87) 2024-09-11T02:52:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 3 client.14328.objecter handle_osd_map got epochs [87,87] > 0 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 3 client.14328.objecter handle_osd_map decoding full epoch 87 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51662640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c138bd0 0x7f6e4c120a40 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6e48010b90 tx=0x7f6e4800e5a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 20 client.14328.objecter dump_active .. 0 homeless 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 client.14328.objecter ms_handle_connect 0x7f6e4c138bd0 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 client.14328.objecter resend_mon_ops 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e48018020 con 0x7f6e4c138bd0 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 monclient: dump: 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6e48010e90 con 0x7f6e4c138bd0 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51e63640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6e3003df40 0x7f6e30040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c138bd0 msgr2=0x7f6e4c120a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c138bd0 0x7f6e4c120a40 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6e48010b90 tx=0x7f6e4800e5a0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c138bd0 msgr2=0x7f6e4c120a40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c138bd0 0x7f6e4c120a40 secure :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6e48010b90 tx=0x7f6e4800e5a0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51e63640 10 monclient: get_auth_request con 0x7f6e3003df40 auth_method 0 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e41ffb640 10 client.14328.objecter ms_handle_connect 0x7f6e3003df40 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.027+0000 7f6e51e63640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6e3003df40 0x7f6e30040400 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f6e440ae000 tx=0x7f6e440a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e538c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e538c6640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c120f80 0x7f6e4c13b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e538c6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6e4c121390 con 0x7f6e4c120f80 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c120f80 0x7f6e4c13b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient: get_auth_request con 0x7f6e4c120f80 auth_method 0 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 10 monclient(hunting): handle_auth_done global_id 23480 payload 824 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 1 -- 172.21.15.98:0/2022223237 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e4c139b10 con 0x7f6e4c120f80 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e51662640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c120f80 0x7f6e4c13b8c0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6e48010b90 tx=0x7f6e480046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 10 client.14328.objecter ms_handle_connect 0x7f6e4c120f80 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 10 client.14328.objecter resend_mon_ops 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e48018040 con 0x7f6e4c120f80 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 10 monclient: dump: 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f6e48003830 con 0x7f6e4c120f80 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c120f80 msgr2=0x7f6e4c13b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c120f80 0x7f6e4c13b8c0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6e48010b90 tx=0x7f6e480046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c120f80 msgr2=0x7f6e4c13b8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.031+0000 7f6e41ffb640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6e4c120f80 0x7f6e4c13b8c0 secure :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f6e48010b90 tx=0x7f6e480046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 148823, 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:13.059 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:13.060 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.061 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:13.063 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.064 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:13.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:13.070 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.075 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.078 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.081 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.082 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:13.082 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 10 librados: watch_flush enter 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 10 librados: watch_flush exit 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 20 client.14328.objecter shutdown clearing up homeless session... 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 10 client.14328.objecter successfully canceled tick 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6e3003df40 msgr2=0x7f6e30040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6e3003df40 0x7f6e30040400 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7f6e440ae000 tx=0x7f6e440a7070 comp rx=0 tx=0).stop 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 10 monclient: shutdown 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0a58c0 msgr2=0x7f6e4c138690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 --2- 172.21.15.98:0/2022223237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e4c0a58c0 0x7f6e4c138690 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00d800 tx=0x7f6e3c00dcc0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e51e63640 1 -- 172.21.15.98:0/2022223237 reap_dead start 2024-09-11T02:52:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e51e63640 1 -- 172.21.15.98:0/2022223237 reap_dead start 2024-09-11T02:52:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 shutdown_connections 2024-09-11T02:52:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 >> 172.21.15.98:0/2022223237 conn(0x7f6e4c0aa4b0 msgr2=0x7f6e4c071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 shutdown_connections 2024-09-11T02:52:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 -- 172.21.15.98:0/2022223237 wait complete. 2024-09-11T02:52:13.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.035+0000 7f6e538c6640 1 librados: shutdown 2024-09-11T02:52:13.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:13.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:52:13.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.195+0000 7f53fde69640 10 monclient: get_monmap_and_config 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.195+0000 7f53fde69640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.195+0000 7f53fde69640 10 monclient: monmap: 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:13.200534+0000 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:13.200534+0000 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.199 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:13.200 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 1 Processor -- start 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 1 -- start start 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient: init 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient: _add_conns ranks=[10,4,0,5,9,3,6,1,7,2,8] 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f53f8139b10 0x7f53f8072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient(hunting): picked mon.noname-f con 0x7f53f8139b10 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53f8139ef0 0x7f53f80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient(hunting): picked mon.noname-c con 0x7f53f8139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53f813a2d0 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f80702b0 con 0x7f53f813a2d0 2024-09-11T02:52:13.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f53f8071bf0 con 0x7f53f8139b10 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f53f8071350 con 0x7f53f8139ef0 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient(hunting): authenticate will time out at 1087.080790s 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53f8139ef0 0x7f53f80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53f8139ef0 0x7f53f80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45924/0 (socket says 172.21.15.98:45924) 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 -- 172.21.15.98:0/1281744007 learned_addr learned my addr 172.21.15.98:0/1281744007 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): get_auth_request con 0x7f53f8139ef0 auth_method 0 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f77fe640 1 --2- 172.21.15.98:0/1281744007 >> v2:172.21.15.190:3300/0 conn(0x7f53f8139b10 0x7f53f8072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f77fe640 10 monclient(hunting): get_auth_request con 0x7f53f8139b10 auth_method 0 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_done global_id 23044 payload 274 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 -- 172.21.15.98:0/1281744007 >> v2:172.21.15.190:3300/0 conn(0x7f53f8139b10 msgr2=0x7f53f8072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1281744007 >> v2:172.21.15.190:3300/0 conn(0x7f53f8139b10 0x7f53f8072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 1 -- 172.21.15.98:0/1281744007 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554884891 0 0) 0x7f53f00014a0 con 0x7f53f813a2d0 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 -- 172.21.15.98:0/1281744007 >> v1:172.21.15.98:6789/0 conn(0x7f53f813a2d0 legacy=0x7f53f80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 monclient: found mon.noname-c 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 -- 172.21.15.98:0/1281744007 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f80a9620 con 0x7f53f8139ef0 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1281744007 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53f8139ef0 0x7f53f80a7a10 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009870 tx=0x7f53e801c4b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=859faa4e544f7927 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 1 -- 172.21.15.98:0/1281744007 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53e80b0070 con 0x7f53f8139ef0 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: dump: 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.209 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 1 -- 172.21.15.98:0/1281744007 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e80aa810 con 0x7f53f8139ef0 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 5 monclient: authenticate success, global_id 23044 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 1 -- 172.21.15.98:0/1281744007 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53e80aa9c0 con 0x7f53f8139ef0 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53fde69640 4 set_mon_vals no callback set 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:13.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.199+0000 7f53f67fc640 10 monclient: dump: 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.211 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: shutdown 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1281744007 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53f8139ef0 msgr2=0x7f53f80a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 --2- 172.21.15.98:0/1281744007 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53f8139ef0 0x7f53f80a7a10 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009870 tx=0x7f53e801c4b0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1281744007 shutdown_connections 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 --2- 172.21.15.98:0/1281744007 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f53f8139ef0 0x7f53f80a7a10 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 --2- 172.21.15.98:0/1281744007 >> v2:172.21.15.190:3300/0 conn(0x7f53f8139b10 0x7f53f8072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1281744007 >> 172.21.15.98:0/1281744007 conn(0x7f53f80ac790 msgr2=0x7f53f80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1281744007 shutdown_connections 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1281744007 wait complete. 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: monmap: 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.213 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 librados: starting msgr at 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 librados: starting objecter 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 Processor -- start 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- start start 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 librados: setting wanted keys 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 librados: calling monclient init 2024-09-11T02:52:13.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: init 2024-09-11T02:52:13.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: _add_conns ranks=[6,1,7,2,4,3,0,8,5] 2024-09-11T02:52:13.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f53f8139b10 0x7f53f81337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): picked mon.noname-d con 0x7f53f8139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f8139ef0 0x7f53f8133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): picked mon.noname-f con 0x7f53f8139ef0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53f813a2d0 0x7f53f8134270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): picked mon.noname-i con 0x7f53f813a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53f8144470 con 0x7f53f8139ef0 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f53f80a8870 con 0x7f53f8139b10 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f53f81445f0 con 0x7f53f813a2d0 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): authenticate will time out at 1087.085823s 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f53f8139b10 0x7f53f81337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f53f8139b10 0x7f53f81337f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58464/0 (socket says 172.21.15.98:58464) 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 -- 172.21.15.98:0/1819966640 learned_addr learned my addr 172.21.15.98:0/1819966640 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): get_auth_request con 0x7f53f8139b10 auth_method 0 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f7fff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53f813a2d0 0x7f53f8134270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f8139ef0 0x7f53f8133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f7fff640 10 monclient(hunting): get_auth_request con 0x7f53f813a2d0 auth_method 0 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient(hunting): handle_auth_done global_id 14298 payload 824 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53f813a2d0 msgr2=0x7f53f8134270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f53f813a2d0 0x7f53f8134270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f8139ef0 msgr2=0x7f53f8133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f8139ef0 0x7f53f8133d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 monclient: found mon.noname-d 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 -- 172.21.15.98:0/1819966640 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f813ce90 con 0x7f53f8139b10 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient: get_auth_request con 0x7f53f8139ef0 auth_method 0 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f8139ef0 0x7f53f8133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f77fe640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f53f8139b10 0x7f53f81337f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d800 tx=0x7f53ec00dcc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 client.?.objecter ms_handle_connect 0x7f53f8139b10 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53ec00bb30 con 0x7f53f8139b10 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: dump: 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53ec019070 con 0x7f53f8139b10 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 5 monclient: authenticate success, global_id 14298 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: _renew_subs 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f813d170 con 0x7f53f8139b10 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53ec00bcd0 con 0x7f53f8139b10 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 client.14298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f5ffb640 4 set_mon_vals no callback set 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 monclient: dump: 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.228 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: _renew_subs 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f813c610 con 0x7f53f8139b10 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 librados: init done 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a0e60 0x7f53f80a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53f806ca10 con 0x7f53f80a0e60 2024-09-11T02:52:13.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f53ec013070 con 0x7f53f8139b10 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a0e60 0x7f53f80a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient: get_auth_request con 0x7f53f80a0e60 auth_method 0 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53e403df40 0x7f53e4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(87..87 src has 1..87) ==== 9323+0+0 (secure 0 0 0) 0x7f53ec00f040 con 0x7f53f8139b10 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 10 client.14298.objecter ms_dispatch 0x7f53f8000ba0 osd_map(87..87 src has 1..87) 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 3 client.14298.objecter handle_osd_map got epochs [87,87] > 0 2024-09-11T02:52:13.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.203+0000 7f53dffff640 3 client.14298.objecter handle_osd_map decoding full epoch 87 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 20 client.14298.objecter dump_active .. 0 homeless 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53f7fff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53e403df40 0x7f53e4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53f7fff640 10 monclient: get_auth_request con 0x7f53e403df40 auth_method 0 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_done global_id 24148 payload 824 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53f6ffd640 1 -- 172.21.15.98:0/1819966640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53f806e680 con 0x7f53f80a0e60 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 client.14298.objecter ms_handle_connect 0x7f53f80a0e60 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 client.14298.objecter resend_mon_ops 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a0e60 0x7f53f80a1240 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f53f80a9e90 tx=0x7f53e8019250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 client.14298.objecter ms_handle_connect 0x7f53e403df40 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53f7fff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53e403df40 0x7f53e4040400 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f53f80a5b50 tx=0x7f53f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53e80ba070 con 0x7f53f80a0e60 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:52:13.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 monclient: dump: 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.232 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f53e8003620 con 0x7f53f80a0e60 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a0e60 msgr2=0x7f53f80a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a0e60 0x7f53f80a1240 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f53f80a9e90 tx=0x7f53e8019250 comp rx=0 tx=0).stop 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a0e60 msgr2=0x7f53f80a1240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.207+0000 7f53dffff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a0e60 0x7f53f80a1240 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.211+0000 7f53fde69640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.211+0000 7f53fde69640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a1780 0x7f53f813a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.211+0000 7f53fde69640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.211+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53f806e870 con 0x7f53f80a1780 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a1780 0x7f53f813a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient: get_auth_request con 0x7f53f80a1780 auth_method 0 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 10 monclient(hunting): handle_auth_done global_id 24157 payload 824 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 1 -- 172.21.15.98:0/1819966640 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53f8072600 con 0x7f53f80a1780 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f6ffd640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a1780 0x7f53f813a780 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f53e801a0e0 tx=0x7f53e80a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 10 client.14298.objecter ms_handle_connect 0x7f53f80a1780 2024-09-11T02:52:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 10 client.14298.objecter resend_mon_ops 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53e80ba040 con 0x7f53f80a1780 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 10 monclient: dump: 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7f53e80b6750 con 0x7f53f80a1780 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a1780 msgr2=0x7f53f813a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a1780 0x7f53f813a780 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f53e801a0e0 tx=0x7f53e80a8000 comp rx=0 tx=0).stop 2024-09-11T02:52:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a1780 msgr2=0x7f53f813a780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53dffff640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53f80a1780 0x7f53f813a780 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 148984, 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:13.237 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:13.238 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:13.239 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:13.241 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.242 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.244 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:13.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:13.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.253 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.258 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.259 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.260 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.261 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:13.261 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 10 librados: watch_flush enter 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 10 librados: watch_flush exit 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 20 client.14298.objecter shutdown clearing up homeless session... 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 10 client.14298.objecter successfully canceled tick 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53e403df40 msgr2=0x7f53e4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f53e403df40 0x7f53e4040400 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f53f80a5b50 tx=0x7f53f0009040 comp rx=0 tx=0).stop 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 10 monclient: shutdown 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f53f8139b10 msgr2=0x7f53f81337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 --2- 172.21.15.98:0/1819966640 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f53f8139b10 0x7f53f81337f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d800 tx=0x7f53ec00dcc0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f7fff640 1 -- 172.21.15.98:0/1819966640 reap_dead start 2024-09-11T02:52:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53f7fff640 1 -- 172.21.15.98:0/1819966640 reap_dead start 2024-09-11T02:52:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 shutdown_connections 2024-09-11T02:52:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 >> 172.21.15.98:0/1819966640 conn(0x7f53f80ac790 msgr2=0x7f53f80a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 shutdown_connections 2024-09-11T02:52:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 -- 172.21.15.98:0/1819966640 wait complete. 2024-09-11T02:52:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.215+0000 7f53fde69640 1 librados: shutdown 2024-09-11T02:52:13.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:13.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:52:13.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient: get_monmap_and_config 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient: monmap: 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:13.403464+0000 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:13.403464+0000 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.402 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:13.403 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 Processor -- start 2024-09-11T02:52:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 -- start start 2024-09-11T02:52:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient: init 2024-09-11T02:52:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient: _add_conns ranks=[7,9,4,2,1,0,6,10,5,8,3] 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1c0a5830 0x7f0d1c0a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): picked mon.noname-i con 0x7f0d1c0a5830 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d1c0a1770 0x7f0d1c0a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): picked mon.noname-a con 0x7f0d1c0a1770 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c1375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): picked mon.noname-c con 0x7f0d1c071ed0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0d1c0a6690 con 0x7f0d1c0a1770 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0d1c0739b0 con 0x7f0d1c071ed0 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0d1c071c90 con 0x7f0d1c0a5830 2024-09-11T02:52:13.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d24118640 10 monclient(hunting): authenticate will time out at 1087.283697s 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d1c0a1770 0x7f0d1c0a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d1c0a1770 0x7f0d1c0a6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59606/0 (socket says 172.21.15.98:59606) 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 1 -- 172.21.15.98:0/2099901314 learned_addr learned my addr 172.21.15.98:0/2099901314 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 10 monclient(hunting): get_auth_request con 0x7f0d1c0a1770 auth_method 0 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d226b5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c1375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d226b5640 10 monclient(hunting): get_auth_request con 0x7f0d1c071ed0 auth_method 0 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d226b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d226b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d226b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.399+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d226b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d226b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d226b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 10 monclient(hunting): handle_auth_done global_id 14346 payload 274 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 1 -- 172.21.15.98:0/2099901314 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1c0a5830 msgr2=0x7f0d1c0a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1c0a5830 0x7f0d1c0a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 1 -- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 msgr2=0x7f0d1c1375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c1375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 1 monclient: found mon.noname-a 2024-09-11T02:52:13.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:52:13.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 1 -- 172.21.15.98:0/2099901314 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1c073c90 con 0x7f0d1c0a1770 2024-09-11T02:52:13.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d226b5640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c1375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d216b3640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0a1770 0x7f0d1c0a6150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0098d0 tx=0x7f0d0c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1316e7febd56e1f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 1 -- 172.21.15.98:0/2099901314 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d0c0b0070 con 0x7f0d1c0a1770 2024-09-11T02:52:13.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: dump: 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.417 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 1 -- 172.21.15.98:0/2099901314 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d0c0aa810 con 0x7f0d1c0a1770 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 1 -- 172.21.15.98:0/2099901314 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d0c0a9300 con 0x7f0d1c0a1770 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d20eb2640 10 monclient: dump: 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.419 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 5 monclient: authenticate success, global_id 14346 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 4 set_mon_vals no callback set 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 10 monclient: shutdown 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 -- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0a1770 msgr2=0x7f0d1c0a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0a1770 0x7f0d1c0a6150 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0098d0 tx=0x7f0d0c01cba0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 -- 172.21.15.98:0/2099901314 shutdown_connections 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c1375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d1c0a1770 0x7f0d1c0a6150 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 --2- 172.21.15.98:0/2099901314 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1c0a5830 0x7f0d1c0a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 -- 172.21.15.98:0/2099901314 >> 172.21.15.98:0/2099901314 conn(0x7f0d1c0aa420 msgr2=0x7f0d1c0a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 -- 172.21.15.98:0/2099901314 shutdown_connections 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.403+0000 7f0d24118640 1 -- 172.21.15.98:0/2099901314 wait complete. 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: monmap: 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:13.421 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.422 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 librados: starting msgr at 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 librados: starting objecter 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 Processor -- start 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 -- start start 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 librados: setting wanted keys 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 librados: calling monclient init 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: init 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: _add_conns ranks=[4,6,3,1,7,2,0,8,5] 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c134f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): picked mon.noname-c con 0x7f0d1c071ed0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:13.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d1c0a5830 0x7f0d1c13ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): picked mon.noname-d con 0x7f0d1c0a5830 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06ed00 0x7f0d1c06f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): picked mon.noname-g con 0x7f0d1c06ed00 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1c140de0 con 0x7f0d1c06ed00 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0d1c140c60 con 0x7f0d1c071ed0 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0d1c140f60 con 0x7f0d1c0a5830 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): authenticate will time out at 1087.289152s 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d1c0a5830 0x7f0d1c13ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c134f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45946/0 (socket says 172.21.15.98:45946) 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 -- 172.21.15.98:0/1307095582 learned_addr learned my addr 172.21.15.98:0/1307095582 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): get_auth_request con 0x7f0d1c071ed0 auth_method 0 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): get_auth_request con 0x7f0d1c0a5830 auth_method 0 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d226b5640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06ed00 0x7f0d1c06f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d226b5640 10 monclient(hunting): get_auth_request con 0x7f0d1c06ed00 auth_method 0 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d226b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d226b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d226b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient(hunting): handle_auth_done global_id 23062 payload 824 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d1c0a5830 msgr2=0x7f0d1c13ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d1c0a5830 0x7f0d1c13ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06ed00 msgr2=0x7f0d1c06f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06ed00 0x7f0d1c06f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 monclient: found mon.noname-c 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 -- 172.21.15.98:0/1307095582 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1c13d510 con 0x7f0d1c071ed0 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d226b5640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06ed00 0x7f0d1c06f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d1c0a5830 0x7f0d1c13ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d21eb4640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c134f50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800b410 tx=0x7f0d1800b8d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 client.?.objecter ms_handle_connect 0x7f0d1c071ed0 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d18015020 con 0x7f0d1c071ed0 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:13.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: dump: 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d18011aa0 con 0x7f0d1c071ed0 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 5 monclient: authenticate success, global_id 23062 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d18011c90 con 0x7f0d1c071ed0 2024-09-11T02:52:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d13fff640 4 set_mon_vals no callback set 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 monclient: dump: 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: _renew_subs 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1c0a2d00 con 0x7f0d1c071ed0 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 client.23062.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: _renew_subs 2024-09-11T02:52:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d1c13b370 con 0x7f0d1c071ed0 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 librados: init done 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c0a1770 0x7f0d1c02e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1c06ca10 con 0x7f0d1c0a1770 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0d18011630 con 0x7f0d1c071ed0 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c0a1770 0x7f0d1c02e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d0003df40 0x7f0d00040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient: get_auth_request con 0x7f0d1c0a1770 auth_method 0 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(88..88 src has 1..88) ==== 8927+0+0 (secure 0 0 0) 0x7f0d18050380 con 0x7f0d1c071ed0 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 10 client.23062.objecter ms_dispatch 0x7f0d1c000ba0 osd_map(88..88 src has 1..88) 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 3 client.23062.objecter handle_osd_map got epochs [88,88] > 0 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 3 client.23062.objecter handle_osd_map decoding full epoch 88 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d127fc640 20 client.23062.objecter dump_active .. 0 homeless 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d226b5640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d0003df40 0x7f0d00040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.407+0000 7f0d216b3640 10 monclient(hunting): handle_auth_done global_id 19506 payload 824 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d216b3640 1 -- 172.21.15.98:0/1307095582 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c13dd80 con 0x7f0d1c0a1770 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d226b5640 10 monclient: get_auth_request con 0x7f0d0003df40 auth_method 0 2024-09-11T02:52:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d216b3640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c0a1770 0x7f0d1c02e5d0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0a72c0 tx=0x7f0d0c018ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 client.23062.objecter ms_handle_connect 0x7f0d1c0a1770 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 client.23062.objecter resend_mon_ops 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d0c0ba030 con 0x7f0d1c0a1770 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 monclient: dump: 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 client.23062.objecter ms_handle_connect 0x7f0d0003df40 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d226b5640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d0003df40 0x7f0d00040400 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007520 tx=0x7f0d1400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d0c0aa750 con 0x7f0d1c0a1770 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c0a1770 msgr2=0x7f0d1c02e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c0a1770 0x7f0d1c02e5d0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0a72c0 tx=0x7f0d0c018ee0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c0a1770 msgr2=0x7f0d1c02e5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.411+0000 7f0d127fc640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c0a1770 0x7f0d1c02e5d0 secure :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c0a72c0 tx=0x7f0d0c018ee0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d24118640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d24118640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06f680 0x7f0d1c063e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d24118640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1c06e870 con 0x7f0d1c06f680 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06f680 0x7f0d1c063e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient: get_auth_request con 0x7f0d1c06f680 auth_method 0 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 10 monclient(hunting): handle_auth_done global_id 19515 payload 824 2024-09-11T02:52:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 1 -- 172.21.15.98:0/1307095582 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d1c0a3250 con 0x7f0d1c06f680 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d127fc640 10 client.23062.objecter ms_handle_connect 0x7f0d1c06f680 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d127fc640 10 client.23062.objecter resend_mon_ops 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.415+0000 7f0d216b3640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06f680 0x7f0d1c063e00 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0097c0 tx=0x7f0d0c0a6000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d0c0ba040 con 0x7f0d1c06f680 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 10 monclient: dump: 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f0d0c003620 con 0x7f0d1c06f680 2024-09-11T02:52:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06f680 msgr2=0x7f0d1c063e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06f680 0x7f0d1c063e00 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0097c0 tx=0x7f0d0c0a6000 comp rx=0 tx=0).stop 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06f680 msgr2=0x7f0d1c063e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d127fc640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0d1c06f680 0x7f0d1c063e00 secure :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0097c0 tx=0x7f0d0c0a6000 comp rx=0 tx=0).stop 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:13.441 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 149164, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:13.442 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.443 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.444 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:13.445 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.446 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:13.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.454 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:13.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.461 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:13.465 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 10 librados: watch_flush enter 2024-09-11T02:52:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 10 librados: watch_flush exit 2024-09-11T02:52:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 20 client.23062.objecter shutdown clearing up homeless session... 2024-09-11T02:52:13.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 10 client.23062.objecter successfully canceled tick 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d0003df40 msgr2=0x7f0d00040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d0003df40 0x7f0d00040400 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f0d14007520 tx=0x7f0d1400b040 comp rx=0 tx=0).stop 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 10 monclient: shutdown 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 msgr2=0x7f0d1c134f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 --2- 172.21.15.98:0/1307095582 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d1c071ed0 0x7f0d1c134f50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f0d1800b410 tx=0x7f0d1800b8d0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d226b5640 1 -- 172.21.15.98:0/1307095582 reap_dead start 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d226b5640 1 -- 172.21.15.98:0/1307095582 reap_dead start 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 shutdown_connections 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 >> 172.21.15.98:0/1307095582 conn(0x7f0d1c0aa420 msgr2=0x7f0d1c0a8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 shutdown_connections 2024-09-11T02:52:13.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 -- 172.21.15.98:0/1307095582 wait complete. 2024-09-11T02:52:13.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.419+0000 7f0d24118640 1 librados: shutdown 2024-09-11T02:52:13.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:13.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:52:13.469 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient: get_monmap_and_config 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient: monmap: 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:13.589432+0000 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:13.589432+0000 2024-09-11T02:52:13.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:13.589 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 Processor -- start 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 -- start start 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient: init 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient: _add_conns ranks=[5,2,10,6,1,8,0,7,9,4,3] 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a5830 0x7f34300a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): picked mon.noname-h con 0x7f34300a5830 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a1770 0x7f34300a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): picked mon.noname-b con 0x7f34300a1770 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3430071ed0 0x7f34301375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): picked mon.noname-f con 0x7f3430071ed0 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:13.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f34300a6690 con 0x7f3430071ed0 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f34300739b0 con 0x7f34300a1770 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3430071c90 con 0x7f34300a5830 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3437a59640 10 monclient(hunting): authenticate will time out at 1087.469674s 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a1770 0x7f34300a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a1770 0x7f34300a6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56640/0 (socket says 172.21.15.98:56640) 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 -- 172.21.15.98:0/4125275138 learned_addr learned my addr 172.21.15.98:0/4125275138 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): get_auth_request con 0x7f34300a1770 auth_method 0 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f34357f5640 1 --2- 172.21.15.98:0/4125275138 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a5830 0x7f34300a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f34357f5640 10 monclient(hunting): get_auth_request con 0x7f34300a5830 auth_method 0 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f34357f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f34357f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f34357f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3435ff6640 1 --2- 172.21.15.98:0/4125275138 >> v2:172.21.15.190:3300/0 conn(0x7f3430071ed0 0x7f34301375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3435ff6640 10 monclient(hunting): get_auth_request con 0x7f3430071ed0 auth_method 0 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3435ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3435ff6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3435ff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient(hunting): handle_auth_done global_id 23078 payload 274 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 -- 172.21.15.98:0/4125275138 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a5830 msgr2=0x7f34300a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 --2- 172.21.15.98:0/4125275138 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a5830 0x7f34300a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 -- 172.21.15.98:0/4125275138 >> v2:172.21.15.190:3300/0 conn(0x7f3430071ed0 msgr2=0x7f34301375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 --2- 172.21.15.98:0/4125275138 >> v2:172.21.15.190:3300/0 conn(0x7f3430071ed0 0x7f34301375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 monclient: found mon.noname-b 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.587+0000 7f3434ff4640 1 -- 172.21.15.98:0/4125275138 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3430073c90 con 0x7f34300a1770 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 --2- 172.21.15.98:0/4125275138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a1770 0x7f34300a6150 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342401cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcebf4819a56a6ae server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 1 -- 172.21.15.98:0/4125275138 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34240b0070 con 0x7f34300a1770 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: dump: 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.598 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 1 -- 172.21.15.98:0/4125275138 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34240aa810 con 0x7f34300a1770 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 5 monclient: authenticate success, global_id 23078 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 1 -- 172.21.15.98:0/4125275138 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34240a9300 con 0x7f34300a1770 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342ffff640 10 monclient: dump: 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 4 set_mon_vals no callback set 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: shutdown 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- 172.21.15.98:0/4125275138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a1770 msgr2=0x7f34300a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 --2- 172.21.15.98:0/4125275138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a1770 0x7f34300a6150 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f3424009870 tx=0x7f342401cbc0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- 172.21.15.98:0/4125275138 shutdown_connections 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 --2- 172.21.15.98:0/4125275138 >> v2:172.21.15.190:3300/0 conn(0x7f3430071ed0 0x7f34301375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 --2- 172.21.15.98:0/4125275138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a1770 0x7f34300a6150 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 --2- 172.21.15.98:0/4125275138 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a5830 0x7f34300a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- 172.21.15.98:0/4125275138 >> 172.21.15.98:0/4125275138 conn(0x7f34300aa420 msgr2=0x7f34300a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- 172.21.15.98:0/4125275138 shutdown_connections 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- 172.21.15.98:0/4125275138 wait complete. 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: monmap: 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:13.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 librados: starting msgr at 2024-09-11T02:52:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 librados: starting objecter 2024-09-11T02:52:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 Processor -- start 2024-09-11T02:52:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- start start 2024-09-11T02:52:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 librados: setting wanted keys 2024-09-11T02:52:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 librados: calling monclient init 2024-09-11T02:52:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: init 2024-09-11T02:52:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: _add_conns ranks=[0,2,6,8,1,5,4,7,3] 2024-09-11T02:52:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430071ed0 0x7f34301ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): picked mon.noname-a con 0x7f3430071ed0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a5830 0x7f3430136510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): picked mon.noname-b con 0x7f34300a5830 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3430136a50 0x7f343013b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): picked mon.noname-d con 0x7f3430136a50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3430140de0 con 0x7f3430071ed0 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3430140c60 con 0x7f34300a5830 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3430140f60 con 0x7f3430136a50 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient(hunting): authenticate will time out at 1087.474933s 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a5830 0x7f3430136510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430071ed0 0x7f34301ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3430136a50 0x7f343013b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a5830 0x7f3430136510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:56652/0 (socket says 172.21.15.98:56652) 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 -- 172.21.15.98:0/982624802 learned_addr learned my addr 172.21.15.98:0/982624802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): get_auth_request con 0x7f34300a5830 auth_method 0 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 10 monclient(hunting): get_auth_request con 0x7f3430071ed0 auth_method 0 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 10 monclient(hunting): get_auth_request con 0x7f3430136a50 auth_method 0 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient(hunting): handle_auth_done global_id 23087 payload 824 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3430136a50 msgr2=0x7f343013b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3430136a50 0x7f343013b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430071ed0 msgr2=0x7f34301ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430071ed0 0x7f34301ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 monclient: found mon.noname-b 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 -- 172.21.15.98:0/982624802 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34300a2c90 con 0x7f34300a5830 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3435ff6640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3430136a50 0x7f343013b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f34357f5640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3430071ed0 0x7f34301ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3434ff4640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a5830 0x7f3430136510 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f34240097c0 tx=0x7f3424018f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 client.?.objecter ms_handle_connect 0x7f34300a5830 2024-09-11T02:52:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34240b9070 con 0x7f34300a5830 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: dump: 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34240aae20 con 0x7f34300a5830 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 5 monclient: authenticate success, global_id 23087 2024-09-11T02:52:13.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: _renew_subs 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343013d240 con 0x7f34300a5830 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34240b0040 con 0x7f34300a5830 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342dffb640 10 monclient: dump: 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f342f7fe640 4 set_mon_vals no callback set 2024-09-11T02:52:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 client.23087.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: _renew_subs 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f343013d920 con 0x7f34300a5830 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.591+0000 7f3437a59640 1 librados: init done 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f3437a59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f3437a59640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a1750 0x7f343013bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f343006ca10 con 0x7f34300a1750 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3424004040 con 0x7f34300a5830 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a1750 0x7f343013bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f341c03df60 0x7f341c040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient: get_auth_request con 0x7f34300a1750 auth_method 0 2024-09-11T02:52:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 8927+0+0 (secure 0 0 0) 0x7f34240e9720 con 0x7f34300a5830 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 client.23087.objecter ms_dispatch 0x7f3430000ba0 osd_map(88..88 src has 1..88) 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 3 client.23087.objecter handle_osd_map got epochs [88,88] > 0 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 3 client.23087.objecter handle_osd_map decoding full epoch 88 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 20 client.23087.objecter dump_active .. 0 homeless 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f3435ff6640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f341c03df60 0x7f341c040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f3435ff6640 10 monclient: get_auth_request con 0x7f341c03df60 auth_method 0 2024-09-11T02:52:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 10 monclient(hunting): handle_auth_done global_id 19193 payload 824 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 1 -- 172.21.15.98:0/982624802 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3430136e30 con 0x7f34300a1750 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 client.23087.objecter ms_handle_connect 0x7f34300a1750 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 client.23087.objecter resend_mon_ops 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f34357f5640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a1750 0x7f343013bc90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3428004980 tx=0x7f34280077c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3428014a00 con 0x7f34300a1750 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 monclient: dump: 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 client.23087.objecter ms_handle_connect 0x7f341c03df60 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f3435ff6640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f341c03df60 0x7f341c040420 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7f3420006aa0 tx=0x7f3420008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f342801a070 con 0x7f34300a1750 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a1750 msgr2=0x7f343013bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a1750 0x7f343013bc90 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3428004980 tx=0x7f34280077c0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a1750 msgr2=0x7f343013bc90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.595+0000 7f342dffb640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f34300a1750 0x7f343013bc90 secure :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f3428004980 tx=0x7f34280077c0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f343013c1d0 0x7f343009ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f343006e870 con 0x7f343013c1d0 2024-09-11T02:52:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f343013c1d0 0x7f343009ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient: get_auth_request con 0x7f343013c1d0 auth_method 0 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 10 monclient(hunting): handle_auth_done global_id 19202 payload 824 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 1 -- 172.21.15.98:0/982624802 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f343013c5b0 con 0x7f343013c1d0 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 10 client.23087.objecter ms_handle_connect 0x7f343013c1d0 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 10 client.23087.objecter resend_mon_ops 2024-09-11T02:52:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f34357f5640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f343013c1d0 0x7f343009ea60 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f3428000f90 tx=0x7f3428004a50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3428014d60 con 0x7f343013c1d0 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 10 monclient: dump: 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f34280148f0 con 0x7f343013c1d0 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f343013c1d0 msgr2=0x7f343009ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f343013c1d0 0x7f343009ea60 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f3428000f90 tx=0x7f3428004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f343013c1d0 msgr2=0x7f343009ea60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f342dffb640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f343013c1d0 0x7f343009ea60 secure :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f3428000f90 tx=0x7f3428004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 149352, 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:13.627 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.628 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:13.630 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:13.631 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.632 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:13.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.639 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:13.650 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 10 librados: watch_flush enter 2024-09-11T02:52:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 10 librados: watch_flush exit 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 20 client.23087.objecter shutdown clearing up homeless session... 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 10 client.23087.objecter successfully canceled tick 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f341c03df60 msgr2=0x7f341c040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f341c03df60 0x7f341c040420 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7f3420006aa0 tx=0x7f3420008040 comp rx=0 tx=0).stop 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 10 monclient: shutdown 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a5830 msgr2=0x7f3430136510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 --2- 172.21.15.98:0/982624802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f34300a5830 0x7f3430136510 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f34240097c0 tx=0x7f3424018f00 comp rx=0 tx=0).stop 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3435ff6640 1 -- 172.21.15.98:0/982624802 reap_dead start 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3435ff6640 1 -- 172.21.15.98:0/982624802 reap_dead start 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 shutdown_connections 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 >> 172.21.15.98:0/982624802 conn(0x7f34300aa420 msgr2=0x7f34300a8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.603+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 shutdown_connections 2024-09-11T02:52:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.607+0000 7f3437a59640 1 -- 172.21.15.98:0/982624802 wait complete. 2024-09-11T02:52:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.607+0000 7f3437a59640 1 librados: shutdown 2024-09-11T02:52:13.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:13.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:52:13.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:52:13.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient: get_monmap_and_config 2024-09-11T02:52:13.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient: monmap: 2024-09-11T02:52:13.768 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.768 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:13.769879+0000 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:13.769879+0000 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:13.770 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 1 Processor -- start 2024-09-11T02:52:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 1 -- start start 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient: init 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient: _add_conns ranks=[5,10,1,3,8,7,2,6,9,0,4] 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb034139b10 0x7fb034072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient(hunting): picked mon.noname-h con 0x7fb034139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb034139ef0 0x7fb0340a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient(hunting): picked mon.noname-f con 0x7fb034139ef0 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb03413a2d0 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0340702b0 con 0x7fb03413a2d0 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb034071bf0 con 0x7fb034139ef0 2024-09-11T02:52:13.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb034071350 con 0x7fb034139b10 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03c5f1640 10 monclient(hunting): authenticate will time out at 1087.650368s 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb034139b10 0x7fb034072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb034139ef0 0x7fb0340a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb034139b10 0x7fb034072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:44122/0 (socket says 172.21.15.98:44122) 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 -- 172.21.15.98:0/1527220563 learned_addr learned my addr 172.21.15.98:0/1527220563 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): get_auth_request con 0x7fb034139b10 auth_method 0 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 10 monclient(hunting): get_auth_request con 0x7fb034139ef0 auth_method 0 2024-09-11T02:52:13.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient(hunting): handle_auth_done global_id 19211 payload 274 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 -- 172.21.15.98:0/1527220563 >> v2:172.21.15.190:3300/0 conn(0x7fb034139ef0 msgr2=0x7fb0340a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 --2- 172.21.15.98:0/1527220563 >> v2:172.21.15.190:3300/0 conn(0x7fb034139ef0 0x7fb0340a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 -- 172.21.15.98:0/1527220563 >> v1:172.21.15.190:6789/0 conn(0x7fb03413a2d0 legacy=0x7fb0340a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 monclient: found mon.noname-h 2024-09-11T02:52:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:52:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb03a38d640 1 -- 172.21.15.98:0/1527220563 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0340a9620 con 0x7fb034139b10 2024-09-11T02:52:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.767+0000 7fb039b8c640 1 --2- 172.21.15.98:0/1527220563 >> v2:172.21.15.190:3300/0 conn(0x7fb034139ef0 0x7fb0340a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03a38d640 1 --2- 172.21.15.98:0/1527220563 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb034139b10 0x7fb034072600 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fb030009870 tx=0x7fb03001c4b0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=c6b9f16f184ef36b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 1 -- 172.21.15.98:0/1527220563 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0300b0070 con 0x7fb034139b10 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: dump: 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.782 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 5 monclient: authenticate success, global_id 19211 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 1 -- 172.21.15.98:0/1527220563 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0300aa810 con 0x7fb034139b10 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 1 -- 172.21.15.98:0/1527220563 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0300a9300 con 0x7fb034139b10 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 4 set_mon_vals no callback set 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03938b640 10 monclient: dump: 2024-09-11T02:52:13.784 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 10 monclient: shutdown 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1527220563 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb034139b10 msgr2=0x7fb034072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1527220563 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb034139b10 0x7fb034072600 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fb030009870 tx=0x7fb03001c4b0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1527220563 shutdown_connections 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1527220563 >> v2:172.21.15.190:3300/0 conn(0x7fb034139ef0 0x7fb0340a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1527220563 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb034139b10 0x7fb034072600 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1527220563 >> 172.21.15.98:0/1527220563 conn(0x7fb0340ac790 msgr2=0x7fb0340a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1527220563 shutdown_connections 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1527220563 wait complete. 2024-09-11T02:52:13.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 10 monclient: build_initial_monmap 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 10 monclient: monmap: 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:13.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 librados: starting msgr at 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 librados: starting objecter 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.771+0000 7fb03c5f1640 1 Processor -- start 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 -- start start 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 librados: setting wanted keys 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 librados: calling monclient init 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient: init 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:13.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient: _add_conns ranks=[4,2,6,5,0,7,3,1,8] 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb034139b10 0x7fb03413a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): picked mon.noname-c con 0x7fb034139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb034139ef0 0x7fb03413aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): picked mon.noname-b con 0x7fb034139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb03413a2d0 0x7fb03413b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): picked mon.noname-d con 0x7fb03413a2d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb034144470 con 0x7fb034139ef0 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb0340a8870 con 0x7fb034139b10 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb0341445f0 con 0x7fb03413a2d0 2024-09-11T02:52:13.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): authenticate will time out at 1087.656485s 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb034139b10 0x7fb03413a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb03413a2d0 0x7fb03413b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb034139b10 0x7fb03413a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45954/0 (socket says 172.21.15.98:45954) 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 1 -- 172.21.15.98:0/1739089001 learned_addr learned my addr 172.21.15.98:0/1739089001 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): get_auth_request con 0x7fb03413a2d0 auth_method 0 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): get_auth_request con 0x7fb034139b10 auth_method 0 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb034139ef0 0x7fb03413aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient(hunting): get_auth_request con 0x7fb034139ef0 auth_method 0 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient(hunting): handle_auth_done global_id 14325 payload 824 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb034139b10 msgr2=0x7fb03413a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb034139b10 0x7fb03413a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb034139ef0 msgr2=0x7fb03413aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb034139ef0 0x7fb03413aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient: _finish_hunting 0 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 monclient: found mon.noname-d 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 -- 172.21.15.98:0/1739089001 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03413ce20 con 0x7fb03413a2d0 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb034139b10 0x7fb03413a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03ab8e640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb03413a2d0 0x7fb03413b1e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00c680 tx=0x7fb02c00cb40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 client.?.objecter ms_handle_connect 0x7fb03413a2d0 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb02c003df0 con 0x7fb03413a2d0 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: dump: 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.793 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: _finish_auth 0 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 5 monclient: authenticate success, global_id 14325 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb02c0050d0 con 0x7fb03413a2d0 2024-09-11T02:52:13.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb02c0056f0 con 0x7fb03413a2d0 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient: _renew_subs 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb03413d100 con 0x7fb03413a2d0 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb038b8a640 4 set_mon_vals no callback set 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: dump: 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 client.14325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient: _renew_subs 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fb034073880 con 0x7fb03413a2d0 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 librados: init done 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0341337f0 0x7fb034133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb03406ca10 con 0x7fb0341337f0 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb02c005280 con 0x7fb03413a2d0 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0341337f0 0x7fb034133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb01803df40 0x7fb018040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient: get_auth_request con 0x7fb0341337f0 auth_method 0 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(88..88 src has 1..88) ==== 8927+0+0 (secure 0 0 0) 0x7fb02c04fe30 con 0x7fb03413a2d0 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 client.14325.objecter ms_dispatch 0x7fb034000ba0 osd_map(88..88 src has 1..88) 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 3 client.14325.objecter handle_osd_map got epochs [88,88] > 0 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 3 client.14325.objecter handle_osd_map decoding full epoch 88 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 20 client.14325.objecter dump_active .. 0 homeless 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb01803df40 0x7fb018040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 10 monclient: get_auth_request con 0x7fb01803df40 auth_method 0 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 10 monclient(hunting): handle_auth_done global_id 18880 payload 824 2024-09-11T02:52:13.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 1 -- 172.21.15.98:0/1739089001 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0341415c0 con 0x7fb0341337f0 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 client.14325.objecter ms_handle_connect 0x7fb0341337f0 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 client.14325.objecter resend_mon_ops 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb03a38d640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0341337f0 0x7fb034133bd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb0300097c0 tx=0x7fb030019250 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0300ba020 con 0x7fb0341337f0 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 monclient: dump: 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb02affd640 10 client.14325.objecter ms_handle_connect 0x7fb01803df40 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.775+0000 7fb039b8c640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb01803df40 0x7fb018040400 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7fb024009580 tx=0x7fb024009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.779+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0300aa750 con 0x7fb0341337f0 2024-09-11T02:52:13.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.779+0000 7fb02affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.779+0000 7fb02affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.779+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0341337f0 msgr2=0x7fb034133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.779+0000 7fb02affd640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0341337f0 0x7fb034133bd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb0300097c0 tx=0x7fb030019250 comp rx=0 tx=0).stop 2024-09-11T02:52:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.779+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0341337f0 msgr2=0x7fb034133bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.779+0000 7fb02affd640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb0341337f0 0x7fb034133bd0 secure :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fb0300097c0 tx=0x7fb030019250 comp rx=0 tx=0).stop 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03c5f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb034134110 0x7fb0340a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03c5f1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb03406e870 con 0x7fb034134110 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb034134110 0x7fb0340a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient: get_auth_request con 0x7fb034134110 auth_method 0 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:13.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 10 monclient(hunting): handle_auth_done global_id 18889 payload 824 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 1 -- 172.21.15.98:0/1739089001 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb03406e680 con 0x7fb034134110 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb03a38d640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb034134110 0x7fb0340a0e60 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fb03001c4f0 tx=0x7fb0300a8000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb02affd640 10 client.14325.objecter ms_handle_connect 0x7fb034134110 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.783+0000 7fb02affd640 10 client.14325.objecter resend_mon_ops 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0300ba040 con 0x7fb034134110 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 10 monclient: dump: 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:13.804 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:13.805 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fb0300c1020 con 0x7fb034134110 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb034134110 msgr2=0x7fb0340a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb034134110 0x7fb0340a0e60 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fb03001c4f0 tx=0x7fb0300a8000 comp rx=0 tx=0).stop 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb034134110 msgr2=0x7fb0340a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb02affd640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb034134110 0x7fb0340a0e60 secure :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fb03001c4f0 tx=0x7fb0300a8000 comp rx=0 tx=0).stop 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:13.806 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 26, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 149552, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:13.807 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.808 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.809 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.810 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:13.811 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:13.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:13.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.824 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.826 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:13.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:13.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:13.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:13.830 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 10 librados: watch_flush enter 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 10 librados: watch_flush exit 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 20 client.14325.objecter shutdown clearing up homeless session... 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 10 client.14325.objecter successfully canceled tick 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb01803df40 msgr2=0x7fb018040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb01803df40 0x7fb018040400 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7fb024009580 tx=0x7fb024009190 comp rx=0 tx=0).stop 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 10 monclient: shutdown 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb03413a2d0 msgr2=0x7fb03413b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb03413a2d0 0x7fb03413b1e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00c680 tx=0x7fb02c00cb40 comp rx=0 tx=0).stop 2024-09-11T02:52:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03ab8e640 1 -- 172.21.15.98:0/1739089001 reap_dead start 2024-09-11T02:52:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 shutdown_connections 2024-09-11T02:52:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 --2- 172.21.15.98:0/1739089001 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb03413a2d0 0x7fb03413b1e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 >> 172.21.15.98:0/1739089001 conn(0x7fb0340ac790 msgr2=0x7fb0340a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 shutdown_connections 2024-09-11T02:52:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 -- 172.21.15.98:0/1739089001 wait complete. 2024-09-11T02:52:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:13.787+0000 7fb03c5f1640 1 librados: shutdown 2024-09-11T02:52:13.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:52:13.833 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-11T02:52:13.834 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-11T02:52:13.834 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T02:52:13.834 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T02:52:13.834 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T02:52:13.834 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:52:13.845 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T02:52:13.845 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T02:52:13.845 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T02:52:13.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T02:52:28.846 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T02:52:28.846 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T02:52:28.847 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-09-11T02:52:28.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:52:28.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient: get_monmap_and_config 2024-09-11T02:52:28.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient: build_initial_monmap 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient: monmap: 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:28.975632+0000 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:28.975632+0000 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:28.975 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:28.976 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:28.976 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:28.976 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:28.976 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:28.976 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:28.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 1 Processor -- start 2024-09-11T02:52:28.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 1 -- start start 2024-09-11T02:52:28.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient: init 2024-09-11T02:52:28.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:28.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient: _add_conns ranks=[6,10,0,8,9,7,5,1,2,3,4] 2024-09-11T02:52:28.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient(hunting): picked mon.noname-d con 0x7fbac806f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient(hunting): picked mon.noname-f con 0x7fbac80a8380 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbac806ff70 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbac8071a80 con 0x7fbac806ff70 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbac80a8800 con 0x7fbac80a8380 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbac8071d60 con 0x7fbac806f650 2024-09-11T02:52:28.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.970+0000 7fbacf05d640 10 monclient(hunting): authenticate will time out at 1102.855952s 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbacd5fa640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fbac806ff70 legacy=0x7fbac81397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:48516/0 (socket says 172.21.15.98:48516) 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 learned_addr learned my addr 172.21.15.98:0/2191234094 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbac806f650 auth_method 0 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:28.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbac80a8380 auth_method 0 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:28.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:28.974+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbac80a8380 auth_method 0 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> v1:172.21.15.98:6789/0 conn(0x7fbac806ff70 legacy=0x7fbac81397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:52:29.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbac806f650 auth_method 0 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:29.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.178+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:52:29.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbac806f650 auth_method 0 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> v1:172.21.15.98:6789/0 conn(0x7fbac806ff70 legacy=0x7fbac81397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbac80a8380 auth_method 0 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:29.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:29.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:29.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:29.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:29.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:29.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:29.582+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:52:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbac806f650 auth_method 0 2024-09-11T02:52:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> v1:172.21.15.98:6789/0 conn(0x7fbac806ff70 legacy=0x7fbac81397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbac80a8380 auth_method 0 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:30.386+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:52:31.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): tick 2024-09-11T02:52:31.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 monclient(hunting): continuing hunt 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 msgr2=0x7fbac8072640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbac806f650 0x7fbac8072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 msgr2=0x7fbac806fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> v2:172.21.15.190:3300/0 conn(0x7fbac80a8380 0x7fbac806fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 >> v1:172.21.15.98:6789/0 conn(0x7fbac806ff70 legacy=0x7fbac81397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient: _add_conns ranks=[2,5,4,8,7,1,3,9] 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): picked mon.noname-b con 0x7fbab40037e0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:31.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): picked mon.noname-h con 0x7fbab4006210 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): picked mon.noname-c con 0x7fbab4008d90 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80a8800 con 0x7fbab40037e0 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbac8071d60 con 0x7fbab4008d90 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbab400b9e0 con 0x7fbab4006210 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac67fc640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:31.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbab40037e0 auth_method 0 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request con 0x7fbab4008d90 auth_method 0 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbab4006210 auth_method 0 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:31.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:31.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:31.974+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbab40037e0 auth_method 0 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request con 0x7fbab4008d90 auth_method 0 2024-09-11T02:52:32.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:52:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbab4006210 auth_method 0 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:32.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.178+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:52:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request con 0x7fbab4008d90 auth_method 0 2024-09-11T02:52:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbab40037e0 auth_method 0 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbab4006210 auth_method 0 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:32.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:32.582+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:52:33.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:33.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:33.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request con 0x7fbab4008d90 auth_method 0 2024-09-11T02:52:33.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:33.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbab40037e0 auth_method 0 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:52:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbab4006210 auth_method 0 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:52:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:33.386+0000 7fbac7fff640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListEC.L api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-11T02:52:33.914 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: seed 28279 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (7 ms) 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (312 ms) 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (19 ms) 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (19 ms) 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (9 ms) 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (6 ms) 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-11T02:52:33.915 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (6 ms) 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (14 ms) 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (4 ms) 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (29 ms) 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (22 ms) 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-11T02:52:33.916 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (4070 ms) 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (717 ms) 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: done waiting, doing copies 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: done waiting 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60994 ms) 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (11 ms) 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-11T02:52:33.917 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4613 ms) 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-11T02:52:33.918 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-11T02:52:33.968 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 22 tests from LibRadosMisc snapshots: Running main() from gmock_main.cc 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] Global test environment set-up. 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4183 ms) 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5007 ms) 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3756 ms) 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5029 ms) 2024-09-11T02:52:33.969 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7036 ms) 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25012 ms total) 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4988 ms) 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6037 ms) 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7927 ms) 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6048 ms) 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-11T02:52:33.970 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3960 ms) 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi190-29148-11. 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: Waiting for snaps to purge. 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (38633 ms) 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (67593 ms total) 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [----------] Global test environment tear-down 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (92605 ms total) 2024-09-11T02:52:33.971 INFO:tasks.workunit.client.0.smithi190.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-11T02:52:34.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-09-11T02:52:34.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: seed 28403 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2006 ms) 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2031 ms) 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2030 ms) 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1995 ms) 2024-09-11T02:52:34.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3760 ms) 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11822 ms total) 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4194 ms) 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3830 ms) 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5983 ms) 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-11T02:52:34.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4034 ms) 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2134 ms) 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi190-28403-7 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17868 ms) 2024-09-11T02:52:34.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38043 ms total) 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (24 ms) 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (10080 ms) 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (10104 ms total) 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3011 ms) 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-11T02:52:34.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (16246 ms) 2024-09-11T02:52:34.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-11T02:52:34.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (5750 ms) 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): tick 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 monclient(hunting): continuing hunt 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 msgr2=0x7fbab40086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4006210 0x7fbab40086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 msgr2=0x7fbab400b220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:34.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbab4008d90 0x7fbab400b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 msgr2=0x7fbab4005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbab40037e0 0x7fbab4005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 reap_dead start 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 reap_dead start 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient: _add_conns ranks=[7,1,3,8,9] 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbab4003c00 0x7fbab4006690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): picked mon.noname-i con 0x7fbab4003c00 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbab4006bd0 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4008730 0x7fbab400b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): picked mon.noname-g con 0x7fbab4008730 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:34.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbac80714c0 con 0x7fbab4006bd0 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbac80a8800 con 0x7fbab4008730 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 1 -- 172.21.15.98:0/2191234094 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbac8071d60 con 0x7fbab4003c00 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac67fc640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4008730 0x7fbab400b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbab4003c00 0x7fbab4006690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request con 0x7fbab4008730 auth_method 0 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:34.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request con 0x7fbab4003c00 auth_method 0 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac77fe640 1 -- 172.21.15.98:0/2191234094 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202200004 0 0) 0x7fbac80714c0 con 0x7fbab4006bd0 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac77fe640 10 monclient(hunting): _negotiate 2024-09-11T02:52:34.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac77fe640 10 monclient(hunting): my global_id is 24211 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbac77fe640 1 -- 172.21.15.98:0/2191234094 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbab0003530 con 0x7fbab4006bd0 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient(hunting): handle_auth_done global_id 19542 payload 274 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbab4003c00 msgr2=0x7fbab4006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbab4003c00 0x7fbab4006690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 >> v1:172.21.15.190:6789/0 conn(0x7fbab4006bd0 legacy=0x7fbab4007010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient: _finish_hunting 0 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 monclient: found mon.noname-g 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:52:34.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbacd5fa640 1 -- 172.21.15.98:0/2191234094 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac8071a80 con 0x7fbab4008730 2024-09-11T02:52:34.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.974+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbab4003c00 0x7fbab4006690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.978+0000 7fbacd5fa640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4008730 0x7fbab400b180 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fbac80a8980 tx=0x7fbac001da70 comp rx=0 tx=0).ready entity=mon.3 client_cookie=f14bfdfe22f42f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 1 -- 172.21.15.98:0/2191234094 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbac00b2070 con 0x7fbab4008730 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: dump: 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:34.993 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:34.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 1 -- 172.21.15.98:0/2191234094 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac00ac820 con 0x7fbab4008730 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 5 monclient: authenticate success, global_id 19542 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 1 -- 172.21.15.98:0/2191234094 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbac00ace40 con 0x7fbab4008730 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 4 set_mon_vals no callback set 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:34.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbac77fe640 10 monclient: dump: 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:34.996 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 10 monclient: shutdown 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 -- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4008730 msgr2=0x7fbab400b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4008730 0x7fbab400b180 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fbac80a8980 tx=0x7fbac001da70 comp rx=0 tx=0).stop 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 -- 172.21.15.98:0/2191234094 shutdown_connections 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4008730 0x7fbab400b180 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 --2- 172.21.15.98:0/2191234094 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbab4003c00 0x7fbab4006690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:34.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 -- 172.21.15.98:0/2191234094 >> 172.21.15.98:0/2191234094 conn(0x7fbac80ac8e0 msgr2=0x7fbac80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 -- 172.21.15.98:0/2191234094 shutdown_connections 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 -- 172.21.15.98:0/2191234094 wait complete. 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 10 monclient: build_initial_monmap 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 10 monclient: monmap: 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:34.998 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 librados: starting msgr at 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 librados: starting objecter 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 Processor -- start 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 -- start start 2024-09-11T02:52:34.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 librados: setting wanted keys 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 1 librados: calling monclient init 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.982+0000 7fbacf05d640 10 monclient: init 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _add_conns ranks=[0,3,5,2,8,1,7,4,6] 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbab4003c00 0x7fbac81389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): picked mon.noname-a con 0x7fbab4003c00 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4006bd0 0x7fbac8138f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): picked mon.noname-g con 0x7fbab4006bd0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4008730 0x7fbac8122650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): picked mon.noname-h con 0x7fbab4008730 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbac813a360 con 0x7fbab4003c00 2024-09-11T02:52:35.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbac813a1e0 con 0x7fbab4006bd0 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbaccdf9640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbab4003c00 msgr2=0x7fbac81389e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbac813a4e0 con 0x7fbab4008730 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient(hunting): authenticate will time out at 1108.868351s 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbaccdf9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbab4003c00 0x7fbac81389e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4006bd0 0x7fbac8138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4008730 0x7fbac8122650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4006bd0 0x7fbac8138f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:54646/0 (socket says 172.21.15.98:54646) 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 -- 172.21.15.98:0/670819074 learned_addr learned my addr 172.21.15.98:0/670819074 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): get_auth_request con 0x7fbab4006bd0 auth_method 0 2024-09-11T02:52:35.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request con 0x7fbab4008730 auth_method 0 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacd5fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient(hunting): handle_auth_done global_id 19551 payload 824 2024-09-11T02:52:35.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 -- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4008730 msgr2=0x7fbac8122650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4008730 0x7fbac8122650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 -- 172.21.15.98:0/670819074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbab4003c00 msgr2=0x7fbac81389e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbab4003c00 0x7fbac81389e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 monclient: found mon.noname-g 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 -- 172.21.15.98:0/670819074 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac8124040 con 0x7fbab4006bd0 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac7fff640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4006bd0 0x7fbac8138f20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fbabc017a10 tx=0x7fbabc017ed0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 client.?.objecter ms_handle_connect 0x7fbab4006bd0 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 1 -- 172.21.15.98:0/670819074 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbabc026070 con 0x7fbab4006bd0 2024-09-11T02:52:35.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: dump: 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.004 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: _finish_auth 0 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: _check_auth_tickets 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 1 -- 172.21.15.98:0/670819074 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbabc0044a0 con 0x7fbab4006bd0 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 1 -- 172.21.15.98:0/670819074 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbabc016c20 con 0x7fbab4006bd0 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 5 monclient: authenticate success, global_id 19551 2024-09-11T02:52:35.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 10 monclient: dump: 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.006 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac67fc640 4 set_mon_vals no callback set 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _renew_subs 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac81263a0 con 0x7fbab4006bd0 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 client.19551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _renew_subs 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:35.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac8124ba0 con 0x7fbab4006bd0 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 librados: init done 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbac813cc00 con 0x7fbab4006bd0 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 1 -- 172.21.15.98:0/670819074 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbabc004030 con 0x7fbab4006bd0 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.986+0000 7fbac57fa640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbab4040010 0x7fbab40424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbac57fa640 1 -- 172.21.15.98:0/670819074 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(91..91 src has 1..91) ==== 8517+0+0 (secure 0 0 0) 0x7fbabc01e030 con 0x7fbab4006bd0 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbac57fa640 10 client.19551.objecter ms_dispatch 0x7fbac8000ba0 osd_map(91..91 src has 1..91) 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbac57fa640 3 client.19551.objecter handle_osd_map got epochs [91,91] > 0 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbac57fa640 3 client.19551.objecter handle_osd_map decoding full epoch 91 2024-09-11T02:52:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbac57fa640 20 client.19551.objecter dump_active .. 0 homeless 2024-09-11T02:52:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbab4040010 0x7fbab40424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbaccdf9640 10 monclient: get_auth_request con 0x7fbab4040010 auth_method 0 2024-09-11T02:52:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbac57fa640 10 client.19551.objecter ms_handle_connect 0x7fbab4040010 2024-09-11T02:52:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:34.990+0000 7fbaccdf9640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbab4040010 0x7fbab40424d0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009100 tx=0x7fbab801b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.002+0000 7fbac57fa640 1 -- 172.21.15.98:0/670819074 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbabc003e20 con 0x7fbab4006bd0 2024-09-11T02:52:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.002+0000 7fbac57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.002+0000 7fbac57fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.238+0000 7fbacf05d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:52:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.238+0000 7fbacf05d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.238+0000 7fbacf05d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T02:52:35.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.238+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbac8125ee0 con 0x7fbab4006bd0 2024-09-11T02:52:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbac57fa640 1 -- 172.21.15.98:0/670819074 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7fbabc04f040 con 0x7fbab4006bd0 2024-09-11T02:52:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbac57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:52:35.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbac57fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:35.244 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":32,"quorum":[1,2,3,4,5,6,7,8],"quorum_names":["f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:52:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 10 librados: watch_flush enter 2024-09-11T02:52:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 10 librados: watch_flush exit 2024-09-11T02:52:35.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 20 client.19551.objecter shutdown clearing up homeless session... 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 10 client.19551.objecter successfully canceled tick 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbab4040010 msgr2=0x7fbab40424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbab4040010 0x7fbab40424d0 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fbab8009100 tx=0x7fbab801b040 comp rx=0 tx=0).stop 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 10 monclient: shutdown 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4006bd0 msgr2=0x7fbac8138f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4006bd0 0x7fbac8138f20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fbabc017a10 tx=0x7fbabc017ed0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 shutdown_connections 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbab4040010 0x7fbab40424d0 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbab4008730 0x7fbac8122650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbab4006bd0 0x7fbac8138f20 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 --2- 172.21.15.98:0/670819074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbab4003c00 0x7fbac81389e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 >> 172.21.15.98:0/670819074 conn(0x7fbac80ac8e0 msgr2=0x7fbac80a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 shutdown_connections 2024-09-11T02:52:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 -- 172.21.15.98:0/670819074 wait complete. 2024-09-11T02:52:35.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.242+0000 7fbacf05d640 1 librados: shutdown 2024-09-11T02:52:35.263 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-09-11T02:52:35.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-09-11T02:52:35.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:52:35.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient: get_monmap_and_config 2024-09-11T02:52:35.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient: build_initial_monmap 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient: monmap: 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:35.411383+0000 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:35.411383+0000 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:35.411 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 Processor -- start 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 -- start start 2024-09-11T02:52:35.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient: init 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient: _add_conns ranks=[2,7,10,0,6,5,4,1,9,3,8] 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c0a09d0 0x7ff64c0a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): picked mon.noname-b con 0x7ff64c0a09d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff64c0a0200 0x7ff64c1365c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): picked mon.noname-i con 0x7ff64c0a0200 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff64c136b00 0x7ff64c136f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): picked mon.noname-f con 0x7ff64c136b00 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff64c09ff40 con 0x7ff64c136b00 2024-09-11T02:52:35.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c0a50c0 con 0x7ff64c0a09d0 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7ff64c0a5340 con 0x7ff64c0a0200 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff6530e1640 10 monclient(hunting): authenticate will time out at 1109.291378s 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c0a09d0 0x7ff64c0a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c0a09d0 0x7ff64c0a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54620/0 (socket says 172.21.15.98:54620) 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 -- 172.21.15.98:0/1143656915 learned_addr learned my addr 172.21.15.98:0/1143656915 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): get_auth_request con 0x7ff64c0a09d0 auth_method 0 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff65167e640 1 --2- 172.21.15.98:0/1143656915 >> v2:172.21.15.190:3300/0 conn(0x7ff64c136b00 0x7ff64c136f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff65167e640 10 monclient(hunting): get_auth_request con 0x7ff64c136b00 auth_method 0 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff65167e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff65167e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff65167e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient(hunting): handle_auth_done global_id 23132 payload 274 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 -- 172.21.15.98:0/1143656915 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff64c0a0200 msgr2=0x7ff64c1365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 --2- 172.21.15.98:0/1143656915 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff64c0a0200 0x7ff64c1365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 -- 172.21.15.98:0/1143656915 >> v2:172.21.15.190:3300/0 conn(0x7ff64c136b00 msgr2=0x7ff64c136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 --2- 172.21.15.98:0/1143656915 >> v2:172.21.15.190:3300/0 conn(0x7ff64c136b00 0x7ff64c136f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient: _finish_hunting 0 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 monclient: found mon.noname-b 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.406+0000 7ff650e7d640 1 -- 172.21.15.98:0/1143656915 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c0a1f20 con 0x7ff64c0a09d0 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff650e7d640 1 --2- 172.21.15.98:0/1143656915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c0a09d0 0x7ff64c0a0db0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7ff640009870 tx=0x7ff64001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6c6a2417a763ab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 1 -- 172.21.15.98:0/1143656915 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6400b0070 con 0x7ff64c0a09d0 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: dump: 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 5 monclient: authenticate success, global_id 23132 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 1 -- 172.21.15.98:0/1143656915 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6400aa3a0 con 0x7ff64c0a09d0 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 1 -- 172.21.15.98:0/1143656915 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6400aaa40 con 0x7ff64c0a09d0 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 4 set_mon_vals no callback set 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff64b7fe640 10 monclient: dump: 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 10 monclient: shutdown 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 -- 172.21.15.98:0/1143656915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c0a09d0 msgr2=0x7ff64c0a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 --2- 172.21.15.98:0/1143656915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c0a09d0 0x7ff64c0a0db0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7ff640009870 tx=0x7ff64001c420 comp rx=0 tx=0).stop 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 -- 172.21.15.98:0/1143656915 shutdown_connections 2024-09-11T02:52:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 --2- 172.21.15.98:0/1143656915 >> v2:172.21.15.190:3300/0 conn(0x7ff64c136b00 0x7ff64c136f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 --2- 172.21.15.98:0/1143656915 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff64c0a0200 0x7ff64c1365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 --2- 172.21.15.98:0/1143656915 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c0a09d0 0x7ff64c0a0db0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 -- 172.21.15.98:0/1143656915 >> 172.21.15.98:0/1143656915 conn(0x7ff64c06eed0 msgr2=0x7ff64c06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 -- 172.21.15.98:0/1143656915 shutdown_connections 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.410+0000 7ff6530e1640 1 -- 172.21.15.98:0/1143656915 wait complete. 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: build_initial_monmap 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: monmap: 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 librados: starting msgr at 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 librados: starting objecter 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 Processor -- start 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 -- start start 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 librados: setting wanted keys 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 librados: calling monclient init 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: init 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: _add_conns ranks=[6,8,1,3,5,2,0,7,4] 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff64c0a0200 0x7ff64c076b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): picked mon.noname-d con 0x7ff64c0a0200 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff64c0a09d0 0x7ff64c132c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): picked mon.noname-e con 0x7ff64c0a09d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff64c136b00 0x7ff64c138420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): picked mon.noname-f con 0x7ff64c136b00 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c140520 con 0x7ff64c136b00 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff64c0a0850 con 0x7ff64c0a0200 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff64c1406a0 con 0x7ff64c0a09d0 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): authenticate will time out at 1109.297285s 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff64c0a0200 0x7ff64c076b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff64c0a0200 0x7ff64c076b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:48546/0 (socket says 172.21.15.98:48546) 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 -- 172.21.15.98:0/3017371074 learned_addr learned my addr 172.21.15.98:0/3017371074 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): get_auth_request con 0x7ff64c0a0200 auth_method 0 2024-09-11T02:52:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff64c0a09d0 0x7ff64c132c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 10 monclient(hunting): get_auth_request con 0x7ff64c0a09d0 auth_method 0 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient(hunting): handle_auth_done global_id 14352 payload 824 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff64c0a09d0 msgr2=0x7ff64c132c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff64c0a09d0 0x7ff64c132c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff64c136b00 msgr2=0x7ff64c138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff64c136b00 0x7ff64c138420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient: _finish_hunting 0 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 monclient: found mon.noname-d 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 -- 172.21.15.98:0/3017371074 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c0773d0 con 0x7ff64c0a0200 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff650e7d640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff64c0a0200 0x7ff64c076b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff64001ccc0 tx=0x7ff6400191c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 client.?.objecter ms_handle_connect 0x7ff64c0a0200 2024-09-11T02:52:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6400b0040 con 0x7ff64c0a0200 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: dump: 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: _finish_auth 0 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: _check_auth_tickets 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 5 monclient: authenticate success, global_id 14352 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff640003c90 con 0x7ff64c0a0200 2024-09-11T02:52:35.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: _renew_subs 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64c13b070 con 0x7ff64c0a0200 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff640003e40 con 0x7ff64c0a0200 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff64affd640 4 set_mon_vals no callback set 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 client.14352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 10 monclient: dump: 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.440 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: _renew_subs 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64c13a4a0 con 0x7ff64c0a0200 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 librados: init done 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c138960 0x7ff64c138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c06ca10 con 0x7ff64c138960 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ff6400b3070 con 0x7ff64c0a0200 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff63003df40 0x7ff630040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.414+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(92..92 src has 1..92) ==== 7856+0+0 (secure 0 0 0) 0x7ff6400e9660 con 0x7ff64c0a0200 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 client.14352.objecter ms_dispatch 0x7ff64c000ba0 osd_map(92..92 src has 1..92) 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 3 client.14352.objecter handle_osd_map got epochs [92,92] > 0 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 3 client.14352.objecter handle_osd_map decoding full epoch 92 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 20 client.14352.objecter dump_active .. 0 homeless 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff65167e640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff63003df40 0x7ff630040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff65167e640 10 monclient: get_auth_request con 0x7ff63003df40 auth_method 0 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c138960 0x7ff64c138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient: get_auth_request con 0x7ff64c138960 auth_method 0 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 10 monclient(hunting): handle_auth_done global_id 23141 payload 824 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 client.14352.objecter ms_handle_connect 0x7ff63003df40 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff65167e640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff63003df40 0x7ff630040400 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7ff6440070b0 tx=0x7ff644007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 1 -- 172.21.15.98:0/3017371074 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff64c0a0db0 con 0x7ff64c138960 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 client.14352.objecter ms_handle_connect 0x7ff64c138960 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 client.14352.objecter resend_mon_ops 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff64bfff640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c138960 0x7ff64c138d40 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0a4e50 tx=0x7ff63800ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff638017070 con 0x7ff64c138960 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 monclient: dump: 2024-09-11T02:52:35.443 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.444 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff63800c7a0 con 0x7ff64c138960 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c138960 msgr2=0x7ff64c138d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c138960 0x7ff64c138d40 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0a4e50 tx=0x7ff63800ec00 comp rx=0 tx=0).stop 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c138960 msgr2=0x7ff64c138d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:35.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.418+0000 7ff6497fa640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c138960 0x7ff64c138d40 secure :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0a4e50 tx=0x7ff63800ec00 comp rx=0 tx=0).stop 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c070610 0x7ff64c070a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c06e870 con 0x7ff64c070610 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c070610 0x7ff64c070a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient: get_auth_request con 0x7ff64c070610 auth_method 0 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 10 monclient(hunting): handle_auth_done global_id 23150 payload 824 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 1 -- 172.21.15.98:0/3017371074 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff64c0a3de0 con 0x7ff64c070610 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 10 client.14352.objecter ms_handle_connect 0x7ff64c070610 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff64bfff640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c070610 0x7ff64c070a20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0a4e50 tx=0x7ff638004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 10 client.14352.objecter resend_mon_ops 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff638017070 con 0x7ff64c070610 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 10 monclient: dump: 2024-09-11T02:52:35.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7ff6380062d0 con 0x7ff64c070610 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c070610 msgr2=0x7ff64c070a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c070610 0x7ff64c070a20 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0a4e50 tx=0x7ff638004740 comp rx=0 tx=0).stop 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c070610 msgr2=0x7ff64c070a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:35.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6497fa640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff64c070610 0x7ff64c070a20 secure :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0a4e50 tx=0x7ff638004740 comp rx=0 tx=0).stop 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 171211, 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:35.451 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.452 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:35.453 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.454 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:35.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.456 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:35.460 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:35.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:35.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:35.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:35.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.473 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.474 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:35.475 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 10 librados: watch_flush enter 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 10 librados: watch_flush exit 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 20 client.14352.objecter shutdown clearing up homeless session... 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 10 client.14352.objecter successfully canceled tick 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff63003df40 msgr2=0x7ff630040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff63003df40 0x7ff630040400 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7ff6440070b0 tx=0x7ff644007040 comp rx=0 tx=0).stop 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 10 monclient: shutdown 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff64c0a0200 msgr2=0x7ff64c076b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff6530e1640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff64c0a0200 0x7ff64c076b00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff64001ccc0 tx=0x7ff6400191c0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.426+0000 7ff65167e640 1 -- 172.21.15.98:0/3017371074 reap_dead start 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.430+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 shutdown_connections 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.430+0000 7ff6530e1640 1 --2- 172.21.15.98:0/3017371074 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff64c0a0200 0x7ff64c076b00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.430+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 >> 172.21.15.98:0/3017371074 conn(0x7ff64c06eed0 msgr2=0x7ff64c09f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.430+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 shutdown_connections 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.430+0000 7ff6530e1640 1 -- 172.21.15.98:0/3017371074 wait complete. 2024-09-11T02:52:35.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.430+0000 7ff6530e1640 1 librados: shutdown 2024-09-11T02:52:35.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:35.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-09-11T02:52:35.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:52:35.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient: get_monmap_and_config 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient: build_initial_monmap 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient: monmap: 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:35.594399+0000 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:35.594399+0000 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:35.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:35.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 1 Processor -- start 2024-09-11T02:52:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 1 -- start start 2024-09-11T02:52:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient: init 2024-09-11T02:52:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:35.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient: _add_conns ranks=[8,0,2,1,5,7,3,4,10,6,9] 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient(hunting): picked mon.noname-e con 0x7fcbcc139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbcc139ef0 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbcc13a2d0 0x7fcbcc0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient(hunting): picked mon.noname-b con 0x7fcbcc13a2d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbcc0702b0 con 0x7fcbcc139ef0 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcbcc071bf0 con 0x7fcbcc13a2d0 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcbcc071350 con 0x7fcbcc139b10 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:35.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd425b640 10 monclient(hunting): authenticate will time out at 1109.474615s 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd17f6640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fcbcc139ef0 legacy=0x7fcbcc0a7a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbcc13a2d0 0x7fcbcc0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbcc13a2d0 0x7fcbcc0a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54638/0 (socket says 172.21.15.98:54638) 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44282/0 (socket says 172.21.15.98:44282) 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 -- 172.21.15.98:0/1757378324 learned_addr learned my addr 172.21.15.98:0/1757378324 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): get_auth_request con 0x7fcbcc13a2d0 auth_method 0 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 10 monclient(hunting): get_auth_request con 0x7fcbcc139b10 auth_method 0 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient(hunting): handle_auth_done global_id 23159 payload 274 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 -- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 msgr2=0x7fcbcc072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 --2- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 -- 172.21.15.98:0/1757378324 >> v1:172.21.15.98:6789/0 conn(0x7fcbcc139ef0 legacy=0x7fcbcc0a7a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient: _finish_hunting 0 2024-09-11T02:52:35.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 monclient: found mon.noname-b 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd27f8640 1 -- 172.21.15.98:0/1757378324 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc0a9620 con 0x7fcbcc13a2d0 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.590+0000 7fcbd1ff7640 1 --2- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd27f8640 1 --2- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbcc13a2d0 0x7fcbcc0a8120 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80098a0 tx=0x7fcbc801c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f070df58c2eee37f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 1 -- 172.21.15.98:0/1757378324 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbc80b0070 con 0x7fcbcc13a2d0 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: dump: 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.703 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: _finish_auth 0 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: _check_auth_tickets 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 1 -- 172.21.15.98:0/1757378324 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc80aa810 con 0x7fcbcc13a2d0 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 5 monclient: authenticate success, global_id 23159 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 1 -- 172.21.15.98:0/1757378324 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbc80a9300 con 0x7fcbcc13a2d0 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 4 set_mon_vals no callback set 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd0ff5640 10 monclient: dump: 2024-09-11T02:52:35.705 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.706 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 10 monclient: shutdown 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 -- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbcc13a2d0 msgr2=0x7fcbcc0a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 --2- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbcc13a2d0 0x7fcbcc0a8120 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80098a0 tx=0x7fcbc801c2f0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 -- 172.21.15.98:0/1757378324 shutdown_connections 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 --2- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbcc13a2d0 0x7fcbcc0a8120 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 --2- 172.21.15.98:0/1757378324 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 -- 172.21.15.98:0/1757378324 >> 172.21.15.98:0/1757378324 conn(0x7fcbcc0ac790 msgr2=0x7fcbcc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 -- 172.21.15.98:0/1757378324 shutdown_connections 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 -- 172.21.15.98:0/1757378324 wait complete. 2024-09-11T02:52:35.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 10 monclient: build_initial_monmap 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 10 monclient: monmap: 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:35.708 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 librados: starting msgr at 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 librados: starting objecter 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 Processor -- start 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 -- start start 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 librados: setting wanted keys 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 1 librados: calling monclient init 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.594+0000 7fcbd425b640 10 monclient: init 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:35.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient: _add_conns ranks=[8,6,7,3,4,1,0,2,5] 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc13a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): picked mon.noname-e con 0x7fcbcc139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbcc139ef0 0x7fcbcc13ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): picked mon.noname-d con 0x7fcbcc139ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcbcc13a2d0 0x7fcbcc13b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): picked mon.noname-i con 0x7fcbcc13a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcbcc144470 con 0x7fcbcc139ef0 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fcbcc0a8870 con 0x7fcbcc13a2d0 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcbcc1445f0 con 0x7fcbcc139b10 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:35.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): authenticate will time out at 1109.480232s 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbcc139ef0 0x7fcbcc13ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc13a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbcc139ef0 0x7fcbcc13ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:48548/0 (socket says 172.21.15.98:48548) 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 -- 172.21.15.98:0/4132058325 learned_addr learned my addr 172.21.15.98:0/4132058325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): get_auth_request con 0x7fcbcc139ef0 auth_method 0 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): get_auth_request con 0x7fcbcc139b10 auth_method 0 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient(hunting): handle_auth_done global_id 14361 payload 824 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 msgr2=0x7fcbcc13a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc13a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcbcc13a2d0 msgr2=0x7fcbcc13b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd27f8640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcbcc13a2d0 0x7fcbcc13b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcbcc13a2d0 0x7fcbcc13b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient: _finish_hunting 0 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 monclient: found mon.noname-d 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 -- 172.21.15.98:0/4132058325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc13cac0 con 0x7fcbcc139ef0 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbcc139b10 0x7fcbcc13a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd17f6640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbcc139ef0 0x7fcbcc13ac20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00b580 tx=0x7fcbbc00ba40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 client.?.objecter ms_handle_connect 0x7fcbcc139ef0 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbbc016070 con 0x7fcbcc139ef0 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: dump: 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.713 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.714 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbbc003bd0 con 0x7fcbcc139ef0 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbbc003dc0 con 0x7fcbcc139ef0 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 5 monclient: authenticate success, global_id 14361 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient: _renew_subs 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc13cda0 con 0x7fcbcc139ef0 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc3fff640 4 set_mon_vals no callback set 2024-09-11T02:52:35.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: dump: 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.716 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 client.14361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient: _renew_subs 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc134790 con 0x7fcbcc139ef0 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 librados: init done 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc133760 0x7fcbcc133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcbcc06ca10 con 0x7fcbcc133760 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc133760 0x7fcbcc133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient: get_auth_request con 0x7fcbcc133760 auth_method 0 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fcbbc0050d0 con 0x7fcbcc139ef0 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_done global_id 23107 payload 824 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbb003df40 0x7fcbb0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 1 -- 172.21.15.98:0/4132058325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbcc06e680 con 0x7fcbcc133760 2024-09-11T02:52:35.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(92..92 src has 1..92) ==== 7856+0+0 (secure 0 0 0) 0x7fcbbc04efa0 con 0x7fcbcc139ef0 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 client.14361.objecter ms_dispatch 0x7fcbcc000ba0 osd_map(92..92 src has 1..92) 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 3 client.14361.objecter handle_osd_map got epochs [92,92] > 0 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 3 client.14361.objecter handle_osd_map decoding full epoch 92 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd1ff7640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc133760 0x7fcbcc133b40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4004980 tx=0x7fcbc400e8c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 20 client.14361.objecter dump_active .. 0 homeless 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 client.14361.objecter ms_handle_connect 0x7fcbcc133760 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 client.14361.objecter resend_mon_ops 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbc4019070 con 0x7fcbcc133760 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: dump: 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.719 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.720 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd27f8640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbb003df40 0x7fcbb0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd27f8640 10 monclient: get_auth_request con 0x7fcbb003df40 auth_method 0 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcbc4015720 con 0x7fcbcc133760 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc133760 msgr2=0x7fcbcc133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc133760 0x7fcbcc133b40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4004980 tx=0x7fcbc400e8c0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc133760 msgr2=0x7fcbcc133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc133760 0x7fcbcc133b40 secure :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4004980 tx=0x7fcbc400e8c0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbc27fc640 10 client.14361.objecter ms_handle_connect 0x7fcbb003df40 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.598+0000 7fcbd27f8640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbb003df40 0x7fcbb0040400 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80098a0 tx=0x7fcbc80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc134080 0x7fcbcc0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcbcc06e870 con 0x7fcbcc134080 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc134080 0x7fcbcc0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient: get_auth_request con 0x7fcbcc134080 auth_method 0 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 10 monclient(hunting): handle_auth_done global_id 23116 payload 824 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 1 -- 172.21.15.98:0/4132058325 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcbcc0a9e00 con 0x7fcbcc134080 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 10 client.14361.objecter ms_handle_connect 0x7fcbcc134080 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 10 client.14361.objecter resend_mon_ops 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd1ff7640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc134080 0x7fcbcc0a0e60 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4000f90 tx=0x7fcbc4004a50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbc401ecb0 con 0x7fcbcc134080 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 10 monclient: dump: 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.723 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9256 (secure 0 0 0) 0x7fcbc40283f0 con 0x7fcbcc134080 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:35.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc134080 msgr2=0x7fcbcc0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc134080 0x7fcbcc0a0e60 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4000f90 tx=0x7fcbc4004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc134080 msgr2=0x7fcbcc0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbc27fc640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcbcc134080 0x7fcbcc0a0e60 secure :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4000f90 tx=0x7fcbc4004a50 comp rx=0 tx=0).stop 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:35.725 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 171391, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:52:35.726 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.727 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.728 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:35.729 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.730 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.731 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.732 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.733 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.736 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:35.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:35.741 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.744 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.748 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:35.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:35.750 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 10 librados: watch_flush enter 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 10 librados: watch_flush exit 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 20 client.14361.objecter shutdown clearing up homeless session... 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 10 client.14361.objecter successfully canceled tick 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbb003df40 msgr2=0x7fcbb0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbb003df40 0x7fcbb0040400 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80098a0 tx=0x7fcbc80a6000 comp rx=0 tx=0).stop 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 10 monclient: shutdown 2024-09-11T02:52:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbcc139ef0 msgr2=0x7fcbcc13ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 --2- 172.21.15.98:0/4132058325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbcc139ef0 0x7fcbcc13ac20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00b580 tx=0x7fcbbc00ba40 comp rx=0 tx=0).stop 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd27f8640 1 -- 172.21.15.98:0/4132058325 reap_dead start 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd27f8640 1 -- 172.21.15.98:0/4132058325 reap_dead start 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 shutdown_connections 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 >> 172.21.15.98:0/4132058325 conn(0x7fcbcc0ac790 msgr2=0x7fcbcc0aa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 shutdown_connections 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 -- 172.21.15.98:0/4132058325 wait complete. 2024-09-11T02:52:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.606+0000 7fcbd425b640 1 librados: shutdown 2024-09-11T02:52:35.753 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:35.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-09-11T02:52:35.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient: get_monmap_and_config 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient: build_initial_monmap 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient: monmap: 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:35.882066+0000 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:35.882066+0000 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:35.883 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:35.883 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 Processor -- start 2024-09-11T02:52:35.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 -- start start 2024-09-11T02:52:35.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient: init 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient: _add_conns ranks=[2,3,4,0,7,6,10,5,1,9,8] 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139b10 0x7fa070072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): picked mon.noname-b con 0x7fa070139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa070139ef0 0x7fa0700a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): picked mon.noname-g con 0x7fa070139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa07013a2d0 0x7fa0700a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): picked mon.noname-c con 0x7fa07013a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa0700702b0 con 0x7fa070139b10 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa070071bf0 con 0x7fa070139ef0 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa070071350 con 0x7fa07013a2d0 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa075100640 10 monclient(hunting): authenticate will time out at 1109.762293s 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa07013a2d0 0x7fa0700a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139b10 0x7fa070072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa07013a2d0 0x7fa0700a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57894/0 (socket says 172.21.15.98:57894) 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 -- 172.21.15.98:0/653236969 learned_addr learned my addr 172.21.15.98:0/653236969 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): get_auth_request con 0x7fa07013a2d0 auth_method 0 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 10 monclient(hunting): get_auth_request con 0x7fa070139b10 auth_method 0 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06e59c640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa070139ef0 0x7fa0700a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06e59c640 10 monclient(hunting): get_auth_request con 0x7fa070139ef0 auth_method 0 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient(hunting): handle_auth_done global_id 23125 payload 274 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 -- 172.21.15.98:0/653236969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa070139ef0 msgr2=0x7fa0700a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:35.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa070139ef0 0x7fa0700a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 -- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139b10 msgr2=0x7fa070072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139b10 0x7fa070072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient: _finish_hunting 0 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 monclient: found mon.noname-c 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06f59e640 1 -- 172.21.15.98:0/653236969 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0700a9620 con 0x7fa07013a2d0 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.878+0000 7fa06ed9d640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139b10 0x7fa070072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06f59e640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa07013a2d0 0x7fa0700a7f50 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa0680098a0 tx=0x7fa06801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ce93ca3ad7380a71 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 1 -- 172.21.15.98:0/653236969 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0680b0070 con 0x7fa07013a2d0 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: dump: 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.898 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: _finish_auth 0 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: _check_auth_tickets 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 1 -- 172.21.15.98:0/653236969 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0680aa810 con 0x7fa07013a2d0 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 5 monclient: authenticate success, global_id 23125 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 4 set_mon_vals no callback set 2024-09-11T02:52:35.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 1 -- 172.21.15.98:0/653236969 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0680a9300 con 0x7fa07013a2d0 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06dd9b640 10 monclient: dump: 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.900 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient: shutdown 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa07013a2d0 msgr2=0x7fa0700a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa07013a2d0 0x7fa0700a7f50 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fa0680098a0 tx=0x7fa06801c440 comp rx=0 tx=0).stop 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- 172.21.15.98:0/653236969 shutdown_connections 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa07013a2d0 0x7fa0700a7f50 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa070139ef0 0x7fa0700a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 --2- 172.21.15.98:0/653236969 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139b10 0x7fa070072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- 172.21.15.98:0/653236969 >> 172.21.15.98:0/653236969 conn(0x7fa0700ac790 msgr2=0x7fa0700a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- 172.21.15.98:0/653236969 shutdown_connections 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- 172.21.15.98:0/653236969 wait complete. 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient: build_initial_monmap 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient: monmap: 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:35.902 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 librados: starting msgr at 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 librados: starting objecter 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 Processor -- start 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- start start 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 librados: setting wanted keys 2024-09-11T02:52:35.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 librados: calling monclient init 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient: init 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient: _add_conns ranks=[8,2,3,5,6,4,7,0,1] 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa070139b10 0x7fa07013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): picked mon.noname-e con 0x7fa070139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139ef0 0x7fa07013abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): picked mon.noname-b con 0x7fa070139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa07013a2d0 0x7fa07013b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): picked mon.noname-g con 0x7fa07013a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa070143460 con 0x7fa070139ef0 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa0700a88a0 con 0x7fa07013a2d0 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa0701435e0 con 0x7fa070139b10 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa075100640 10 monclient(hunting): authenticate will time out at 1109.767436s 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa070139b10 0x7fa07013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139ef0 0x7fa07013abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa070139b10 0x7fa07013a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44302/0 (socket says 172.21.15.98:44302) 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 -- 172.21.15.98:0/208031262 learned_addr learned my addr 172.21.15.98:0/208031262 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): get_auth_request con 0x7fa070139b10 auth_method 0 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 10 monclient(hunting): get_auth_request con 0x7fa070139ef0 auth_method 0 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06f59e640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa07013a2d0 0x7fa07013b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06f59e640 10 monclient(hunting): get_auth_request con 0x7fa07013a2d0 auth_method 0 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06f59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient(hunting): handle_auth_done global_id 23543 payload 824 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa07013a2d0 msgr2=0x7fa07013b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa07013a2d0 0x7fa07013b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139ef0 msgr2=0x7fa07013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa070139ef0 0x7fa07013abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient: _finish_hunting 0 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 monclient: found mon.noname-e 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:35.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.882+0000 7fa06ed9d640 1 -- 172.21.15.98:0/208031262 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07013d290 con 0x7fa070139b10 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06ed9d640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa070139b10 0x7fa07013a6b0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fa060009f60 tx=0x7fa060009f90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 client.?.objecter ms_handle_connect 0x7fa070139b10 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa060018020 con 0x7fa070139b10 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: dump: 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.908 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:35.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0600148f0 con 0x7fa070139b10 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 5 monclient: authenticate success, global_id 23543 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 10 monclient: _renew_subs 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa070135a10 con 0x7fa070139b10 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0600133e0 con 0x7fa070139b10 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: dump: 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.910 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.911 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06d59a640 4 set_mon_vals no callback set 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 10 client.23543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 10 monclient: _renew_subs 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0700aa510 con 0x7fa070139b10 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 1 librados: init done 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063640 0x7fa070063a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa07013cc30 con 0x7fa070063640 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063640 0x7fa070063a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa060014480 con 0x7fa070139b10 2024-09-11T02:52:35.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient: get_auth_request con 0x7fa070063640 auth_method 0 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa05c03df40 0x7fa05c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 10 monclient(hunting): handle_auth_done global_id 14370 payload 824 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 1 -- 172.21.15.98:0/208031262 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa07006e680 con 0x7fa070063640 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(92..92 src has 1..92) ==== 7856+0+0 (secure 0 0 0) 0x7fa060050980 con 0x7fa070139b10 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 client.23543.objecter ms_dispatch 0x7fa070000ba0 osd_map(92..92 src has 1..92) 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 3 client.23543.objecter handle_osd_map got epochs [92,92] > 0 2024-09-11T02:52:35.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 3 client.23543.objecter handle_osd_map decoding full epoch 92 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 20 client.23543.objecter dump_active .. 0 homeless 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 client.23543.objecter ms_handle_connect 0x7fa070063640 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 client.23543.objecter resend_mon_ops 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06e59c640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063640 0x7fa070063a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0700a5f40 tx=0x7fa064010c80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa064018020 con 0x7fa070063640 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: dump: 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06f59e640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa05c03df40 0x7fa05c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06f59e640 10 monclient: get_auth_request con 0x7fa05c03df40 auth_method 0 2024-09-11T02:52:35.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa064004240 con 0x7fa070063640 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063640 msgr2=0x7fa070063a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063640 0x7fa070063a20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0700a5f40 tx=0x7fa064010c80 comp rx=0 tx=0).stop 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063640 msgr2=0x7fa070063a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063640 0x7fa070063a20 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa0700a5f40 tx=0x7fa064010c80 comp rx=0 tx=0).stop 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa0577fe640 10 client.23543.objecter ms_handle_connect 0x7fa05c03df40 2024-09-11T02:52:35.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.886+0000 7fa06f59e640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa05c03df40 0x7fa05c040400 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7fa0680ad000 tx=0x7fa0680a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063f60 0x7fa0700a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa07006ca10 con 0x7fa070063f60 2024-09-11T02:52:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063f60 0x7fa0700a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:35.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient: get_auth_request con 0x7fa070063f60 auth_method 0 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 10 monclient(hunting): handle_auth_done global_id 14379 payload 824 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 1 -- 172.21.15.98:0/208031262 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa070072600 con 0x7fa070063f60 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 10 client.23543.objecter ms_handle_connect 0x7fa070063f60 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06e59c640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063f60 0x7fa0700a0e60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa064000c10 tx=0x7fa0640046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 10 client.23543.objecter resend_mon_ops 2024-09-11T02:52:35.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa064018040 con 0x7fa070063f60 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 10 monclient: dump: 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:35.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9255 (secure 0 0 0) 0x7fa06400edd0 con 0x7fa070063f60 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:35.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063f60 msgr2=0x7fa0700a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063f60 0x7fa0700a0e60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa064000c10 tx=0x7fa0640046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063f60 msgr2=0x7fa0700a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa0577fe640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa070063f60 0x7fa0700a0e60 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa064000c10 tx=0x7fa0640046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:35.922 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 35946, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:35.923 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.924 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.925 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:35.926 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:35.927 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.928 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.929 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.933 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:35.936 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.937 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:35.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:35.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:35.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:35.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:35.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:35.948 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:35.949 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:35.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 10 librados: watch_flush enter 2024-09-11T02:52:35.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 10 librados: watch_flush exit 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 20 client.23543.objecter shutdown clearing up homeless session... 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 10 client.23543.objecter successfully canceled tick 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa05c03df40 msgr2=0x7fa05c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa05c03df40 0x7fa05c040400 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7fa0680ad000 tx=0x7fa0680a6000 comp rx=0 tx=0).stop 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 10 monclient: shutdown 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa070139b10 msgr2=0x7fa07013a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 --2- 172.21.15.98:0/208031262 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa070139b10 0x7fa07013a6b0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fa060009f60 tx=0x7fa060009f90 comp rx=0 tx=0).stop 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06f59e640 1 -- 172.21.15.98:0/208031262 reap_dead start 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa06f59e640 1 -- 172.21.15.98:0/208031262 reap_dead start 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 shutdown_connections 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 >> 172.21.15.98:0/208031262 conn(0x7fa0700ac790 msgr2=0x7fa0700a7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 shutdown_connections 2024-09-11T02:52:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 -- 172.21.15.98:0/208031262 wait complete. 2024-09-11T02:52:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:35.894+0000 7fa075100640 1 librados: shutdown 2024-09-11T02:52:35.952 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:35.952 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-09-11T02:52:35.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:52:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.062+0000 7fad6b61e640 10 monclient: get_monmap_and_config 2024-09-11T02:52:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.062+0000 7fad6b61e640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.062+0000 7fad6b61e640 10 monclient: monmap: 2024-09-11T02:52:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:36.069194+0000 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:36.069194+0000 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:36.069 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:36.070 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 Processor -- start 2024-09-11T02:52:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 -- start start 2024-09-11T02:52:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient: init 2024-09-11T02:52:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient: _add_conns ranks=[9,5,4,2,7,0,3,6,8,10,1] 2024-09-11T02:52:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad64139b10 0x7fad64072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): picked mon.noname-a con 0x7fad64139b10 addr v2:172.21.15.98:3300/0 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139ef0 0x7fad640a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): picked mon.noname-h con 0x7fad64139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad6413a2d0 0x7fad640a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): picked mon.noname-c con 0x7fad6413a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fad640702b0 con 0x7fad64139b10 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fad64071bf0 con 0x7fad6413a2d0 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fad64071350 con 0x7fad64139ef0 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad693ba640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7fad64139b10 msgr2=0x7fad64072600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad6b61e640 10 monclient(hunting): authenticate will time out at 1109.949584s 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad693ba640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fad64139b10 0x7fad64072600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad6413a2d0 0x7fad640a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad6413a2d0 0x7fad640a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57906/0 (socket says 172.21.15.98:57906) 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 -- 172.21.15.98:0/2400789653 learned_addr learned my addr 172.21.15.98:0/2400789653 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): get_auth_request con 0x7fad6413a2d0 auth_method 0 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad68bb9640 1 --2- 172.21.15.98:0/2400789653 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139ef0 0x7fad640a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient(hunting): handle_auth_done global_id 23134 payload 274 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 -- 172.21.15.98:0/2400789653 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139ef0 msgr2=0x7fad640a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 --2- 172.21.15.98:0/2400789653 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139ef0 0x7fad640a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 -- 172.21.15.98:0/2400789653 >> v2:172.21.15.98:3300/0 conn(0x7fad64139b10 msgr2=0x7fad64072600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 --2- 172.21.15.98:0/2400789653 >> v2:172.21.15.98:3300/0 conn(0x7fad64139b10 0x7fad64072600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 monclient: found mon.noname-c 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad69bbb640 1 -- 172.21.15.98:0/2400789653 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad640a9620 con 0x7fad6413a2d0 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad68bb9640 10 monclient: get_auth_request con 0x7fad64139ef0 auth_method 0 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.066+0000 7fad68bb9640 1 --2- 172.21.15.98:0/2400789653 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139ef0 0x7fad640a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad69bbb640 1 --2- 172.21.15.98:0/2400789653 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad6413a2d0 0x7fad640a7f50 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009e80 tx=0x7fad5c01c980 comp rx=0 tx=0).ready entity=mon.4 client_cookie=959cc0e9f609e6a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 1 -- 172.21.15.98:0/2400789653 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad5c0090d0 con 0x7fad6413a2d0 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: dump: 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.077 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 5 monclient: authenticate success, global_id 23134 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 1 -- 172.21.15.98:0/2400789653 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad5c0a55c0 con 0x7fad6413a2d0 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 1 -- 172.21.15.98:0/2400789653 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad5c009270 con 0x7fad6413a2d0 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 4 set_mon_vals no callback set 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad5bfff640 10 monclient: dump: 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.079 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient: shutdown 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- 172.21.15.98:0/2400789653 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad6413a2d0 msgr2=0x7fad640a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 --2- 172.21.15.98:0/2400789653 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad6413a2d0 0x7fad640a7f50 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009e80 tx=0x7fad5c01c980 comp rx=0 tx=0).stop 2024-09-11T02:52:36.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- 172.21.15.98:0/2400789653 shutdown_connections 2024-09-11T02:52:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 --2- 172.21.15.98:0/2400789653 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad6413a2d0 0x7fad640a7f50 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 --2- 172.21.15.98:0/2400789653 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139ef0 0x7fad640a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 --2- 172.21.15.98:0/2400789653 >> v2:172.21.15.98:3300/0 conn(0x7fad64139b10 0x7fad64072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- 172.21.15.98:0/2400789653 >> 172.21.15.98:0/2400789653 conn(0x7fad640ac790 msgr2=0x7fad640a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- 172.21.15.98:0/2400789653 shutdown_connections 2024-09-11T02:52:36.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- 172.21.15.98:0/2400789653 wait complete. 2024-09-11T02:52:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient: monmap: 2024-09-11T02:52:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:36.088 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 librados: starting msgr at 2024-09-11T02:52:36.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 librados: starting objecter 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 Processor -- start 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- start start 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 librados: setting wanted keys 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 librados: calling monclient init 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient: init 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient: _add_conns ranks=[5,4,0,7,1,8,2,6,3] 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139b10 0x7fad641d1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): picked mon.noname-h con 0x7fad64139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad64139ef0 0x7fad641d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): picked mon.noname-c con 0x7fad64139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad6413a2d0 0x7fad6413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): picked mon.noname-a con 0x7fad6413a2d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fad64143460 con 0x7fad6413a2d0 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fad640a88a0 con 0x7fad64139ef0 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fad641435e0 con 0x7fad64139b10 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad6b61e640 10 monclient(hunting): authenticate will time out at 1109.955146s 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad69bbb640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad6413a2d0 msgr2=0x7fad6413a6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad69bbb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad6413a2d0 0x7fad6413a6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad64139ef0 0x7fad641d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad64139ef0 0x7fad641d1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57916/0 (socket says 172.21.15.98:57916) 2024-09-11T02:52:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 -- 172.21.15.98:0/4173606959 learned_addr learned my addr 172.21.15.98:0/4173606959 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): get_auth_request con 0x7fad64139ef0 auth_method 0 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad693ba640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139b10 0x7fad641d1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad693ba640 10 monclient(hunting): get_auth_request con 0x7fad64139b10 auth_method 0 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad693ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad693ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad693ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient(hunting): handle_auth_done global_id 23143 payload 824 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139b10 msgr2=0x7fad641d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fad64139b10 0x7fad641d1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad6413a2d0 msgr2=0x7fad6413a6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fad6413a2d0 0x7fad6413a6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 monclient: found mon.noname-c 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 -- 172.21.15.98:0/4173606959 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6413dda0 con 0x7fad64139ef0 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad68bb9640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad64139ef0 0x7fad641d1ca0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fad5400c7f0 tx=0x7fad5400ccb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad59ffb640 10 client.?.objecter ms_handle_connect 0x7fad64139ef0 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.070+0000 7fad59ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad54007ca0 con 0x7fad64139ef0 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: dump: 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad540113f0 con 0x7fad64139ef0 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 5 monclient: authenticate success, global_id 23143 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 10 monclient: _renew_subs 2024-09-11T02:52:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad64135810 con 0x7fad64139ef0 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad54011a10 con 0x7fad64139ef0 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 10 client.23143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad5b7fe640 4 set_mon_vals no callback set 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: dump: 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 10 monclient: _renew_subs 2024-09-11T02:52:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6413dac0 con 0x7fad64139ef0 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 1 librados: init done 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413ac20 0x7fad6413b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fad6413c880 con 0x7fad6413ac20 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413ac20 0x7fad6413b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient: get_auth_request con 0x7fad6413ac20 auth_method 0 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fad54011bb0 con 0x7fad64139ef0 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fad4803df70 0x7fad48040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(92..92 src has 1..92) ==== 7856+0+0 (secure 0 0 0) 0x7fad54050040 con 0x7fad64139ef0 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 client.23143.objecter ms_dispatch 0x7fad64000ba0 osd_map(92..92 src has 1..92) 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 3 client.23143.objecter handle_osd_map got epochs [92,92] > 0 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 3 client.23143.objecter handle_osd_map decoding full epoch 92 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 20 client.23143.objecter dump_active .. 0 homeless 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad69bbb640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fad4803df70 0x7fad48040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad69bbb640 10 monclient: get_auth_request con 0x7fad4803df70 auth_method 0 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 10 monclient(hunting): handle_auth_done global_id 23552 payload 824 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 1 -- 172.21.15.98:0/4173606959 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad6406e680 con 0x7fad6413ac20 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 client.23143.objecter ms_handle_connect 0x7fad6413ac20 2024-09-11T02:52:36.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 client.23143.objecter resend_mon_ops 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad693ba640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413ac20 0x7fad6413b020 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fad60003990 tx=0x7fad6000db90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad60018070 con 0x7fad6413ac20 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: dump: 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.100 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fad60005a50 con 0x7fad6413ac20 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413ac20 msgr2=0x7fad6413b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413ac20 0x7fad6413b020 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fad60003990 tx=0x7fad6000db90 comp rx=0 tx=0).stop 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413ac20 msgr2=0x7fad6413b020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413ac20 0x7fad6413b020 secure :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fad60003990 tx=0x7fad6000db90 comp rx=0 tx=0).stop 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad59ffb640 10 client.23143.objecter ms_handle_connect 0x7fad4803df70 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.074+0000 7fad69bbb640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fad4803df70 0x7fad48040430 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fad5c0ae000 tx=0x7fad5c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413b560 0x7fad640a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fad6406ca10 con 0x7fad6413b560 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413b560 0x7fad640a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient: get_auth_request con 0x7fad6413b560 auth_method 0 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 10 monclient(hunting): handle_auth_done global_id 23561 payload 824 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 1 -- 172.21.15.98:0/4173606959 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fad64072600 con 0x7fad6413b560 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 10 client.23143.objecter ms_handle_connect 0x7fad6413b560 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 10 client.23143.objecter resend_mon_ops 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad693ba640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413b560 0x7fad640a0e60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fad6000b130 tx=0x7fad600059e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fad60018040 con 0x7fad6413b560 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 10 monclient: dump: 2024-09-11T02:52:36.103 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.104 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9256 (secure 0 0 0) 0x7fad60003d80 con 0x7fad6413b560 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413b560 msgr2=0x7fad640a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413b560 0x7fad640a0e60 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fad6000b130 tx=0x7fad600059e0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413b560 msgr2=0x7fad640a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad59ffb640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fad6413b560 0x7fad640a0e60 secure :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fad6000b130 tx=0x7fad600059e0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.105 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 171875, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:36.106 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.107 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.108 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:36.109 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.110 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.111 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:36.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:36.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.121 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:36.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.126 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.127 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:36.128 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:36.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 10 librados: watch_flush enter 2024-09-11T02:52:36.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 10 librados: watch_flush exit 2024-09-11T02:52:36.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 20 client.23143.objecter shutdown clearing up homeless session... 2024-09-11T02:52:36.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 10 client.23143.objecter successfully canceled tick 2024-09-11T02:52:36.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fad4803df70 msgr2=0x7fad48040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fad4803df70 0x7fad48040430 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fad5c0ae000 tx=0x7fad5c0a7070 comp rx=0 tx=0).stop 2024-09-11T02:52:36.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 10 monclient: shutdown 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad64139ef0 msgr2=0x7fad641d1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 --2- 172.21.15.98:0/4173606959 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fad64139ef0 0x7fad641d1ca0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fad5400c7f0 tx=0x7fad5400ccb0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad69bbb640 1 -- 172.21.15.98:0/4173606959 reap_dead start 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad69bbb640 1 -- 172.21.15.98:0/4173606959 reap_dead start 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 shutdown_connections 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 >> 172.21.15.98:0/4173606959 conn(0x7fad640ac790 msgr2=0x7fad640aa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 shutdown_connections 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 -- 172.21.15.98:0/4173606959 wait complete. 2024-09-11T02:52:36.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.082+0000 7fad6b61e640 1 librados: shutdown 2024-09-11T02:52:36.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:36.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-09-11T02:52:36.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:52:36.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient: get_monmap_and_config 2024-09-11T02:52:36.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient: monmap: 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:36.249592+0000 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:36.249592+0000 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:36.249 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:36.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 1 Processor -- start 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 1 -- start start 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient: init 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient: _add_conns ranks=[6,8,1,10,5,7,2,4,3,0,9] 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c04139b80 0x7f8c04139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient(hunting): picked mon.noname-d con 0x7f8c04139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c0406fe50 0x7f8c040a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient(hunting): picked mon.noname-e con 0x7f8c0406fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8c04072640 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c04071370 con 0x7f8c04072640 2024-09-11T02:52:36.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8c040719c0 con 0x7f8c04139b80 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8c04071c10 con 0x7f8c0406fe50 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c08f8a640 10 monclient(hunting): authenticate will time out at 1110.129846s 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c0406fe50 0x7f8c040a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c04139b80 0x7f8c04139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c04139b80 0x7f8c04139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:48576/0 (socket says 172.21.15.98:48576) 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 -- 172.21.15.98:0/1914655639 learned_addr learned my addr 172.21.15.98:0/1914655639 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): get_auth_request con 0x7f8c04139b80 auth_method 0 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 10 monclient(hunting): get_auth_request con 0x7f8c0406fe50 auth_method 0 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient(hunting): handle_auth_done global_id 14388 payload 274 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 -- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c0406fe50 msgr2=0x7f8c040a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 --2- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c0406fe50 0x7f8c040a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 -- 172.21.15.98:0/1914655639 >> v1:172.21.15.190:6789/0 conn(0x7f8c04072640 legacy=0x7f8c040a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 monclient: found mon.noname-d 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c0259c640 1 -- 172.21.15.98:0/1914655639 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c040a9600 con 0x7f8c04139b80 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.246+0000 7f8c01d9b640 1 --2- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c0406fe50 0x7f8c040a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0259c640 1 --2- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c04139b80 0x7f8c04139f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80098d0 tx=0x7f8bf801c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=34d8a190214c1d21 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 1 -- 172.21.15.98:0/1914655639 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80b1070 con 0x7f8c04139b80 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: dump: 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 1 -- 172.21.15.98:0/1914655639 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bf80aa7b0 con 0x7f8c04139b80 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 1 -- 172.21.15.98:0/1914655639 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80aae50 con 0x7f8c04139b80 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 5 monclient: authenticate success, global_id 14388 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0159a640 10 monclient: dump: 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.260 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 4 set_mon_vals no callback set 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient: shutdown 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c04139b80 msgr2=0x7f8c04139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 --2- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c04139b80 0x7f8c04139f60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80098d0 tx=0x7f8bf801c440 comp rx=0 tx=0).stop 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- 172.21.15.98:0/1914655639 shutdown_connections 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 --2- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c0406fe50 0x7f8c040a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 --2- 172.21.15.98:0/1914655639 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c04139b80 0x7f8c04139f60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- 172.21.15.98:0/1914655639 >> 172.21.15.98:0/1914655639 conn(0x7f8c040ac730 msgr2=0x7f8c040a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- 172.21.15.98:0/1914655639 shutdown_connections 2024-09-11T02:52:36.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- 172.21.15.98:0/1914655639 wait complete. 2024-09-11T02:52:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient: monmap: 2024-09-11T02:52:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:36.268 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 librados: starting msgr at 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 librados: starting objecter 2024-09-11T02:52:36.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 Processor -- start 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- start start 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 librados: setting wanted keys 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 librados: calling monclient init 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient: init 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient: _add_conns ranks=[5,3,4,0,2,8,6,1,7] 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0406fe50 0x7f8c0413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): picked mon.noname-h con 0x7f8c0406fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c04072640 0x7f8c0413ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): picked mon.noname-g con 0x7f8c04072640 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8c04139b80 0x7f8c0413b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): picked mon.noname-c con 0x7f8c04139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:36.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8c04144450 con 0x7f8c04072640 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8c040a8870 con 0x7f8c04139b80 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8c041445d0 con 0x7f8c0406fe50 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c08f8a640 10 monclient(hunting): authenticate will time out at 1110.135493s 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8c04139b80 0x7f8c0413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8c04139b80 0x7f8c0413b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57922/0 (socket says 172.21.15.98:57922) 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 -- 172.21.15.98:0/999654276 learned_addr learned my addr 172.21.15.98:0/999654276 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): get_auth_request con 0x7f8c04139b80 auth_method 0 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0259c640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0406fe50 0x7f8c0413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c01d9b640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c04072640 0x7f8c0413ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0259c640 10 monclient(hunting): get_auth_request con 0x7f8c0406fe50 auth_method 0 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c01d9b640 10 monclient(hunting): get_auth_request con 0x7f8c04072640 auth_method 0 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient(hunting): handle_auth_done global_id 23152 payload 824 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0406fe50 msgr2=0x7f8c0413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0406fe50 0x7f8c0413a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c04072640 msgr2=0x7f8c0413ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c04072640 0x7f8c0413ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 monclient: found mon.noname-c 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.250+0000 7f8c02d9d640 1 -- 172.21.15.98:0/999654276 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0413cd80 con 0x7f8c04139b80 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c02d9d640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8c04139b80 0x7f8c0413b250 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00c7c0 tx=0x7f8bfc00cc80 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 client.?.objecter ms_handle_connect 0x7f8c04139b80 2024-09-11T02:52:36.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bfc001880 con 0x7f8c04139b80 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: dump: 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.274 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bfc003df0 con 0x7f8c04139b80 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bfc00acc0 con 0x7f8c04139b80 2024-09-11T02:52:36.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c00d99640 4 set_mon_vals no callback set 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: dump: 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.276 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 5 monclient: authenticate success, global_id 23152 2024-09-11T02:52:36.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 10 monclient: _renew_subs 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0413d060 con 0x7f8c04139b80 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 10 client.23152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 10 monclient: _renew_subs 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c04134800 con 0x7f8c04139b80 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 1 librados: init done 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04133f20 0x7f8c0402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0406ca10 con 0x7f8c04133f20 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8bfc00a850 con 0x7f8c04139b80 2024-09-11T02:52:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04133f20 0x7f8c0402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient: get_auth_request con 0x7f8c04133f20 auth_method 0 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be403df40 0x7f8be4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(92..92 src has 1..92) ==== 7856+0+0 (secure 0 0 0) 0x7f8bfc04fc60 con 0x7f8c04139b80 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 client.23152.objecter ms_dispatch 0x7f8c04000ba0 osd_map(92..92 src has 1..92) 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 3 client.23152.objecter handle_osd_map got epochs [92,92] > 0 2024-09-11T02:52:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 3 client.23152.objecter handle_osd_map decoding full epoch 92 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 20 client.23152.objecter dump_active .. 0 homeless 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c01d9b640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be403df40 0x7f8be4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c01d9b640 10 monclient: get_auth_request con 0x7f8be403df40 auth_method 0 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 10 monclient(hunting): handle_auth_done global_id 24238 payload 824 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 1 -- 172.21.15.98:0/999654276 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c040aa120 con 0x7f8c04133f20 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 client.23152.objecter ms_handle_connect 0x7f8c04133f20 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 client.23152.objecter resend_mon_ops 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c0259c640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04133f20 0x7f8c0402d450 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f8c040704e0 tx=0x7f8bf80191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 client.23152.objecter ms_handle_connect 0x7f8be403df40 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8c01d9b640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be403df40 0x7f8be4040400 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009580 tx=0x7f8bf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80bb030 con 0x7f8c04133f20 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: dump: 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8bf8003620 con 0x7f8c04133f20 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04133f20 msgr2=0x7f8c0402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04133f20 0x7f8c0402d450 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f8c040704e0 tx=0x7f8bf80191e0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04133f20 msgr2=0x7f8c0402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.254+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04133f20 0x7f8c0402d450 secure :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f8c040704e0 tx=0x7f8bf80191e0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c08f8a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c08f8a640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04134300 0x7f8c0402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c08f8a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0406e870 con 0x7f8c04134300 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04134300 0x7f8c0402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient: get_auth_request con 0x7f8c04134300 auth_method 0 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 10 monclient(hunting): handle_auth_done global_id 24247 payload 824 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 1 -- 172.21.15.98:0/999654276 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8c040a82b0 con 0x7f8c04134300 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8bf2ffd640 10 client.23152.objecter ms_handle_connect 0x7f8c04134300 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8c0259c640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04134300 0x7f8c0402d990 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8000c10 tx=0x7f8bf80a6000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.262+0000 7f8bf2ffd640 10 client.23152.objecter resend_mon_ops 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80bb040 con 0x7f8c04134300 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 10 monclient: dump: 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.284 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.285 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.286 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.286 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.286 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.286 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.286 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9425 (secure 0 0 0) 0x7f8bf80039e0 con 0x7f8c04134300 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04134300 msgr2=0x7f8c0402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04134300 0x7f8c0402d990 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8000c10 tx=0x7f8bf80a6000 comp rx=0 tx=0).stop 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04134300 msgr2=0x7f8c0402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8bf2ffd640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c04134300 0x7f8c0402d990 secure :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8000c10 tx=0x7f8bf80a6000 comp rx=0 tx=0).stop 2024-09-11T02:52:36.287 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 172031, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:36.288 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.289 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.290 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:36.291 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.292 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.293 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.296 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:36.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:36.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:36.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.303 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:36.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.308 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:36.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.310 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:36.311 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:36.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 10 librados: watch_flush enter 2024-09-11T02:52:36.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 10 librados: watch_flush exit 2024-09-11T02:52:36.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 20 client.23152.objecter shutdown clearing up homeless session... 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 10 client.23152.objecter successfully canceled tick 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be403df40 msgr2=0x7f8be4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be403df40 0x7f8be4040400 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009580 tx=0x7f8bf4008040 comp rx=0 tx=0).stop 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 10 monclient: shutdown 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8c04139b80 msgr2=0x7f8c0413b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 --2- 172.21.15.98:0/999654276 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8c04139b80 0x7f8c0413b250 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00c7c0 tx=0x7f8bfc00cc80 comp rx=0 tx=0).stop 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c02d9d640 1 -- 172.21.15.98:0/999654276 reap_dead start 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c02d9d640 1 -- 172.21.15.98:0/999654276 reap_dead start 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 shutdown_connections 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 >> 172.21.15.98:0/999654276 conn(0x7f8c040ac730 msgr2=0x7f8c04134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 shutdown_connections 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 -- 172.21.15.98:0/999654276 wait complete. 2024-09-11T02:52:36.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.266+0000 7f8c08f8a640 1 librados: shutdown 2024-09-11T02:52:36.314 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:36.314 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-09-11T02:52:36.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:52:36.381 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshoistObjectsStart 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 1 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 10 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 13 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 7 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 14 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 0 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 15 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 11 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 5 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 8 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 6 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 3 0 2024-09-11T02:52:36.382 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 4 0 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 12 0 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 9 0 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2 0 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (120 ms) 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 3 tests from LibRadosListEC (1230 ms total) 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2414 ms) 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] 1 test from LibRadosListNP (2414 ms total) 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: 2024-09-11T02:52:36.383 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [----------] Global test environment tear-down 2024-09-11T02:52:36.384 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [==========] 11 tests from 3 test suites ran. (95350 ms total) 2024-09-11T02:52:36.384 INFO:tasks.workunit.client.0.smithi190.stdout: api_list: [ PASSED ] 11 tests. 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.458+0000 7fe6ea33a640 10 monclient: get_monmap_and_config 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.458+0000 7fe6ea33a640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.458+0000 7fe6ea33a640 10 monclient: monmap: 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:36.464059+0000 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:36.464059+0000 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:36.464 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.458+0000 7fe6ea33a640 1 Processor -- start 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.458+0000 7fe6ea33a640 1 -- start start 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.458+0000 7fe6ea33a640 10 monclient: init 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient: _add_conns ranks=[4,3,10,2,0,7,5,1,8,6,9] 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6e40a5830 0x7fe6e40a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e40a5830 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1770 0x7fe6e40a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): picked mon.noname-g con 0x7fe6e40a1770 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe6e4071ed0 0x7fe6e41375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): picked mon.noname-f con 0x7fe6e4071ed0 addr v2:172.21.15.190:3300/0 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe6e40a6690 con 0x7fe6e4071ed0 2024-09-11T02:52:36.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e40739b0 con 0x7fe6e40a1770 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe6e4071c90 con 0x7fe6e40a5830 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient(hunting): authenticate will time out at 1110.344246s 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6e40a5830 0x7fe6e40a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6e40a5830 0x7fe6e40a5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57932/0 (socket says 172.21.15.98:57932) 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 -- 172.21.15.98:0/2027553783 learned_addr learned my addr 172.21.15.98:0/2027553783 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): get_auth_request con 0x7fe6e40a5830 auth_method 0 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 1 --2- 172.21.15.98:0/2027553783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1770 0x7fe6e40a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 10 monclient(hunting): get_auth_request con 0x7fe6e40a1770 auth_method 0 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e88d7640 1 --2- 172.21.15.98:0/2027553783 >> v2:172.21.15.190:3300/0 conn(0x7fe6e4071ed0 0x7fe6e41375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e88d7640 10 monclient(hunting): get_auth_request con 0x7fe6e4071ed0 auth_method 0 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e88d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e88d7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e88d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_done global_id 23161 payload 274 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 -- 172.21.15.98:0/2027553783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1770 msgr2=0x7fe6e40a6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 --2- 172.21.15.98:0/2027553783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1770 0x7fe6e40a6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 -- 172.21.15.98:0/2027553783 >> v2:172.21.15.190:3300/0 conn(0x7fe6e4071ed0 msgr2=0x7fe6e41375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 --2- 172.21.15.98:0/2027553783 >> v2:172.21.15.190:3300/0 conn(0x7fe6e4071ed0 0x7fe6e41375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 monclient: found mon.noname-c 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 -- 172.21.15.98:0/2027553783 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e4073c90 con 0x7fe6e40a5830 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e88d7640 1 --2- 172.21.15.98:0/2027553783 >> v2:172.21.15.190:3300/0 conn(0x7fe6e4071ed0 0x7fe6e41375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e3fff640 1 --2- 172.21.15.98:0/2027553783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6e40a5830 0x7fe6e40a5c10 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40098a0 tx=0x7fe6d401cba0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=4fcf5885ac3d3f8d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 1 -- 172.21.15.98:0/2027553783 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6d40b0070 con 0x7fe6e40a5830 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: dump: 2024-09-11T02:52:36.473 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 1 -- 172.21.15.98:0/2027553783 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d40aa810 con 0x7fe6e40a5830 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 1 -- 172.21.15.98:0/2027553783 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6d40a9300 con 0x7fe6e40a5830 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6e37fe640 10 monclient: dump: 2024-09-11T02:52:36.475 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.476 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 5 monclient: authenticate success, global_id 23161 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 4 set_mon_vals no callback set 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 10 monclient: shutdown 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- 172.21.15.98:0/2027553783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6e40a5830 msgr2=0x7fe6e40a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/2027553783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6e40a5830 0x7fe6e40a5c10 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40098a0 tx=0x7fe6d401cba0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- 172.21.15.98:0/2027553783 shutdown_connections 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/2027553783 >> v2:172.21.15.190:3300/0 conn(0x7fe6e4071ed0 0x7fe6e41375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/2027553783 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1770 0x7fe6e40a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/2027553783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe6e40a5830 0x7fe6e40a5c10 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- 172.21.15.98:0/2027553783 >> 172.21.15.98:0/2027553783 conn(0x7fe6e40aa420 msgr2=0x7fe6e40a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- 172.21.15.98:0/2027553783 shutdown_connections 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.462+0000 7fe6ea33a640 1 -- 172.21.15.98:0/2027553783 wait complete. 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: monmap: 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:36.478 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.479 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 librados: starting msgr at 2024-09-11T02:52:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 librados: starting objecter 2024-09-11T02:52:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 Processor -- start 2024-09-11T02:52:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 -- start start 2024-09-11T02:52:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 librados: setting wanted keys 2024-09-11T02:52:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 librados: calling monclient init 2024-09-11T02:52:36.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: init 2024-09-11T02:52:36.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: _add_conns ranks=[8,0,3,5,4,2,1,6,7] 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe6e4071ed0 0x7fe6e41d1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): picked mon.noname-e con 0x7fe6e4071ed0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e40a5830 0x7fe6e41d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e40a5830 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e41d1bf0 0x7fe6e406edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): picked mon.noname-g con 0x7fe6e41d1bf0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e4140de0 con 0x7fe6e40a5830 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e4140c60 con 0x7fe6e41d1bf0 2024-09-11T02:52:36.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe6e4140f60 con 0x7fe6e4071ed0 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): authenticate will time out at 1110.349603s 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e40a5830 msgr2=0x7fe6e41d16b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e40a5830 0x7fe6e41d16b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe6e4071ed0 0x7fe6e41d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe6e4071ed0 0x7fe6e41d1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44344/0 (socket says 172.21.15.98:44344) 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 -- 172.21.15.98:0/343199974 learned_addr learned my addr 172.21.15.98:0/343199974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): get_auth_request con 0x7fe6e4071ed0 auth_method 0 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e41d1bf0 0x7fe6e406edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 10 monclient(hunting): get_auth_request con 0x7fe6e41d1bf0 auth_method 0 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e88d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient(hunting): handle_auth_done global_id 23579 payload 824 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e41d1bf0 msgr2=0x7fe6e406edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e41d1bf0 0x7fe6e406edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e40a5830 msgr2=0x7fe6e41d16b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:52:36.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6e40a5830 0x7fe6e41d16b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 monclient: found mon.noname-e 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 -- 172.21.15.98:0/343199974 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e413ceb0 con 0x7fe6e4071ed0 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e3fff640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe6e4071ed0 0x7fe6e41d1080 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40ae040 tx=0x7fe6d4018ee0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 client.?.objecter ms_handle_connect 0x7fe6e4071ed0 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6d40b0040 con 0x7fe6e4071ed0 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: dump: 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 5 monclient: authenticate success, global_id 23579 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d40044b0 con 0x7fe6e4071ed0 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: _renew_subs 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6e413b7d0 con 0x7fe6e4071ed0 2024-09-11T02:52:36.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6d40aac80 con 0x7fe6e4071ed0 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e2ffd640 4 set_mon_vals no callback set 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 client.23579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 10 monclient: dump: 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: _renew_subs 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6e413dcf0 con 0x7fe6e4071ed0 2024-09-11T02:52:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 librados: init done 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1750 0x7fe6e402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e406ca10 con 0x7fe6e40a1750 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe6d40b3070 con 0x7fe6e4071ed0 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1750 0x7fe6e402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient: get_auth_request con 0x7fe6e40a1750 auth_method 0 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6e17fa640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6c403df60 0x7fe6c4040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_done global_id 19605 payload 824 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.466+0000 7fe6dbfff640 1 -- 172.21.15.98:0/343199974 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6e40a30e0 con 0x7fe6e40a1750 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(93..93 src has 1..93) ==== 8614+0+0 (secure 0 0 0) 0x7fe6d40e9da0 con 0x7fe6e4071ed0 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 client.23579.objecter ms_dispatch 0x7fe6e4000ba0 osd_map(93..93 src has 1..93) 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 3 client.23579.objecter handle_osd_map got epochs [93,93] > 0 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 3 client.23579.objecter handle_osd_map decoding full epoch 93 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e88d7640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6c403df60 0x7fe6c4040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e88d7640 10 monclient: get_auth_request con 0x7fe6c403df60 auth_method 0 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 20 client.23579.objecter dump_active .. 0 homeless 2024-09-11T02:52:36.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 client.23579.objecter ms_handle_connect 0x7fe6e40a1750 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 client.23579.objecter resend_mon_ops 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6dbfff640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1750 0x7fe6e402d450 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4070460 tx=0x7fe6cc00ec00 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6cc017070 con 0x7fe6e40a1750 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 monclient: dump: 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.498 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6cc00ee50 con 0x7fe6e40a1750 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1750 msgr2=0x7fe6e402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1750 0x7fe6e402d450 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4070460 tx=0x7fe6cc00ec00 comp rx=0 tx=0).stop 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1750 msgr2=0x7fe6e402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e40a1750 0x7fe6e402d450 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e88d7640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6c403df60 0x7fe6c4040420 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc007440 tx=0x7fe6dc009850 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.470+0000 7fe6e17fa640 10 client.23579.objecter ms_handle_connect 0x7fe6c403df60 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6ea33a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e402d990 0x7fe6e402dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6ea33a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e406e870 con 0x7fe6e402d990 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e402d990 0x7fe6e402dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 10 monclient: get_auth_request con 0x7fe6e402d990 auth_method 0 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.474+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6dbfff640 10 monclient(hunting): handle_auth_done global_id 19614 payload 824 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6dbfff640 1 -- 172.21.15.98:0/343199974 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6e41375e0 con 0x7fe6e402d990 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6dbfff640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e402d990 0x7fe6e402dda0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc019000 tx=0x7fe6cc004740 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 10 client.23579.objecter ms_handle_connect 0x7fe6e402d990 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 10 client.23579.objecter resend_mon_ops 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6cc017070 con 0x7fe6e402d990 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 10 monclient: dump: 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.502 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7fe6cc006620 con 0x7fe6e402d990 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e402d990 msgr2=0x7fe6e402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e402d990 0x7fe6e402dda0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc019000 tx=0x7fe6cc004740 comp rx=0 tx=0).stop 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e402d990 msgr2=0x7fe6e402dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e17fa640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe6e402d990 0x7fe6e402dda0 secure :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc019000 tx=0x7fe6cc004740 comp rx=0 tx=0).stop 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:36.503 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 172223, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.504 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.505 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.506 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:36.507 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.508 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.509 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:36.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:36.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:36.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.526 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:36.527 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 10 librados: watch_flush enter 2024-09-11T02:52:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 10 librados: watch_flush exit 2024-09-11T02:52:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 20 client.23579.objecter shutdown clearing up homeless session... 2024-09-11T02:52:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 10 client.23579.objecter successfully canceled tick 2024-09-11T02:52:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6c403df60 msgr2=0x7fe6c4040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe6c403df60 0x7fe6c4040420 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc007440 tx=0x7fe6dc009850 comp rx=0 tx=0).stop 2024-09-11T02:52:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 10 monclient: shutdown 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe6e4071ed0 msgr2=0x7fe6e41d1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 --2- 172.21.15.98:0/343199974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe6e4071ed0 0x7fe6e41d1080 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fe6d40ae040 tx=0x7fe6d4018ee0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e88d7640 1 -- 172.21.15.98:0/343199974 reap_dead start 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6e88d7640 1 -- 172.21.15.98:0/343199974 reap_dead start 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 shutdown_connections 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 >> 172.21.15.98:0/343199974 conn(0x7fe6e40aa420 msgr2=0x7fe6e40a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 shutdown_connections 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 -- 172.21.15.98:0/343199974 wait complete. 2024-09-11T02:52:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.478+0000 7fe6ea33a640 1 librados: shutdown 2024-09-11T02:52:36.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:36.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-09-11T02:52:36.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:52:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient: get_monmap_and_config 2024-09-11T02:52:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient: monmap: 2024-09-11T02:52:36.649 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:36.650708+0000 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:36.650708+0000 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.650 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:36.651 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:36.651 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 1 Processor -- start 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 1 -- start start 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient: init 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient: _add_conns ranks=[0,2,8,4,3,1,10,9,7,6,5] 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9720139b10 addr v1:172.21.15.98:6789/0 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9720072600 0x7f972013ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient(hunting): picked mon.noname-b con 0x7f9720072600 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:36.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f972013ef60 0x7f9720141350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient(hunting): picked mon.noname-e con 0x7f972013ef60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f972013a600 con 0x7f9720139b10 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f97200702b0 con 0x7f9720072600 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9720071bf0 con 0x7f972013ef60 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f9725dbf640 10 monclient(hunting): authenticate will time out at 1110.530877s 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971f7fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f9720139b10 legacy=0x7f9720139ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9720072600 0x7f972013ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f972013ef60 0x7f9720141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9720072600 0x7f972013ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54654/0 (socket says 172.21.15.98:54654) 2024-09-11T02:52:36.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): get_auth_request con 0x7f972013ef60 auth_method 0 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 1 -- 172.21.15.98:0/1362569162 learned_addr learned my addr 172.21.15.98:0/1362569162 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 10 monclient(hunting): get_auth_request con 0x7f9720072600 auth_method 0 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient(hunting): handle_auth_done global_id 23588 payload 274 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 1 -- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9720072600 msgr2=0x7f972013ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 1 --2- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9720072600 0x7f972013ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 1 -- 172.21.15.98:0/1362569162 >> v1:172.21.15.98:6789/0 conn(0x7f9720139b10 legacy=0x7f9720139ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 1 monclient: found mon.noname-e 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971ffff640 1 -- 172.21.15.98:0/1362569162 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9720071350 con 0x7f972013ef60 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.646+0000 7f971effd640 1 --2- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9720072600 0x7f972013ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971ffff640 1 --2- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f972013ef60 0x7f9720141350 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f9718009dc0 tx=0x7f971801c7c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=7ede284cdad8d7a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 1 -- 172.21.15.98:0/1362569162 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f97180b2070 con 0x7f972013ef60 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: dump: 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.659 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 1 -- 172.21.15.98:0/1362569162 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97180a5a90 con 0x7f972013ef60 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 5 monclient: authenticate success, global_id 23588 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 1 -- 172.21.15.98:0/1362569162 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f97180a5cc0 con 0x7f972013ef60 2024-09-11T02:52:36.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 4 set_mon_vals no callback set 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f971e7fc640 10 monclient: dump: 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.661 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 10 monclient: shutdown 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 -- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f972013ef60 msgr2=0x7f9720141350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 --2- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f972013ef60 0x7f9720141350 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f9718009dc0 tx=0x7f971801c7c0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 -- 172.21.15.98:0/1362569162 shutdown_connections 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 --2- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9720072600 0x7f972013ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 --2- 172.21.15.98:0/1362569162 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f972013ef60 0x7f9720141350 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 -- 172.21.15.98:0/1362569162 >> 172.21.15.98:0/1362569162 conn(0x7f97200ac790 msgr2=0x7f97200a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 -- 172.21.15.98:0/1362569162 shutdown_connections 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 -- 172.21.15.98:0/1362569162 wait complete. 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 10 monclient: monmap: 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 librados: starting msgr at 2024-09-11T02:52:36.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 librados: starting objecter 2024-09-11T02:52:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 Processor -- start 2024-09-11T02:52:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 -- start start 2024-09-11T02:52:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 librados: setting wanted keys 2024-09-11T02:52:36.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 librados: calling monclient init 2024-09-11T02:52:36.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 10 monclient: init 2024-09-11T02:52:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 10 monclient: _add_conns ranks=[8,3,7,4,5,6,0,1,2] 2024-09-11T02:52:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.650+0000 7f9725dbf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9720072600 0x7f9720138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): picked mon.noname-e con 0x7f9720072600 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:36.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9720139b10 0x7f97201385c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): picked mon.noname-g con 0x7f9720139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f972013ef60 0x7f97200a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): picked mon.noname-i con 0x7f972013ef60 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9720144230 con 0x7f9720139b10 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f97201440b0 con 0x7f972013ef60 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f97201443b0 con 0x7f9720072600 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): authenticate will time out at 1110.536053s 2024-09-11T02:52:36.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9720072600 0x7f9720138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9720072600 0x7f9720138080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44366/0 (socket says 172.21.15.98:44366) 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 -- 172.21.15.98:0/809148593 learned_addr learned my addr 172.21.15.98:0/809148593 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): get_auth_request con 0x7f9720072600 auth_method 0 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient(hunting): handle_auth_done global_id 23597 payload 824 2024-09-11T02:52:36.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f972013ef60 msgr2=0x7f97200a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f972013ef60 0x7f97200a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9720139b10 msgr2=0x7f97201385c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9720139b10 0x7f97201385c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 monclient: found mon.noname-e 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 -- 172.21.15.98:0/809148593 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9720122c10 con 0x7f9720072600 2024-09-11T02:52:36.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971f7fe640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9720072600 0x7f9720138080 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f971400b580 tx=0x7f971400ba40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 client.?.objecter ms_handle_connect 0x7f9720072600 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9714016070 con 0x7f9720072600 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: dump: 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.676 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9714003bd0 con 0x7f9720072600 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 5 monclient: authenticate success, global_id 23597 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient: _renew_subs 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9720122ef0 con 0x7f9720072600 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9714003dc0 con 0x7f9720072600 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 client.23597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:36.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971dffb640 4 set_mon_vals no callback set 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: dump: 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient: _renew_subs 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f972013eb30 con 0x7f9720072600 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 librados: init done 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a33b0 0x7f97200a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f972006ca10 con 0x7f97200a33b0 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f97140050d0 con 0x7f9720072600 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f970c03df40 0x7f970c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a33b0 0x7f97200a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(93..93 src has 1..93) ==== 8614+0+0 (secure 0 0 0) 0x7f971404f2f0 con 0x7f9720072600 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 client.23597.objecter ms_dispatch 0x7f9720000ba0 osd_map(93..93 src has 1..93) 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 3 client.23597.objecter handle_osd_map got epochs [93,93] > 0 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 3 client.23597.objecter handle_osd_map decoding full epoch 93 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient: get_auth_request con 0x7f97200a33b0 auth_method 0 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 20 client.23597.objecter dump_active .. 0 homeless 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971ffff640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f970c03df40 0x7f970c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971ffff640 10 monclient: get_auth_request con 0x7f970c03df40 auth_method 0 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 10 monclient(hunting): handle_auth_done global_id 19283 payload 824 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 1 -- 172.21.15.98:0/809148593 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f972006e680 con 0x7f97200a33b0 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 client.23597.objecter ms_handle_connect 0x7f97200a33b0 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971effd640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a33b0 0x7f97200a3790 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9710009fd0 tx=0x7f971000ec00 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 client.23597.objecter resend_mon_ops 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 client.23597.objecter ms_handle_connect 0x7f970c03df40 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f971ffff640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f970c03df40 0x7f970c040400 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f97180af000 tx=0x7f9718004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9710017070 con 0x7f97200a33b0 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.654+0000 7f9707fff640 10 monclient: dump: 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.658+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f971000ee50 con 0x7f97200a33b0 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.658+0000 7f9707fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.658+0000 7f9707fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.658+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a33b0 msgr2=0x7f97200a3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.658+0000 7f9707fff640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a33b0 0x7f97200a3790 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9710009fd0 tx=0x7f971000ec00 comp rx=0 tx=0).stop 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.658+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a33b0 msgr2=0x7f97200a3790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.658+0000 7f9707fff640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a33b0 0x7f97200a3790 secure :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f9710009fd0 tx=0x7f971000ec00 comp rx=0 tx=0).stop 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9725dbf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9725dbf640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a3cd0 0x7f972013c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9725dbf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f97200a5440 con 0x7f97200a3cd0 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a3cd0 0x7f972013c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient: get_auth_request con 0x7f97200a3cd0 auth_method 0 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 10 monclient(hunting): handle_auth_done global_id 19292 payload 824 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 1 -- 172.21.15.98:0/809148593 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f972006e870 con 0x7f97200a3cd0 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f971effd640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a3cd0 0x7f972013c7e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9710009fd0 tx=0x7f97100046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9707fff640 10 client.23597.objecter ms_handle_connect 0x7f97200a3cd0 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9707fff640 10 client.23597.objecter resend_mon_ops 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9710017070 con 0x7f97200a3cd0 2024-09-11T02:52:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9707fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9707fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.662+0000 7f9707fff640 10 monclient: dump: 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7f97100065a0 con 0x7f97200a3cd0 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9707fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9707fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a3cd0 msgr2=0x7f972013c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9707fff640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a3cd0 0x7f972013c7e0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f9710009fd0 tx=0x7f97100046d0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9707fff640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a3cd0 msgr2=0x7f972013c7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9707fff640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97200a3cd0 0x7f972013c7e0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 172411, 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:36.689 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:52:36.690 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.691 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.693 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:36.694 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:36.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.696 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.698 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:36.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:36.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:36.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:36.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.721 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:36.723 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 10 librados: watch_flush enter 2024-09-11T02:52:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 10 librados: watch_flush exit 2024-09-11T02:52:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 20 client.23597.objecter shutdown clearing up homeless session... 2024-09-11T02:52:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 10 client.23597.objecter successfully canceled tick 2024-09-11T02:52:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f970c03df40 msgr2=0x7f970c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f970c03df40 0x7f970c040400 secure :-1 s=READY pgs=3392 cs=0 l=1 rev1=1 crypto rx=0x7f97180af000 tx=0x7f9718004ac0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 10 monclient: shutdown 2024-09-11T02:52:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9720072600 msgr2=0x7f9720138080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 --2- 172.21.15.98:0/809148593 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9720072600 0x7f9720138080 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f971400b580 tx=0x7f971400ba40 comp rx=0 tx=0).stop 2024-09-11T02:52:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f971ffff640 1 -- 172.21.15.98:0/809148593 reap_dead start 2024-09-11T02:52:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f971ffff640 1 -- 172.21.15.98:0/809148593 reap_dead start 2024-09-11T02:52:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 shutdown_connections 2024-09-11T02:52:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 >> 172.21.15.98:0/809148593 conn(0x7f97200ac790 msgr2=0x7f97200a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 shutdown_connections 2024-09-11T02:52:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 -- 172.21.15.98:0/809148593 wait complete. 2024-09-11T02:52:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.666+0000 7f9725dbf640 1 librados: shutdown 2024-09-11T02:52:36.726 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:36.726 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-09-11T02:52:36.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:52:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient: get_monmap_and_config 2024-09-11T02:52:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient: monmap: 2024-09-11T02:52:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.849 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:52:36.850234+0000 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:52:36.850234+0000 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.850 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:52:36.851 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:52:36.851 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 1 Processor -- start 2024-09-11T02:52:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 1 -- start start 2024-09-11T02:52:36.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient: init 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient: _add_conns ranks=[1,5,8,3,10,2,7,4,9,0,6] 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc0240a3de0 addr v1:172.21.15.190:6789/0 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 0x7fc0240a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient(hunting): picked mon.noname-h con 0x7fc0240a48d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc02413c9f0 0x7fc02413ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient(hunting): picked mon.noname-e con 0x7fc02413c9f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc024068fa0 con 0x7fc0240a3de0 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc0240a0170 con 0x7fc0240a48d0 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc024068b40 con 0x7fc02413c9f0 2024-09-11T02:52:36.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02addd640 10 monclient(hunting): authenticate will time out at 1110.730790s 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc02413c9f0 0x7fc02413ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc02413c9f0 0x7fc02413ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44376/0 (socket says 172.21.15.98:44376) 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 -- 172.21.15.98:0/3018345807 learned_addr learned my addr 172.21.15.98:0/3018345807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): get_auth_request con 0x7fc02413c9f0 auth_method 0 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc023fff640 1 --2- 172.21.15.98:0/3018345807 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 0x7fc0240a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc023fff640 10 monclient(hunting): get_auth_request con 0x7fc0240a48d0 auth_method 0 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc023fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc023fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc023fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient(hunting): handle_auth_done global_id 23606 payload 274 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 -- 172.21.15.98:0/3018345807 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 msgr2=0x7fc0240a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 --2- 172.21.15.98:0/3018345807 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 0x7fc0240a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 -- 172.21.15.98:0/3018345807 >> v1:172.21.15.190:6789/0 conn(0x7fc0240a3de0 legacy=0x7fc0240a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 monclient: found mon.noname-e 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:52:36.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.846+0000 7fc02937a640 1 -- 172.21.15.98:0/3018345807 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02409eaf0 con 0x7fc02413c9f0 2024-09-11T02:52:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02937a640 1 --2- 172.21.15.98:0/3018345807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc02413c9f0 0x7fc02413ede0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fc014009e80 tx=0x7fc01401c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=88b8643de09b0bb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 1 -- 172.21.15.98:0/3018345807 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0140090d0 con 0x7fc02413c9f0 2024-09-11T02:52:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:52:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: dump: 2024-09-11T02:52:36.862 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.863 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 1 -- 172.21.15.98:0/3018345807 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0140a5b10 con 0x7fc02413c9f0 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 5 monclient: authenticate success, global_id 23606 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 10 monclient: get_monmap_and_config success 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 1 -- 172.21.15.98:0/3018345807 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0140a5cc0 con 0x7fc02413c9f0 2024-09-11T02:52:36.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 4 set_mon_vals no callback set 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc0237fe640 10 monclient: dump: 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 10 monclient: shutdown 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 -- 172.21.15.98:0/3018345807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc02413c9f0 msgr2=0x7fc02413ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 --2- 172.21.15.98:0/3018345807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc02413c9f0 0x7fc02413ede0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fc014009e80 tx=0x7fc01401c910 comp rx=0 tx=0).stop 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 -- 172.21.15.98:0/3018345807 shutdown_connections 2024-09-11T02:52:36.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 --2- 172.21.15.98:0/3018345807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc02413c9f0 0x7fc02413ede0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 --2- 172.21.15.98:0/3018345807 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 0x7fc0240a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 -- 172.21.15.98:0/3018345807 >> 172.21.15.98:0/3018345807 conn(0x7fc0240a4ed0 msgr2=0x7fc0240a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 -- 172.21.15.98:0/3018345807 shutdown_connections 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 -- 172.21.15.98:0/3018345807 wait complete. 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 10 monclient: build_initial_monmap 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 10 monclient: monmap: 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:52:36.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 librados: starting msgr at 2024-09-11T02:52:36.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 librados: starting objecter 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 Processor -- start 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 -- start start 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 librados: setting wanted keys 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 1 librados: calling monclient init 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.850+0000 7fc02addd640 10 monclient: init 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient: _reopen_session rank -1 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient: _add_conns ranks=[7,5,2,0,1,8,3,4,6] 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc0240a3de0 0x7fc02409c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): picked mon.noname-i con 0x7fc0240a3de0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 0x7fc02409c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): picked mon.noname-h con 0x7fc0240a48d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc02413c9f0 0x7fc02409ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): picked mon.noname-b con 0x7fc02413c9f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:52:36.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc024141ed0 con 0x7fc02413c9f0 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc024141d50 con 0x7fc0240a48d0 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc024142050 con 0x7fc0240a3de0 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): _renew_subs 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): authenticate will time out at 1110.736321s 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc02413c9f0 0x7fc02409ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc02413c9f0 0x7fc02409ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:54664/0 (socket says 172.21.15.98:54664) 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 -- 172.21.15.98:0/4138198227 learned_addr learned my addr 172.21.15.98:0/4138198227 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): get_auth_request con 0x7fc02413c9f0 auth_method 0 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc0240a3de0 0x7fc02409c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient(hunting): handle_auth_done global_id 23195 payload 824 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc023fff640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 0x7fc02409c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc0240a3de0 msgr2=0x7fc02409c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc0240a3de0 0x7fc02409c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 msgr2=0x7fc02409c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc0240a48d0 0x7fc02409c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient: _finish_hunting 0 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 monclient: found mon.noname-b 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:52:36.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 -- 172.21.15.98:0/4138198227 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02413c2b0 con 0x7fc02413c9f0 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient: get_auth_request con 0x7fc0240a3de0 auth_method 0 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc0240a3de0 0x7fc02409c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02937a640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc02413c9f0 0x7fc02409ce50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fc014009040 tx=0x7fc0140196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 client.?.objecter ms_handle_connect 0x7fc02413c9f0 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0140bc020 con 0x7fc02413c9f0 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: dump: 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.872 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.873 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: _finish_auth 0 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: _check_auth_tickets 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc014003c30 con 0x7fc02413c9f0 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc014003de0 con 0x7fc02413c9f0 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 5 monclient: authenticate success, global_id 23195 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient: _renew_subs 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc024138270 con 0x7fc02413c9f0 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:52:36.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc022ffd640 4 set_mon_vals no callback set 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: dump: 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 client.23195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient: _renew_subs 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc024131b40 con 0x7fc02413c9f0 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 librados: init done 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b0b0 0x7fc02409b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc02406ca10 con 0x7fc02409b0b0 2024-09-11T02:52:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc014004040 con 0x7fc02413c9f0 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc01003df40 0x7fc010040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(93..93 src has 1..93) ==== 8614+0+0 (secure 0 0 0) 0x7fc0140b3040 con 0x7fc02413c9f0 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 client.23195.objecter ms_dispatch 0x7fc024000ba0 osd_map(93..93 src has 1..93) 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 3 client.23195.objecter handle_osd_map got epochs [93,93] > 0 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 3 client.23195.objecter handle_osd_map decoding full epoch 93 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b0b0 0x7fc02409b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient: get_auth_request con 0x7fc02409b0b0 auth_method 0 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 20 client.23195.objecter dump_active .. 0 homeless 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc023fff640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc01003df40 0x7fc010040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc023fff640 10 monclient: get_auth_request con 0x7fc01003df40 auth_method 0 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 10 monclient(hunting): handle_auth_done global_id 18934 payload 824 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 1 -- 172.21.15.98:0/4138198227 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc02413be30 con 0x7fc02409b0b0 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 client.23195.objecter ms_handle_connect 0x7fc02409b0b0 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc028b79640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b0b0 0x7fc02409b490 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000f70 tx=0x7fc01c00db80 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 client.23195.objecter resend_mon_ops 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 client.23195.objecter ms_handle_connect 0x7fc01003df40 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc023fff640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc01003df40 0x7fc010040400 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7fc02413bbf0 tx=0x7fc018008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc01c00b680 con 0x7fc02409b0b0 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.854+0000 7fc0217fa640 10 monclient: dump: 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.878 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.858+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc01c005070 con 0x7fc02409b0b0 2024-09-11T02:52:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.858+0000 7fc0217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:52:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.858+0000 7fc0217fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:52:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.858+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b0b0 msgr2=0x7fc02409b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.858+0000 7fc0217fa640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b0b0 0x7fc02409b490 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000f70 tx=0x7fc01c00db80 comp rx=0 tx=0).stop 2024-09-11T02:52:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.858+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b0b0 msgr2=0x7fc02409b490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.858+0000 7fc0217fa640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b0b0 0x7fc02409b490 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc02addd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc02addd640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b9d0 0x7fc0241387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc02addd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc02406e870 con 0x7fc02409b9d0 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b9d0 0x7fc0241387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient: get_auth_request con 0x7fc02409b9d0 auth_method 0 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:52:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 10 monclient(hunting): handle_auth_done global_id 18943 payload 824 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 1 -- 172.21.15.98:0/4138198227 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0240a0570 con 0x7fc02409b9d0 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc028b79640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b9d0 0x7fc0241387c0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000fc0 tx=0x7fc01c001130 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc0217fa640 10 client.23195.objecter ms_handle_connect 0x7fc02409b9d0 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.862+0000 7fc0217fa640 10 client.23195.objecter resend_mon_ops 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc01c00bd80 con 0x7fc02409b9d0 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:52:36.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 10 monclient: dump: 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:52:36.884 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7fc01c00b910 con 0x7fc02409b9d0 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b9d0 msgr2=0x7fc0241387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b9d0 0x7fc0241387c0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000fc0 tx=0x7fc01c001130 comp rx=0 tx=0).stop 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b9d0 msgr2=0x7fc0241387c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc0217fa640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc02409b9d0 0x7fc0241387c0 secure :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fc01c000fc0 tx=0x7fc01c001130 comp rx=0 tx=0).stop 2024-09-11T02:52:36.885 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 32, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 172631, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:52:36.886 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.887 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.888 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:52:36.889 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:52:36.890 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:52:36.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:52:36.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:52:36.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:52:36.905 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.906 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:52:36.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:52:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:52:36.908 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:52:36.908 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:52:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 10 librados: watch_flush enter 2024-09-11T02:52:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 10 librados: watch_flush exit 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 20 client.23195.objecter shutdown clearing up homeless session... 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 10 client.23195.objecter successfully canceled tick 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc01003df40 msgr2=0x7fc010040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc01003df40 0x7fc010040400 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7fc02413bbf0 tx=0x7fc018008040 comp rx=0 tx=0).stop 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 10 monclient: shutdown 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc02413c9f0 msgr2=0x7fc02409ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02937a640 1 -- 172.21.15.98:0/4138198227 reap_dead start 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc02413c9f0 0x7fc02409ce50 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fc014009040 tx=0x7fc0140196b0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 shutdown_connections 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 --2- 172.21.15.98:0/4138198227 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc02413c9f0 0x7fc02409ce50 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 >> 172.21.15.98:0/4138198227 conn(0x7fc0240a4ed0 msgr2=0x7fc02413cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 shutdown_connections 2024-09-11T02:52:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 -- 172.21.15.98:0/4138198227 wait complete. 2024-09-11T02:52:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:52:36.866+0000 7fc02addd640 1 librados: shutdown 2024-09-11T02:52:36.911 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T02:52:36.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T02:52:41.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: Running main() from gmock_main.cc 2024-09-11T02:52:41.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-11T02:52:41.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] Global test environment set-up. 2024-09-11T02:52:41.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-11T02:52:41.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-11T02:52:41.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2476 ms) 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3061 ms) 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3045 ms) 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3003 ms) 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2727 ms) 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3023 ms) 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3082 ms) 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-11T02:52:41.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2879 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3069 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2989 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3002 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3041 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2933 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3050 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.Flush (2956 ms) 2024-09-11T02:52:41.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3018 ms) 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3021 ms) 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2960 ms) 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3033 ms) 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (8035 ms) 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3042 ms) 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-11T02:52:41.416 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2929 ms) 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (17246 ms) 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (6748 ms) 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3028 ms) 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3025 ms) 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 26 tests from LibRadosAio (100424 ms total) 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: 2024-09-11T02:52:41.417 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-11T02:52:57.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWritets_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-11T02:52:57.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2020 ms) 2024-09-11T02:52:57.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (27028 ms total) 2024-09-11T02:52:57.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T02:52:57.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-11T02:52:57.373 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4017 ms) 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3979 ms) 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3965 ms) 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11961 ms total) 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (116270 ms total) 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-11T02:52:57.374 INFO:tasks.workunit.client.0.smithi190.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-11T02:52:59.412 INFO:tasks.workunit.client.0.smithi190.stdout:PP (70868 ms total) 2024-09-11T02:52:59.412 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T02:52:59.412 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-11T02:52:59.412 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (159 ms) 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (159 ms total) 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (21 ms) 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (6 ms) 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (27 ms total) 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-11T02:52:59.413 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (52 ms) 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (7 ms) 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (59 ms total) 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (50 ms) 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (53 ms total) 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-11T02:52:59.414 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1033 ms) 2024-09-11T02:52:59.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1033 ms total) 2024-09-11T02:52:59.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: 2024-09-11T02:52:59.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-11T02:52:59.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (118326 ms total) 2024-09-11T02:52:59.415 INFO:tasks.workunit.client.0.smithi190.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-11T02:53:02.392 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-11T02:53:02.392 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-11T02:53:02.392 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-11T02:53:02.392 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-11T02:53:02.392 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-11T02:53:02.392 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2474 ms) 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (17941 ms) 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5951 ms) 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2964 ms) 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3028 ms) 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3031 ms) 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2953 ms) 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-11T02:53:02.393 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3019 ms) 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2974 ms) 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3016 ms) 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3029 ms) 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2953 ms) 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3025 ms) 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (8037 ms) 2024-09-11T02:53:02.394 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3056 ms) 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2920 ms) 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (17236 ms) 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (6759 ms) 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3028 ms) 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3022 ms) 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2934 ms) 2024-09-11T02:53:02.395 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3015 ms) 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3045 ms) 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3007 ms) 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2953 ms) 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3038 ms) 2024-09-11T02:53:02.396 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-11T02:54:06.912 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T02:54:06.913 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T02:54:06.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T02:54:06.918 INFO:tasks.ceph.mon.a:Started 2024-09-11T02:54:06.918 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T02:54:06.918 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T02:54:06.918 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T02:54:21.919 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T02:54:21.920 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T02:54:21.920 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:54:21.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:54:22.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient: get_monmap_and_config 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient: monmap: 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:22.055037+0000 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:22.055037+0000 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.054 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:22.055 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 Processor -- start 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 -- start start 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient: init 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient: _add_conns ranks=[5,2,6,1,8,9,10,4,0,3,7] 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2ec806f6c0 0x7f2ec8072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): picked mon.noname-h con 0x7f2ec806f6c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:22.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806faa0 0x7f2ec806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec806faa0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec81397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): picked mon.noname-d con 0x7f2ec80703c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec8071ff0 con 0x7f2ec806faa0 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f2ec80a8750 con 0x7f2ec806f6c0 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2ec80716c0 con 0x7f2ec80703c0 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecfe70640 10 monclient(hunting): authenticate will time out at 1215.935387s 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806faa0 0x7f2ec806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806faa0 0x7f2ec806fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50750/0 (socket says 172.21.15.98:50750) 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 1 -- 172.21.15.98:0/3986664353 learned_addr learned my addr 172.21.15.98:0/3986664353 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): get_auth_request con 0x7f2ec806faa0 auth_method 0 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec81397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 10 monclient(hunting): get_auth_request con 0x7f2ec80703c0 auth_method 0 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ece40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.049+0000 7f2ecd40b640 10 monclient(hunting): handle_auth_done global_id 23357 payload 274 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 1 -- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 msgr2=0x7f2ec81397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec81397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 1 -- 172.21.15.98:0/3986664353 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2ec806f6c0 msgr2=0x7f2ec8072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:22.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2ec806f6c0 0x7f2ec8072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 1 monclient: found mon.noname-b 2024-09-11T02:54:22.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:22.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 1 -- 172.21.15.98:0/3986664353 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec80a64a0 con 0x7f2ec806faa0 2024-09-11T02:54:22.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.053+0000 7f2ecd40b640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806faa0 0x7f2ec806fe80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098a0 tx=0x7f2ebc01c180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a55959fc84f3173 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 1 -- 172.21.15.98:0/3986664353 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2ebc0b0070 con 0x7f2ec806faa0 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: dump: 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.069 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 1 -- 172.21.15.98:0/3986664353 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ebc0aa810 con 0x7f2ec806faa0 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 1 -- 172.21.15.98:0/3986664353 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2ebc0a9300 con 0x7f2ec806faa0 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 5 monclient: authenticate success, global_id 23357 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 4 set_mon_vals no callback set 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:22.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2eccc0a640 10 monclient: dump: 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.071 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 10 monclient: shutdown 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806faa0 msgr2=0x7f2ec806fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806faa0 0x7f2ec806fe80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0098a0 tx=0x7f2ebc01c180 comp rx=0 tx=0).stop 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3986664353 shutdown_connections 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec81397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806faa0 0x7f2ec806fe80 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3986664353 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2ec806f6c0 0x7f2ec8072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3986664353 >> 172.21.15.98:0/3986664353 conn(0x7f2ec80acac0 msgr2=0x7f2ec80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3986664353 shutdown_connections 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3986664353 wait complete. 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 10 monclient: monmap: 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.073 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 librados: starting msgr at 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 librados: starting objecter 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 Processor -- start 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 -- start start 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 librados: setting wanted keys 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 1 librados: calling monclient init 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.057+0000 7f2ecfe70640 10 monclient: init 2024-09-11T02:54:22.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _add_conns ranks=[2,4,6,8,5,1,0,3,7] 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806f6c0 0x7f2ec81d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec806f6c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ec806faa0 0x7f2ec8138dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec806faa0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec8139310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): picked mon.noname-d con 0x7f2ec80703c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec8143840 con 0x7f2ec806f6c0 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2ec81436c0 con 0x7f2ec806faa0 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2ec81439c0 con 0x7f2ec80703c0 2024-09-11T02:54:22.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient(hunting): authenticate will time out at 1215.944159s 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806f6c0 0x7f2ec81d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec8139310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806f6c0 0x7f2ec81d1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50758/0 (socket says 172.21.15.98:50758) 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 1 -- 172.21.15.98:0/3479696911 learned_addr learned my addr 172.21.15.98:0/3479696911 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient(hunting): get_auth_request con 0x7f2ec806f6c0 auth_method 0 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ec806faa0 0x7f2ec8138dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): get_auth_request con 0x7f2ec80703c0 auth_method 0 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 10 monclient(hunting): get_auth_request con 0x7f2ec806faa0 auth_method 0 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient(hunting): handle_auth_done global_id 14631 payload 824 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 -- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ec806faa0 msgr2=0x7f2ec8138dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ec806faa0 0x7f2ec8138dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 -- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806f6c0 msgr2=0x7f2ec81d1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806f6c0 0x7f2ec81d1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 monclient: found mon.noname-d 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 -- 172.21.15.98:0/3479696911 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec80aa6c0 con 0x7f2ec80703c0 2024-09-11T02:54:22.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecd40b640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ec806faa0 0x7f2ec8138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806f6c0 0x7f2ec81d1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ece40d640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec8139310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800e820 tx=0x7f2eb800ece0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 client.?.objecter ms_handle_connect 0x7f2ec80703c0 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 1 -- 172.21.15.98:0/3479696911 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2eb8017070 con 0x7f2ec80703c0 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: dump: 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.079 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 1 -- 172.21.15.98:0/3479696911 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb800cba0 con 0x7f2ec80703c0 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 5 monclient: authenticate success, global_id 14631 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _renew_subs 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec81242b0 con 0x7f2ec80703c0 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 1 -- 172.21.15.98:0/3479696911 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2eb800cd50 con 0x7f2ec80703c0 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec7fff640 4 set_mon_vals no callback set 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 client.14631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 monclient: dump: 2024-09-11T02:54:22.081 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _renew_subs 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec8123070 con 0x7f2ec80703c0 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 librados: init done 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 1 -- 172.21.15.98:0/3479696911 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2eb800c6d0 con 0x7f2ec80703c0 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec806e820 con 0x7f2ec80703c0 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2eb403df40 0x7f2eb4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 1 -- 172.21.15.98:0/3479696911 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(188..188 src has 1..188) ==== 6643+0+0 (secure 0 0 0) 0x7f2eb800f040 con 0x7f2ec80703c0 2024-09-11T02:54:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 10 client.14631.objecter ms_dispatch 0x7f2ec8000ba0 osd_map(188..188 src has 1..188) 2024-09-11T02:54:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 3 client.14631.objecter handle_osd_map got epochs [188,188] > 0 2024-09-11T02:54:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 3 client.14631.objecter handle_osd_map decoding full epoch 188 2024-09-11T02:54:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ec67fc640 20 client.14631.objecter dump_active .. 0 homeless 2024-09-11T02:54:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2eb403df40 0x7f2eb4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.061+0000 7f2ecdc0c640 10 monclient: get_auth_request con 0x7f2eb403df40 auth_method 0 2024-09-11T02:54:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.065+0000 7f2ec67fc640 10 client.14631.objecter ms_handle_connect 0x7f2eb403df40 2024-09-11T02:54:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.065+0000 7f2ecdc0c640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2eb403df40 0x7f2eb4040400 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000c530 tx=0x7f2ec000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.065+0000 7f2ec67fc640 1 -- 172.21.15.98:0/3479696911 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f2ec806e820 con 0x7f2ec80703c0 2024-09-11T02:54:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.065+0000 7f2ec67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.065+0000 7f2ec67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.313+0000 7f2ecfe70640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:54:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.313+0000 7f2ecfe70640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.313+0000 7f2ecfe70640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.313+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ec80a50c0 con 0x7f2ec80703c0 2024-09-11T02:54:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.313+0000 7f2ec67fc640 1 -- 172.21.15.98:0/3479696911 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f2eb8021070 con 0x7f2ec80703c0 2024-09-11T02:54:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.313+0000 7f2ec67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.313+0000 7f2ec67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:22.318 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":42,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 10 librados: watch_flush enter 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 10 librados: watch_flush exit 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 20 client.14631.objecter shutdown clearing up homeless session... 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 10 client.14631.objecter successfully canceled tick 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2eb403df40 msgr2=0x7f2eb4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2eb403df40 0x7f2eb4040400 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f2ec000c530 tx=0x7f2ec000b040 comp rx=0 tx=0).stop 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 10 monclient: shutdown 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 msgr2=0x7f2ec8139310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec8139310 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2eb800e820 tx=0x7f2eb800ece0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 shutdown_connections 2024-09-11T02:54:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2ec80703c0 0x7f2ec8139310 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ec806faa0 0x7f2ec8138dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2eb403df40 0x7f2eb4040400 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 --2- 172.21.15.98:0/3479696911 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ec806f6c0 0x7f2ec81d1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 >> 172.21.15.98:0/3479696911 conn(0x7f2ec80acac0 msgr2=0x7f2ec80ab870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 shutdown_connections 2024-09-11T02:54:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 -- 172.21.15.98:0/3479696911 wait complete. 2024-09-11T02:54:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.317+0000 7f2ecfe70640 1 librados: shutdown 2024-09-11T02:54:22.337 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:54:22.337 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:54:22.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:54:22.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient: get_monmap_and_config 2024-09-11T02:54:22.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient: monmap: 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:22.458098+0000 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:22.458098+0000 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.457 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.458 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.458 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.458 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:22.458 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:22.458 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 1 Processor -- start 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 1 -- start start 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient: init 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient: _add_conns ranks=[10,5,1,6,0,2,3,7,4,8,9] 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1ba80a39f0 0x7f1ba80a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient(hunting): picked mon.noname-f con 0x7f1ba80a39f0 addr v2:172.21.15.190:3300/0 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1ba80a3dd0 0x7f1ba80a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient(hunting): picked mon.noname-h con 0x7f1ba80a3dd0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1ba81377c0 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ba809ea20 con 0x7f1ba81377c0 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f1ba809d560 con 0x7f1ba80a39f0 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1ba809db20 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bafce7640 10 monclient(hunting): authenticate will time out at 1216.338443s 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1ba80a3dd0 0x7f1ba80a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f1ba80a39f0 0x7f1ba80a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1ba80a3dd0 0x7f1ba80a41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:58490/0 (socket says 172.21.15.98:58490) 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 -- 172.21.15.98:0/3957748405 learned_addr learned my addr 172.21.15.98:0/3957748405 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): get_auth_request con 0x7f1ba80a3dd0 auth_method 0 2024-09-11T02:54:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 10 monclient(hunting): get_auth_request con 0x7f1ba80a39f0 auth_method 0 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient(hunting): handle_auth_done global_id 19571 payload 274 2024-09-11T02:54:22.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 -- 172.21.15.98:0/3957748405 >> v2:172.21.15.190:3300/0 conn(0x7f1ba80a39f0 msgr2=0x7f1ba80a1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 --2- 172.21.15.98:0/3957748405 >> v2:172.21.15.190:3300/0 conn(0x7f1ba80a39f0 0x7f1ba80a1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 -- 172.21.15.98:0/3957748405 >> v1:172.21.15.190:6789/0 conn(0x7f1ba81377c0 legacy=0x7f1ba8137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 monclient: found mon.noname-h 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bad282640 1 -- 172.21.15.98:0/3957748405 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba80a1be0 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.453+0000 7f1bada83640 1 --2- 172.21.15.98:0/3957748405 >> v2:172.21.15.190:3300/0 conn(0x7f1ba80a39f0 0x7f1ba80a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bad282640 1 --2- 172.21.15.98:0/3957748405 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1ba80a3dd0 0x7f1ba80a41b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009870 tx=0x7f1ba401c420 comp rx=0 tx=0).ready entity=mon.5 client_cookie=6bb75ea878734cf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 1 -- 172.21.15.98:0/3957748405 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ba40ab870 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: dump: 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 5 monclient: authenticate success, global_id 19571 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 1 -- 172.21.15.98:0/3957748405 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba40b0070 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 4 set_mon_vals no callback set 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 1 -- 172.21.15.98:0/3957748405 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ba40aa360 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1baca81640 10 monclient: dump: 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 10 monclient: shutdown 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 -- 172.21.15.98:0/3957748405 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1ba80a3dd0 msgr2=0x7f1ba80a41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 --2- 172.21.15.98:0/3957748405 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1ba80a3dd0 0x7f1ba80a41b0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009870 tx=0x7f1ba401c420 comp rx=0 tx=0).stop 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 -- 172.21.15.98:0/3957748405 shutdown_connections 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 --2- 172.21.15.98:0/3957748405 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1ba80a3dd0 0x7f1ba80a41b0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 --2- 172.21.15.98:0/3957748405 >> v2:172.21.15.190:3300/0 conn(0x7f1ba80a39f0 0x7f1ba80a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 -- 172.21.15.98:0/3957748405 >> 172.21.15.98:0/3957748405 conn(0x7f1ba80a8670 msgr2=0x7f1ba80a4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 -- 172.21.15.98:0/3957748405 shutdown_connections 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 1 -- 172.21.15.98:0/3957748405 wait complete. 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.457+0000 7f1bafce7640 10 monclient: monmap: 2024-09-11T02:54:22.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 librados: starting msgr at 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 librados: starting objecter 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 Processor -- start 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 -- start start 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 librados: setting wanted keys 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 librados: calling monclient init 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient: init 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient: _add_conns ranks=[0,2,7,8,1,5,3,6,4] 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba80a39f0 0x7f1ba81d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): picked mon.noname-a con 0x7f1ba80a39f0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:22.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ba80a3dd0 0x7f1ba81d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): picked mon.noname-b con 0x7f1ba80a3dd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1ba81377c0 0x7f1ba813d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): picked mon.noname-i con 0x7f1ba81377c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba81423c0 con 0x7f1ba80a39f0 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1ba8142240 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f1ba8142540 con 0x7f1ba81377c0 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): authenticate will time out at 1216.344711s 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ba80a3dd0 0x7f1ba81d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba80a39f0 0x7f1ba81d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba80a39f0 0x7f1ba81d16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42696/0 (socket says 172.21.15.98:42696) 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ba80a3dd0 0x7f1ba81d1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50764/0 (socket says 172.21.15.98:50764) 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 1 -- 172.21.15.98:0/2991066811 learned_addr learned my addr 172.21.15.98:0/2991066811 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): get_auth_request con 0x7f1ba80a39f0 auth_method 0 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): get_auth_request con 0x7f1ba80a3dd0 auth_method 0 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bae284640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1ba81377c0 0x7f1ba813d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bae284640 10 monclient(hunting): get_auth_request con 0x7f1ba81377c0 auth_method 0 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bae284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bae284640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bae284640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient(hunting): handle_auth_done global_id 23375 payload 824 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1ba81377c0 msgr2=0x7f1ba813d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1ba81377c0 0x7f1ba813d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba80a39f0 msgr2=0x7f1ba81d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba80a39f0 0x7f1ba81d16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 monclient: found mon.noname-b 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 -- 172.21.15.98:0/2991066811 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba8122ae0 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba80a39f0 0x7f1ba81d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bad282640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ba80a3dd0 0x7f1ba81d1be0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ac000 tx=0x7f1ba40191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 client.?.objecter ms_handle_connect 0x7f1ba80a3dd0 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ba40ab400 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: dump: 2024-09-11T02:54:22.480 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.481 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 5 monclient: authenticate success, global_id 23375 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba40ab5e0 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient: _renew_subs 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba8122dc0 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ba40abc00 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:22.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 client.23375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9ffff640 4 set_mon_vals no callback set 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: dump: 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient: _renew_subs 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba80a01e0 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 librados: init done 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813df20 0x7f1ba802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba806ca10 con 0x7f1ba813df20 2024-09-11T02:54:22.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813df20 0x7f1ba802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient: get_auth_request con 0x7f1ba813df20 auth_method 0 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 10 monclient(hunting): handle_auth_done global_id 34110 payload 824 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1ba40ab790 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 1 -- 172.21.15.98:0/2991066811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ba80a1e60 con 0x7f1ba813df20 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bada83640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813df20 0x7f1ba802d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004980 tx=0x7f1ba000e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1b9403df40 0x7f1b94040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 client.23375.objecter ms_handle_connect 0x7f1ba813df20 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 client.23375.objecter resend_mon_ops 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(189..189 src has 1..189) ==== 6360+0+0 (secure 0 0 0) 0x7f1ba40e9d50 con 0x7f1ba80a3dd0 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 client.23375.objecter ms_dispatch 0x7f1ba8000ba0 osd_map(189..189 src has 1..189) 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 3 client.23375.objecter handle_osd_map got epochs [189,189] > 0 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 3 client.23375.objecter handle_osd_map decoding full epoch 189 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 20 client.23375.objecter dump_active .. 0 homeless 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ba0019070 con 0x7f1ba813df20 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: dump: 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.486 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bae284640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1b9403df40 0x7f1b94040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ba0015720 con 0x7f1ba813df20 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813df20 msgr2=0x7f1ba802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813df20 0x7f1ba802d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004980 tx=0x7f1ba000e8c0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813df20 msgr2=0x7f1ba802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1b9e7fc640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813df20 0x7f1ba802d450 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004980 tx=0x7f1ba000e8c0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.461+0000 7f1bae284640 10 monclient: get_auth_request con 0x7f1b9403df40 auth_method 0 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.465+0000 7f1b9e7fc640 10 client.23375.objecter ms_handle_connect 0x7f1b9403df40 2024-09-11T02:54:22.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.465+0000 7f1bae284640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1b9403df40 0x7f1b94040400 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7f1b98000f70 tx=0x7f1b98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813e300 0x7f1ba802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba806e870 con 0x7f1ba813e300 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813e300 0x7f1ba802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient: get_auth_request con 0x7f1ba813e300 auth_method 0 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 10 monclient(hunting): handle_auth_done global_id 34119 payload 824 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 1 -- 172.21.15.98:0/2991066811 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ba8137e60 con 0x7f1ba813e300 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 10 client.23375.objecter ms_handle_connect 0x7f1ba813e300 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bada83640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813e300 0x7f1ba802d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004980 tx=0x7f1ba0004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 10 client.23375.objecter resend_mon_ops 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ba001ecf0 con 0x7f1ba813e300 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:22.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 10 monclient: dump: 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f1ba00143b0 con 0x7f1ba813e300 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813e300 msgr2=0x7f1ba802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813e300 0x7f1ba802d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0004980 tx=0x7f1ba0004ac0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813e300 msgr2=0x7f1ba802d990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:22.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1b9e7fc640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ba813e300 0x7f1ba802d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 15467, 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:22.494 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.495 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:22.497 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:22.498 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:22.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:22.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.504 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:22.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.509 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:22.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.514 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.515 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.516 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:22.516 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 10 librados: watch_flush enter 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 10 librados: watch_flush exit 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 20 client.23375.objecter shutdown clearing up homeless session... 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 10 client.23375.objecter successfully canceled tick 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1b9403df40 msgr2=0x7f1b94040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1b9403df40 0x7f1b94040400 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7f1b98000f70 tx=0x7f1b98009040 comp rx=0 tx=0).stop 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 10 monclient: shutdown 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ba80a3dd0 msgr2=0x7f1ba81d1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 --2- 172.21.15.98:0/2991066811 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ba80a3dd0 0x7f1ba81d1be0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ac000 tx=0x7f1ba40191c0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bae284640 1 -- 172.21.15.98:0/2991066811 reap_dead start 2024-09-11T02:54:22.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bae284640 1 -- 172.21.15.98:0/2991066811 reap_dead start 2024-09-11T02:54:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 shutdown_connections 2024-09-11T02:54:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 >> 172.21.15.98:0/2991066811 conn(0x7f1ba80a8670 msgr2=0x7f1ba813f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 shutdown_connections 2024-09-11T02:54:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 -- 172.21.15.98:0/2991066811 wait complete. 2024-09-11T02:54:22.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.469+0000 7f1bafce7640 1 librados: shutdown 2024-09-11T02:54:22.519 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:22.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:54:22.519 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:54:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient: get_monmap_and_config 2024-09-11T02:54:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient: monmap: 2024-09-11T02:54:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:22.641794+0000 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:22.641794+0000 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.642 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:22.642 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:22.642 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 1 Processor -- start 2024-09-11T02:54:22.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 1 -- start start 2024-09-11T02:54:22.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient: init 2024-09-11T02:54:22.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient: _add_conns ranks=[6,5,0,2,7,4,3,9,10,8,1] 2024-09-11T02:54:22.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01e0139b80 0x7f01e0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient(hunting): picked mon.noname-d con 0x7f01e0139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f01e006fe50 0x7f01e00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient(hunting): picked mon.noname-h con 0x7f01e006fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01e0072640 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e0071370 con 0x7f01e0072640 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f01e00719c0 con 0x7f01e006fe50 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f01e0071c10 con 0x7f01e0139b80 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01e5f0e640 10 monclient(hunting): authenticate will time out at 1216.522956s 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01dffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f01e0072640 0x7f01e00a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56628/0 (socket says 172.21.15.98:56628) 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01e0139b80 0x7f01e0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01dffff640 1 -- 172.21.15.98:0/14265974 learned_addr learned my addr 172.21.15.98:0/14265974 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): get_auth_request con 0x7f01e0139b80 auth_method 0 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01d6ffd640 1 --2- 172.21.15.98:0/14265974 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f01e006fe50 0x7f01e00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01d6ffd640 10 monclient(hunting): get_auth_request con 0x7f01e006fe50 auth_method 0 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01deffd640 1 -- 172.21.15.98:0/14265974 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952572302 0 0) 0x7f01e0071370 con 0x7f01e0072640 2024-09-11T02:54:22.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01deffd640 10 monclient(hunting): _negotiate 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01deffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01deffd640 10 monclient(hunting): my global_id is 34128 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01deffd640 1 -- 172.21.15.98:0/14265974 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01cc003530 con 0x7f01e0072640 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient(hunting): handle_auth_done global_id 14640 payload 274 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 1 -- 172.21.15.98:0/14265974 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f01e006fe50 msgr2=0x7f01e00a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 1 --2- 172.21.15.98:0/14265974 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f01e006fe50 0x7f01e00a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 1 -- 172.21.15.98:0/14265974 >> v1:172.21.15.98:6789/0 conn(0x7f01e0072640 legacy=0x7f01e00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 1 monclient: found mon.noname-d 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:22.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.637+0000 7f01df7fe640 1 -- 172.21.15.98:0/14265974 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e00a9600 con 0x7f01e0139b80 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01df7fe640 1 --2- 172.21.15.98:0/14265974 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01e0139b80 0x7f01e0139f60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f01d00097f0 tx=0x7f01d001c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ad3b4072651df77b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 1 -- 172.21.15.98:0/14265974 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01d00b0070 con 0x7f01e0139b80 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: dump: 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.654 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 1 -- 172.21.15.98:0/14265974 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d00aa810 con 0x7f01e0139b80 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 5 monclient: authenticate success, global_id 14640 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 1 -- 172.21.15.98:0/14265974 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01d00aae30 con 0x7f01e0139b80 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 4 set_mon_vals no callback set 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01deffd640 10 monclient: dump: 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.656 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.657 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient: shutdown 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- 172.21.15.98:0/14265974 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01e0139b80 msgr2=0x7f01e0139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/14265974 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01e0139b80 0x7f01e0139f60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f01d00097f0 tx=0x7f01d001c3a0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- 172.21.15.98:0/14265974 shutdown_connections 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/14265974 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f01e006fe50 0x7f01e00a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/14265974 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f01e0139b80 0x7f01e0139f60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- 172.21.15.98:0/14265974 >> 172.21.15.98:0/14265974 conn(0x7f01e00ac730 msgr2=0x7f01e00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- 172.21.15.98:0/14265974 shutdown_connections 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- 172.21.15.98:0/14265974 wait complete. 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient: monmap: 2024-09-11T02:54:22.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.659 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 librados: starting msgr at 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 librados: starting objecter 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 Processor -- start 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- start start 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 librados: setting wanted keys 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 librados: calling monclient init 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient: init 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient: _add_conns ranks=[0,7,8,6,5,3,4,1,2] 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e006fe50 0x7f01e013ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f01e006fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:22.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f01e0072640 0x7f01e013bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): picked mon.noname-i con 0x7f01e0072640 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f01e0139b80 0x7f01e013c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): picked mon.noname-e con 0x7f01e0139b80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01e0144450 con 0x7f01e006fe50 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f01e00a8870 con 0x7f01e0072640 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f01e01445d0 con 0x7f01e0139b80 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.641+0000 7f01e5f0e640 10 monclient(hunting): authenticate will time out at 1216.527994s 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e006fe50 0x7f01e013ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e006fe50 0x7f01e013ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42730/0 (socket says 172.21.15.98:42730) 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 -- 172.21.15.98:0/2416113777 learned_addr learned my addr 172.21.15.98:0/2416113777 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): get_auth_request con 0x7f01e006fe50 auth_method 0 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f01e0139b80 0x7f01e013c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient(hunting): get_auth_request con 0x7f01e0139b80 auth_method 0 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient(hunting): handle_auth_done global_id 34137 payload 824 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f01e0139b80 msgr2=0x7f01e013c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f01e0139b80 0x7f01e013c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f01e0072640 msgr2=0x7f01e013bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f01e0072640 0x7f01e013bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f01e0072640 0x7f01e013bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 monclient: found mon.noname-a 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 -- 172.21.15.98:0/2416113777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e013f240 con 0x7f01e006fe50 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01df7fe640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e006fe50 0x7f01e013ba40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01d00ae040 tx=0x7f01d0019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 client.?.objecter ms_handle_connect 0x7f01e006fe50 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01d00b0050 con 0x7f01e006fe50 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: dump: 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.664 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d00b1040 con 0x7f01e006fe50 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 5 monclient: authenticate success, global_id 34137 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 10 monclient: _renew_subs 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e006d6f0 con 0x7f01e006fe50 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01d00044b0 con 0x7f01e006fe50 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 10 client.34137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01de7fc640 4 set_mon_vals no callback set 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: dump: 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.666 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.667 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 10 monclient: _renew_subs 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e013ef60 con 0x7f01e006fe50 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 1 librados: init done 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d15c0 0x7f01e01d19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f01e013dd20 con 0x7f01e01d15c0 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d15c0 0x7f01e01d19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient: get_auth_request con 0x7f01e01d15c0 auth_method 0 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f01d0004040 con 0x7f01e006fe50 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f01c003df40 0x7f01c0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6360+0+0 (secure 0 0 0) 0x7f01d00bc020 con 0x7f01e006fe50 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 client.34137.objecter ms_dispatch 0x7f01e0000ba0 osd_map(189..189 src has 1..189) 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 3 client.34137.objecter handle_osd_map got epochs [189,189] > 0 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 3 client.34137.objecter handle_osd_map decoding full epoch 189 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 20 client.34137.objecter dump_active .. 0 homeless 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_done global_id 23384 payload 824 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 1 -- 172.21.15.98:0/2416113777 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01e00a7fb0 con 0x7f01e01d15c0 2024-09-11T02:54:22.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f01c003df40 0x7f01c0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 10 monclient: get_auth_request con 0x7f01c003df40 auth_method 0 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01d6ffd640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d15c0 0x7f01e01d19a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f01c800e500 tx=0x7f01c800e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 client.34137.objecter ms_handle_connect 0x7f01e01d15c0 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 client.34137.objecter resend_mon_ops 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c8017070 con 0x7f01e01d15c0 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: dump: 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.670 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f01c801c450 con 0x7f01e01d15c0 2024-09-11T02:54:22.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d15c0 msgr2=0x7f01e01d19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d15c0 0x7f01e01d19a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f01c800e500 tx=0x7f01c800e9c0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d15c0 msgr2=0x7f01e01d19a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d15c0 0x7f01e01d19a0 secure :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f01c800e500 tx=0x7f01c800e9c0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dcff9640 10 client.34137.objecter ms_handle_connect 0x7f01c003df40 2024-09-11T02:54:22.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.645+0000 7f01dffff640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f01c003df40 0x7f01c0040400 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f01d8005c10 tx=0x7f01d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d1ee0 0x7f01e00a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f01e006ca10 con 0x7f01e01d1ee0 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d1ee0 0x7f01e00a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient: get_auth_request con 0x7f01e01d1ee0 auth_method 0 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 10 monclient(hunting): handle_auth_done global_id 23393 payload 824 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 1 -- 172.21.15.98:0/2416113777 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f01e0139f60 con 0x7f01e01d1ee0 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01d6ffd640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d1ee0 0x7f01e00a1b80 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f01c8000c10 tx=0x7f01c8004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 10 client.34137.objecter ms_handle_connect 0x7f01e01d1ee0 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 10 client.34137.objecter resend_mon_ops 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c800f040 con 0x7f01e01d1ee0 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 10 monclient: dump: 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.675 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f01c8013b80 con 0x7f01e01d1ee0 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d1ee0 msgr2=0x7f01e00a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d1ee0 0x7f01e00a1b80 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f01c8000c10 tx=0x7f01c8004740 comp rx=0 tx=0).stop 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d1ee0 msgr2=0x7f01e00a1b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dcff9640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f01e01d1ee0 0x7f01e00a1b80 secure :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f01c8000c10 tx=0x7f01c8004740 comp rx=0 tx=0).stop 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:22.677 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 278438, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.678 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.679 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:22.681 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.682 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:22.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:22.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.686 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.688 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.689 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:22.691 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:22.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:22.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.698 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.699 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.700 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:22.701 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:22.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 10 librados: watch_flush enter 2024-09-11T02:54:22.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 10 librados: watch_flush exit 2024-09-11T02:54:22.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 20 client.34137.objecter shutdown clearing up homeless session... 2024-09-11T02:54:22.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 10 client.34137.objecter successfully canceled tick 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f01c003df40 msgr2=0x7f01c0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f01c003df40 0x7f01c0040400 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f01d8005c10 tx=0x7f01d8009040 comp rx=0 tx=0).stop 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 10 monclient: shutdown 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e006fe50 msgr2=0x7f01e013ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e006fe50 0x7f01e013ba40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01d00ae040 tx=0x7f01d0019140 comp rx=0 tx=0).stop 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01dffff640 1 -- 172.21.15.98:0/2416113777 reap_dead start 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 shutdown_connections 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 --2- 172.21.15.98:0/2416113777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01e006fe50 0x7f01e013ba40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 >> 172.21.15.98:0/2416113777 conn(0x7f01e00ac730 msgr2=0x7f01e00a0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 shutdown_connections 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 -- 172.21.15.98:0/2416113777 wait complete. 2024-09-11T02:54:22.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.653+0000 7f01e5f0e640 1 librados: shutdown 2024-09-11T02:54:22.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:22.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:54:22.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:54:22.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient: get_monmap_and_config 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient: monmap: 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:22.822148+0000 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:22.822148+0000 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.821 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:22.822 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 1 Processor -- start 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 1 -- start start 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient: init 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient: _add_conns ranks=[3,4,0,7,2,1,10,9,8,6,5] 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 0x7f1e88072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient(hunting): picked mon.noname-g con 0x7f1e88139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88139ef0 0x7f1e880a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient(hunting): picked mon.noname-c con 0x7f1e88139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e8813a2d0 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:22.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e880702b0 con 0x7f1e8813a2d0 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1e88071bf0 con 0x7f1e88139b10 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1e88071350 con 0x7f1e88139ef0 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ff58640 10 monclient(hunting): authenticate will time out at 1216.702377s 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8e4f5640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1e8813a2d0 0x7f1e880a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56630/0 (socket says 172.21.15.98:56630) 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88139ef0 0x7f1e880a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8e4f5640 1 -- 172.21.15.98:0/2776686712 learned_addr learned my addr 172.21.15.98:0/2776686712 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): get_auth_request con 0x7f1e88139ef0 auth_method 0 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8dcf4640 1 --2- 172.21.15.98:0/2776686712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 0x7f1e88072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8dcf4640 10 monclient(hunting): get_auth_request con 0x7f1e88139b10 auth_method 0 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8dcf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient(hunting): handle_auth_done global_id 23368 payload 274 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 1 -- 172.21.15.98:0/2776686712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 msgr2=0x7f1e88072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 1 --2- 172.21.15.98:0/2776686712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 0x7f1e88072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ccf2640 1 -- 172.21.15.98:0/2776686712 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343437485 0 0) 0x7f1e800014a0 con 0x7f1e8813a2d0 2024-09-11T02:54:22.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 1 -- 172.21.15.98:0/2776686712 >> v1:172.21.15.98:6789/0 conn(0x7f1e8813a2d0 legacy=0x7f1e880a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 1 monclient: found mon.noname-c 2024-09-11T02:54:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8d4f3640 1 -- 172.21.15.98:0/2776686712 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e880a9620 con 0x7f1e88139ef0 2024-09-11T02:54:22.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.817+0000 7f1e8ccf2640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:54:22.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8d4f3640 1 --2- 172.21.15.98:0/2776686712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88139ef0 0x7f1e880a7a10 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009820 tx=0x7f1e7801c380 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1810af5f7e90ea8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 1 -- 172.21.15.98:0/2776686712 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e780b0070 con 0x7f1e88139ef0 2024-09-11T02:54:22.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:22.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: dump: 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.834 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 5 monclient: authenticate success, global_id 23368 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 1 -- 172.21.15.98:0/2776686712 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e780aa810 con 0x7f1e88139ef0 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 4 set_mon_vals no callback set 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 1 -- 172.21.15.98:0/2776686712 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e780a9300 con 0x7f1e88139ef0 2024-09-11T02:54:22.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ccf2640 10 monclient: dump: 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient: shutdown 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- 172.21.15.98:0/2776686712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88139ef0 msgr2=0x7f1e880a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 --2- 172.21.15.98:0/2776686712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88139ef0 0x7f1e880a7a10 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f1e78009820 tx=0x7f1e7801c380 comp rx=0 tx=0).stop 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- 172.21.15.98:0/2776686712 shutdown_connections 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 --2- 172.21.15.98:0/2776686712 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88139ef0 0x7f1e880a7a10 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 --2- 172.21.15.98:0/2776686712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 0x7f1e88072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- 172.21.15.98:0/2776686712 >> 172.21.15.98:0/2776686712 conn(0x7f1e880ac790 msgr2=0x7f1e880a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- 172.21.15.98:0/2776686712 shutdown_connections 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- 172.21.15.98:0/2776686712 wait complete. 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient: build_initial_monmap 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient: monmap: 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:22.838 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 librados: starting msgr at 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 librados: starting objecter 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 Processor -- start 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- start start 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 librados: setting wanted keys 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 librados: calling monclient init 2024-09-11T02:54:22.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient: init 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient: _add_conns ranks=[3,5,0,7,2,8,6,4,1] 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 0x7f1e8813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): picked mon.noname-g con 0x7f1e88139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e88139ef0 0x7f1e8813ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): picked mon.noname-h con 0x7f1e88139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e8813a2d0 0x7f1e8813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): picked mon.noname-a con 0x7f1e8813a2d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e88144470 con 0x7f1e8813a2d0 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1e880a8870 con 0x7f1e88139b10 2024-09-11T02:54:22.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1e881445f0 con 0x7f1e88139ef0 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.821+0000 7f1e8ff58640 10 monclient(hunting): authenticate will time out at 1216.707868s 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e8813a2d0 0x7f1e8813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e8813a2d0 0x7f1e8813b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42734/0 (socket says 172.21.15.98:42734) 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 -- 172.21.15.98:0/1311221009 learned_addr learned my addr 172.21.15.98:0/1311221009 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): get_auth_request con 0x7f1e8813a2d0 auth_method 0 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 0x7f1e8813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8d4f3640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e88139ef0 0x7f1e8813ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): get_auth_request con 0x7f1e88139b10 auth_method 0 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient(hunting): handle_auth_done global_id 34155 payload 824 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e88139ef0 msgr2=0x7f1e8813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e88139ef0 0x7f1e8813ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 msgr2=0x7f1e8813a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e88139b10 0x7f1e8813a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient: _finish_hunting 0 2024-09-11T02:54:22.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 monclient: found mon.noname-a 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 -- 172.21.15.98:0/1311221009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e880a5d50 con 0x7f1e8813a2d0 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8e4f5640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e8813a2d0 0x7f1e8813b250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c6b0 tx=0x7f1e8000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 client.?.objecter ms_handle_connect 0x7f1e8813a2d0 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e80018070 con 0x7f1e8813a2d0 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: dump: 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.843 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.844 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e8000a640 con 0x7f1e8813a2d0 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 5 monclient: authenticate success, global_id 34155 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 10 monclient: _renew_subs 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e8813d100 con 0x7f1e8813a2d0 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e8000ace0 con 0x7f1e8813a2d0 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 10 client.34155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:22.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: dump: 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.846 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7f7fe640 4 set_mon_vals no callback set 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 10 monclient: _renew_subs 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e880a5670 con 0x7f1e8813a2d0 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 1 librados: init done 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e881337f0 0x7f1e88133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1e8806ca10 con 0x7f1e881337f0 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e881337f0 0x7f1e88133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient: get_auth_request con 0x7f1e881337f0 auth_method 0 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1e8000a870 con 0x7f1e8813a2d0 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_done global_id 23377 payload 824 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 1 -- 172.21.15.98:0/1311221009 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e880aa090 con 0x7f1e881337f0 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e6c03df40 0x7f1e6c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 6360+0+0 (secure 0 0 0) 0x7f1e8004ebe0 con 0x7f1e8813a2d0 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 client.34155.objecter ms_dispatch 0x7f1e88000ba0 osd_map(189..189 src has 1..189) 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 3 client.34155.objecter handle_osd_map got epochs [189,189] > 0 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 3 client.34155.objecter handle_osd_map decoding full epoch 189 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 20 client.34155.objecter dump_active .. 0 homeless 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 client.34155.objecter ms_handle_connect 0x7f1e881337f0 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 client.34155.objecter resend_mon_ops 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8dcf4640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e881337f0 0x7f1e88133bd0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400f4b0 tx=0x7f1e8400f970 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8d4f3640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e6c03df40 0x7f1e6c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e84019020 con 0x7f1e881337f0 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: dump: 2024-09-11T02:54:22.849 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.850 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8d4f3640 10 monclient: get_auth_request con 0x7f1e6c03df40 auth_method 0 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e84014670 con 0x7f1e881337f0 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e881337f0 msgr2=0x7f1e88133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e881337f0 0x7f1e88133bd0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400f4b0 tx=0x7f1e8400f970 comp rx=0 tx=0).stop 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e881337f0 msgr2=0x7f1e88133bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e881337f0 0x7f1e88133bd0 secure :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400f4b0 tx=0x7f1e8400f970 comp rx=0 tx=0).stop 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e7e7fc640 10 client.34155.objecter ms_handle_connect 0x7f1e6c03df40 2024-09-11T02:54:22.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.825+0000 7f1e8d4f3640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e6c03df40 0x7f1e6c040400 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f1e780ad000 tx=0x7f1e780a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88134110 0x7f1e880a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1e8806e870 con 0x7f1e88134110 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88134110 0x7f1e880a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient: get_auth_request con 0x7f1e88134110 auth_method 0 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:22.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 10 monclient(hunting): handle_auth_done global_id 23386 payload 824 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 1 -- 172.21.15.98:0/1311221009 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e8806e680 con 0x7f1e88134110 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 10 client.34155.objecter ms_handle_connect 0x7f1e88134110 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 10 client.34155.objecter resend_mon_ops 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8dcf4640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88134110 0x7f1e880a0e60 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400f4b0 tx=0x7f1e840046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e84019040 con 0x7f1e88134110 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 10 monclient: dump: 2024-09-11T02:54:22.854 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:22.855 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f1e84004050 con 0x7f1e88134110 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88134110 msgr2=0x7f1e880a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88134110 0x7f1e880a0e60 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400f4b0 tx=0x7f1e840046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88134110 msgr2=0x7f1e880a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e7e7fc640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1e88134110 0x7f1e880a0e60 secure :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400f4b0 tx=0x7f1e840046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:22.856 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 278618, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:22.857 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.858 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:22.860 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:22.861 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.862 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:22.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:22.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.867 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.869 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.872 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.874 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:22.877 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:22.878 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:22.879 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:22.880 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 10 librados: watch_flush enter 2024-09-11T02:54:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 10 librados: watch_flush exit 2024-09-11T02:54:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 20 client.34155.objecter shutdown clearing up homeless session... 2024-09-11T02:54:22.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 10 client.34155.objecter successfully canceled tick 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e6c03df40 msgr2=0x7f1e6c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e6c03df40 0x7f1e6c040400 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f1e780ad000 tx=0x7f1e780a6000 comp rx=0 tx=0).stop 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 10 monclient: shutdown 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e8813a2d0 msgr2=0x7f1e8813b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 --2- 172.21.15.98:0/1311221009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e8813a2d0 0x7f1e8813b250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000c6b0 tx=0x7f1e8000cb70 comp rx=0 tx=0).stop 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8e4f5640 1 -- 172.21.15.98:0/1311221009 reap_dead start 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8e4f5640 1 -- 172.21.15.98:0/1311221009 reap_dead start 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 shutdown_connections 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 >> 172.21.15.98:0/1311221009 conn(0x7f1e880ac790 msgr2=0x7f1e880a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 shutdown_connections 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 -- 172.21.15.98:0/1311221009 wait complete. 2024-09-11T02:54:22.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.833+0000 7f1e8ff58640 1 librados: shutdown 2024-09-11T02:54:22.883 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:22.883 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:54:22.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient: get_monmap_and_config 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient: monmap: 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:23.001493+0000 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:23.001493+0000 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.000 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:23.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 1 Processor -- start 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 1 -- start start 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient: init 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient: _add_conns ranks=[3,4,0,10,7,8,2,5,9,6,1] 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 0x7feb50139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient(hunting): picked mon.noname-g con 0x7feb50139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb5006fe50 0x7feb500a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient(hunting): picked mon.noname-c con 0x7feb5006fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb50072640 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb50071370 con 0x7feb50072640 2024-09-11T02:54:23.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feb500719c0 con 0x7feb50139b80 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7feb50071c10 con 0x7feb5006fe50 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb57d40640 10 monclient(hunting): authenticate will time out at 1216.881807s 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb5006fe50 0x7feb500a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb562dd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feb50072640 0x7feb500a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56634/0 (socket says 172.21.15.98:56634) 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb562dd640 1 -- 172.21.15.98:0/3992235060 learned_addr learned my addr 172.21.15.98:0/3992235060 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): get_auth_request con 0x7feb5006fe50 auth_method 0 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 1 --2- 172.21.15.98:0/3992235060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 0x7feb50139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 10 monclient(hunting): get_auth_request con 0x7feb50139b80 auth_method 0 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb54ada640 1 -- 172.21.15.98:0/3992235060 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569496014 0 0) 0x7feb50071370 con 0x7feb50072640 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb54ada640 10 monclient(hunting): _negotiate 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb54ada640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb54ada640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb54ada640 10 monclient(hunting): my global_id is 34164 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb54ada640 1 -- 172.21.15.98:0/3992235060 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb3c003530 con 0x7feb50072640 2024-09-11T02:54:23.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient(hunting): handle_auth_done global_id 23395 payload 274 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 1 -- 172.21.15.98:0/3992235060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 msgr2=0x7feb50139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 1 --2- 172.21.15.98:0/3992235060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 0x7feb50139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 1 -- 172.21.15.98:0/3992235060 >> v1:172.21.15.98:6789/0 conn(0x7feb50072640 legacy=0x7feb500a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 1 monclient: found mon.noname-c 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:22.997+0000 7feb552db640 1 -- 172.21.15.98:0/3992235060 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb500a9600 con 0x7feb5006fe50 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb552db640 1 --2- 172.21.15.98:0/3992235060 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb5006fe50 0x7feb500a7a10 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009dc0 tx=0x7feb4c01c850 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e05c3db08e3edecb server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 1 -- 172.21.15.98:0/3992235060 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb4c0b2070 con 0x7feb5006fe50 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: dump: 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.016 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 1 -- 172.21.15.98:0/3992235060 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4c0a5a50 con 0x7feb5006fe50 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 5 monclient: authenticate success, global_id 23395 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 4 set_mon_vals no callback set 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 1 -- 172.21.15.98:0/3992235060 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb4c0a5c00 con 0x7feb5006fe50 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb54ada640 10 monclient: dump: 2024-09-11T02:54:23.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.019 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient: shutdown 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- 172.21.15.98:0/3992235060 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb5006fe50 msgr2=0x7feb500a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 --2- 172.21.15.98:0/3992235060 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb5006fe50 0x7feb500a7a10 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7feb4c009dc0 tx=0x7feb4c01c850 comp rx=0 tx=0).stop 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- 172.21.15.98:0/3992235060 shutdown_connections 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 --2- 172.21.15.98:0/3992235060 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb5006fe50 0x7feb500a7a10 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 --2- 172.21.15.98:0/3992235060 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 0x7feb50139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- 172.21.15.98:0/3992235060 >> 172.21.15.98:0/3992235060 conn(0x7feb500ac730 msgr2=0x7feb500a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- 172.21.15.98:0/3992235060 shutdown_connections 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- 172.21.15.98:0/3992235060 wait complete. 2024-09-11T02:54:23.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient: monmap: 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.021 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 librados: starting msgr at 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 librados: starting objecter 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 Processor -- start 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- start start 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 librados: setting wanted keys 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 librados: calling monclient init 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient: init 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient: _add_conns ranks=[2,6,3,1,7,8,0,4,5] 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb5006fe50 0x7feb501d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): picked mon.noname-b con 0x7feb5006fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb50072640 0x7feb501d1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): picked mon.noname-d con 0x7feb50072640 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 0x7feb500a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): picked mon.noname-g con 0x7feb50139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7feb50144450 con 0x7feb5006fe50 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feb500a8870 con 0x7feb50139b80 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7feb501445d0 con 0x7feb50072640 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.001+0000 7feb57d40640 10 monclient(hunting): authenticate will time out at 1216.887938s 2024-09-11T02:54:23.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb5006fe50 0x7feb501d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb50072640 0x7feb501d1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb5006fe50 0x7feb501d17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50788/0 (socket says 172.21.15.98:50788) 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb50072640 0x7feb501d1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:51124/0 (socket says 172.21.15.98:51124) 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 1 -- 172.21.15.98:0/3173871884 learned_addr learned my addr 172.21.15.98:0/3173871884 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): get_auth_request con 0x7feb5006fe50 auth_method 0 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): get_auth_request con 0x7feb50072640 auth_method 0 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 0x7feb500a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 10 monclient(hunting): get_auth_request con 0x7feb50139b80 auth_method 0 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient(hunting): handle_auth_done global_id 14649 payload 824 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 msgr2=0x7feb500a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb50139b80 0x7feb500a48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb5006fe50 msgr2=0x7feb501d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb5006fe50 0x7feb501d17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 monclient: found mon.noname-d 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 -- 172.21.15.98:0/3173871884 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb5013cea0 con 0x7feb50072640 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7feb5006fe50 0x7feb501d17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb552db640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb50072640 0x7feb501d1ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb500704e0 tx=0x7feb4c0195f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 client.?.objecter ms_handle_connect 0x7feb50072640 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb4c0bd020 con 0x7feb50072640 2024-09-11T02:54:23.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: dump: 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.027 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb4c0b3040 con 0x7feb50072640 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb4c0b2050 con 0x7feb50072640 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 5 monclient: authenticate success, global_id 14649 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 10 monclient: _renew_subs 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb500a8020 con 0x7feb50072640 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb47fff640 4 set_mon_vals no callback set 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: dump: 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 10 monclient: _renew_subs 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7feb5013b970 con 0x7feb50072640 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 1 librados: init done 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a4e10 0x7feb500a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7feb500aba20 con 0x7feb500a4e10 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a4e10 0x7feb500a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient: get_auth_request con 0x7feb500a4e10 auth_method 0 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7feb4c0039a0 con 0x7feb50072640 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 10 monclient(hunting): handle_auth_done global_id 14658 payload 824 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 1 -- 172.21.15.98:0/3173871884 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb5013ab90 con 0x7feb500a4e10 2024-09-11T02:54:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb3403df40 0x7feb34040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(189..189 src has 1..189) ==== 6360+0+0 (secure 0 0 0) 0x7feb4c0e9b70 con 0x7feb50072640 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 client.14649.objecter ms_dispatch 0x7feb50000ba0 osd_map(189..189 src has 1..189) 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 3 client.14649.objecter handle_osd_map got epochs [189,189] > 0 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 3 client.14649.objecter handle_osd_map decoding full epoch 189 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 20 client.14649.objecter dump_active .. 0 homeless 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 client.14649.objecter ms_handle_connect 0x7feb500a4e10 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 client.14649.objecter resend_mon_ops 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb55adc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a4e10 0x7feb500a51f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb48011db0 tx=0x7feb4800f610 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb480148b0 con 0x7feb500a4e10 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: dump: 2024-09-11T02:54:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb3403df40 0x7feb34040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 10 monclient: get_auth_request con 0x7feb3403df40 auth_method 0 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb48014440 con 0x7feb500a4e10 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a4e10 msgr2=0x7feb500a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a4e10 0x7feb500a51f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb48011db0 tx=0x7feb4800f610 comp rx=0 tx=0).stop 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a4e10 msgr2=0x7feb500a51f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a4e10 0x7feb500a51f0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb48011db0 tx=0x7feb4800f610 comp rx=0 tx=0).stop 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb467fc640 10 client.14649.objecter ms_handle_connect 0x7feb3403df40 2024-09-11T02:54:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.005+0000 7feb562dd640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb3403df40 0x7feb34040400 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7feb40007fd0 tx=0x7feb40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a5730 0x7feb500a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7feb5006ca10 con 0x7feb500a5730 2024-09-11T02:54:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a5730 0x7feb500a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient: get_auth_request con 0x7feb500a5730 auth_method 0 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 10 monclient(hunting): handle_auth_done global_id 14667 payload 824 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 1 -- 172.21.15.98:0/3173871884 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb50139f60 con 0x7feb500a5730 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb55adc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a5730 0x7feb500a0e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb4801b000 tx=0x7feb480046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 10 client.14649.objecter ms_handle_connect 0x7feb500a5730 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 10 client.14649.objecter resend_mon_ops 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb480273f0 con 0x7feb500a5730 2024-09-11T02:54:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 10 monclient: dump: 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7feb480273f0 con 0x7feb500a5730 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a5730 msgr2=0x7feb500a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a5730 0x7feb500a0e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb4801b000 tx=0x7feb480046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a5730 msgr2=0x7feb500a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb467fc640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb500a5730 0x7feb500a0e60 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb4801b000 tx=0x7feb480046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 143065, 2024-09-11T02:54:23.040 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:23.041 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.042 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.043 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.045 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.047 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.049 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.050 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.052 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:23.055 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.057 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:23.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.060 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:23.061 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.062 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:23.063 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 10 librados: watch_flush enter 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 10 librados: watch_flush exit 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 20 client.14649.objecter shutdown clearing up homeless session... 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 10 client.14649.objecter successfully canceled tick 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb3403df40 msgr2=0x7feb34040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb3403df40 0x7feb34040400 secure :-1 s=READY pgs=3445 cs=0 l=1 rev1=1 crypto rx=0x7feb40007fd0 tx=0x7feb40009040 comp rx=0 tx=0).stop 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 10 monclient: shutdown 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb50072640 msgr2=0x7feb501d1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 --2- 172.21.15.98:0/3173871884 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb50072640 0x7feb501d1ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feb500704e0 tx=0x7feb4c0195f0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb562dd640 1 -- 172.21.15.98:0/3173871884 reap_dead start 2024-09-11T02:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb562dd640 1 -- 172.21.15.98:0/3173871884 reap_dead start 2024-09-11T02:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 shutdown_connections 2024-09-11T02:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 >> 172.21.15.98:0/3173871884 conn(0x7feb500ac730 msgr2=0x7feb500aa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 shutdown_connections 2024-09-11T02:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 -- 172.21.15.98:0/3173871884 wait complete. 2024-09-11T02:54:23.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.013+0000 7feb57d40640 1 librados: shutdown 2024-09-11T02:54:23.066 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:23.066 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:54:23.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient: get_monmap_and_config 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient: monmap: 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:23.191710+0000 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:23.191710+0000 2024-09-11T02:54:23.190 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:23.191 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 1 Processor -- start 2024-09-11T02:54:23.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 1 -- start start 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient: init 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient: _add_conns ranks=[9,0,8,7,1,4,3,2,10,6,5] 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57f00a7030 0x7f57f0139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient(hunting): picked mon.noname-a con 0x7f57f00a7030 addr v2:172.21.15.98:3300/0 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57f006f720 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013a0e0 0x7f57f00a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient(hunting): picked mon.noname-e con 0x7f57f013a0e0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57f006ee20 con 0x7f57f006f720 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f57f00a7410 con 0x7f57f00a7030 2024-09-11T02:54:23.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f57f006f4e0 con 0x7f57f013a0e0 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f7a22640 10 monclient(hunting): authenticate will time out at 1217.071733s 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f5fbf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013a0e0 0x7f57f00a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f57be640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f57f00a7030 0x7f57f0139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f5fbf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013a0e0 0x7f57f00a5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:50762/0 (socket says 172.21.15.98:50762) 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f5fbf640 1 -- 172.21.15.98:0/3518303592 learned_addr learned my addr 172.21.15.98:0/3518303592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f5fbf640 10 monclient(hunting): get_auth_request con 0x7f57f013a0e0 auth_method 0 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f5fbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f5fbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f5fbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f57be640 10 monclient(hunting): get_auth_request con 0x7f57f00a7030 auth_method 0 2024-09-11T02:54:23.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f57be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f57be640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.185+0000 7f57f57be640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f5fbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f5fbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f5fbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 10 monclient(hunting): handle_auth_done global_id 34182 payload 274 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 1 -- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013a0e0 msgr2=0x7f57f00a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 1 --2- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013a0e0 0x7f57f00a5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 1 -- 172.21.15.98:0/3518303592 >> v1:172.21.15.98:6789/0 conn(0x7f57f006f720 legacy=0x7f57f00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 1 monclient: found mon.noname-a 2024-09-11T02:54:23.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:54:23.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 1 -- 172.21.15.98:0/3518303592 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f0070580 con 0x7f57f00a7030 2024-09-11T02:54:23.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f5fbf640 1 --2- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013a0e0 0x7f57f00a5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f57be640 1 --2- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f00a7030 0x7f57f0139ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0097f0 tx=0x7f57ec01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d03e319205faf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 1 -- 172.21.15.98:0/3518303592 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57ec0b0070 con 0x7f57f00a7030 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: dump: 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.204 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 1 -- 172.21.15.98:0/3518303592 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57ec0aa810 con 0x7f57f00a7030 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 5 monclient: authenticate success, global_id 34182 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 1 -- 172.21.15.98:0/3518303592 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57ec0a9300 con 0x7f57f00a7030 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 4 set_mon_vals no callback set 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:23.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57e7fff640 10 monclient: dump: 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.207 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 10 monclient: shutdown 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 -- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f00a7030 msgr2=0x7f57f0139ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f00a7030 0x7f57f0139ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0097f0 tx=0x7f57ec01c330 comp rx=0 tx=0).stop 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 -- 172.21.15.98:0/3518303592 shutdown_connections 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013a0e0 0x7f57f00a5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3518303592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f00a7030 0x7f57f0139ba0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 -- 172.21.15.98:0/3518303592 >> 172.21.15.98:0/3518303592 conn(0x7f57f00aa420 msgr2=0x7f57f00a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 -- 172.21.15.98:0/3518303592 shutdown_connections 2024-09-11T02:54:23.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.189+0000 7f57f7a22640 1 -- 172.21.15.98:0/3518303592 wait complete. 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: monmap: 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 librados: starting msgr at 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 librados: starting objecter 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 Processor -- start 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 -- start start 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 librados: setting wanted keys 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 librados: calling monclient init 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: init 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: _add_conns ranks=[0,7,4,8,1,5,3,6,2] 2024-09-11T02:54:23.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f006f720 0x7f57f01d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): picked mon.noname-a con 0x7f57f006f720 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f57f00a7030 0x7f57f01d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): picked mon.noname-i con 0x7f57f00a7030 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57f013a0e0 0x7f57f013b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): picked mon.noname-c con 0x7f57f013a0e0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f57f00a6600 con 0x7f57f006f720 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f57f00a6480 con 0x7f57f013a0e0 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f57f0144240 con 0x7f57f00a7030 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): authenticate will time out at 1217.077315s 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f006f720 0x7f57f01d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57f013a0e0 0x7f57f013b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f006f720 0x7f57f01d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42754/0 (socket says 172.21.15.98:42754) 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57f013a0e0 0x7f57f013b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34198/0 (socket says 172.21.15.98:34198) 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 1 -- 172.21.15.98:0/3510337372 learned_addr learned my addr 172.21.15.98:0/3510337372 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): get_auth_request con 0x7f57f006f720 auth_method 0 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): get_auth_request con 0x7f57f013a0e0 auth_method 0 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f57f00a7030 0x7f57f01d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient(hunting): get_auth_request con 0x7f57f00a7030 auth_method 0 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient(hunting): handle_auth_done global_id 23404 payload 824 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f57f00a7030 msgr2=0x7f57f01d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f57f00a7030 0x7f57f01d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f006f720 msgr2=0x7f57f01d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f006f720 0x7f57f01d13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 monclient: found mon.noname-c 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 -- 172.21.15.98:0/3510337372 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f00a7c20 con 0x7f57f013a0e0 2024-09-11T02:54:23.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57f006f720 0x7f57f01d13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f5fbf640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57f013a0e0 0x7f57f013b170 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f57e800efc0 tx=0x7f57e800c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 client.?.objecter ms_handle_connect 0x7f57f013a0e0 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57e8014480 con 0x7f57f013a0e0 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: dump: 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.215 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e8014660 con 0x7f57f013a0e0 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 5 monclient: authenticate success, global_id 23404 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: _renew_subs 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f0141420 con 0x7f57f013a0e0 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57e8014c80 con 0x7f57f013a0e0 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: dump: 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.217 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e77fe640 4 set_mon_vals no callback set 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 client.23404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: _renew_subs 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f013e690 con 0x7f57f013a0e0 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 librados: init done 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013be70 0x7f57f013c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f57f006ca10 con 0x7f57f013be70 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013be70 0x7f57f013c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient: get_auth_request con 0x7f57f013be70 auth_method 0 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f57e8014810 con 0x7f57f013a0e0 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 10 monclient(hunting): handle_auth_done global_id 23885 payload 824 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 1 -- 172.21.15.98:0/3510337372 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57f013e8d0 con 0x7f57f013be70 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f57d403df40 0x7f57d4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(190..190 src has 1..190) ==== 6158+0+0 (secure 0 0 0) 0x7f57e8019020 con 0x7f57f013a0e0 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 client.23404.objecter ms_dispatch 0x7f57f0000ba0 osd_map(190..190 src has 1..190) 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 3 client.23404.objecter handle_osd_map got epochs [190,190] > 0 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 3 client.23404.objecter handle_osd_map decoding full epoch 190 2024-09-11T02:54:23.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f57be640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013be70 0x7f57f013c250 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f57ec01a9e0 tx=0x7f57ec0190d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 20 client.23404.objecter dump_active .. 0 homeless 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 client.23404.objecter ms_handle_connect 0x7f57f013be70 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 client.23404.objecter resend_mon_ops 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f57d403df40 0x7f57d4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57f4fbd640 10 monclient: get_auth_request con 0x7f57d403df40 auth_method 0 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57ec0b0040 con 0x7f57f013be70 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.193+0000 7f57e5ffb640 10 monclient: dump: 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f57ec0aa750 con 0x7f57f013be70 2024-09-11T02:54:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013be70 msgr2=0x7f57f013c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013be70 0x7f57f013c250 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f57ec01a9e0 tx=0x7f57ec0190d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013be70 msgr2=0x7f57f013c250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f013be70 0x7f57f013c250 secure :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f57ec01a9e0 tx=0x7f57ec0190d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57e5ffb640 10 client.23404.objecter ms_handle_connect 0x7f57d403df40 2024-09-11T02:54:23.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.197+0000 7f57f4fbd640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f57d403df40 0x7f57d4040400 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7f57e0007fd0 tx=0x7f57e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:23.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f002d450 0x7f57f002d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f57f006e870 con 0x7f57f002d450 2024-09-11T02:54:23.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f002d450 0x7f57f002d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient: get_auth_request con 0x7f57f002d450 auth_method 0 2024-09-11T02:54:23.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 10 monclient(hunting): handle_auth_done global_id 23894 payload 824 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 1 -- 172.21.15.98:0/3510337372 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57f0139ba0 con 0x7f57f002d450 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f57be640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f002d450 0x7f57f002d860 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f57ec000c10 tx=0x7f57ec0a8000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 10 client.23404.objecter ms_handle_connect 0x7f57f002d450 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 10 client.23404.objecter resend_mon_ops 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f57ec003de0 con 0x7f57f002d450 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 10 monclient: dump: 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.227 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f57ec003de0 con 0x7f57f002d450 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f002d450 msgr2=0x7f57f002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f002d450 0x7f57f002d860 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f57ec000c10 tx=0x7f57ec0a8000 comp rx=0 tx=0).stop 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f002d450 msgr2=0x7f57f002d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57e5ffb640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f57f002d450 0x7f57f002d860 secure :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f57ec000c10 tx=0x7f57ec0a8000 comp rx=0 tx=0).stop 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 278994, 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:23.229 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.230 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:23.232 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:23.233 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.234 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.237 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.239 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:23.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.246 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.250 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:23.251 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 10 librados: watch_flush enter 2024-09-11T02:54:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 10 librados: watch_flush exit 2024-09-11T02:54:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 20 client.23404.objecter shutdown clearing up homeless session... 2024-09-11T02:54:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 10 client.23404.objecter successfully canceled tick 2024-09-11T02:54:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f57d403df40 msgr2=0x7f57d4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f57d403df40 0x7f57d4040400 secure :-1 s=READY pgs=3448 cs=0 l=1 rev1=1 crypto rx=0x7f57e0007fd0 tx=0x7f57e0009040 comp rx=0 tx=0).stop 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 10 monclient: shutdown 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57f013a0e0 msgr2=0x7f57f013b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57f013a0e0 0x7f57f013b170 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f57e800efc0 tx=0x7f57e800c490 comp rx=0 tx=0).stop 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f5fbf640 1 -- 172.21.15.98:0/3510337372 reap_dead start 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 shutdown_connections 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 --2- 172.21.15.98:0/3510337372 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f57f013a0e0 0x7f57f013b170 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 >> 172.21.15.98:0/3510337372 conn(0x7f57f00aa420 msgr2=0x7f57f00a3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 shutdown_connections 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 -- 172.21.15.98:0/3510337372 wait complete. 2024-09-11T02:54:23.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.201+0000 7f57f7a22640 1 librados: shutdown 2024-09-11T02:54:23.255 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:23.255 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:54:23.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:54:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient: get_monmap_and_config 2024-09-11T02:54:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient: monmap: 2024-09-11T02:54:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:23.377568+0000 2024-09-11T02:54:23.376 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:23.377568+0000 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.377 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:23.378 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:23.378 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 1 Processor -- start 2024-09-11T02:54:23.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 1 -- start start 2024-09-11T02:54:23.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient: init 2024-09-11T02:54:23.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient: _add_conns ranks=[1,7,0,2,3,8,10,6,9,4,5] 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7feda4139b10 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 0x7feda413ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient(hunting): picked mon.noname-i con 0x7feda4072600 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feda413ef60 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda413a600 con 0x7feda413ef60 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda40702b0 con 0x7feda4139b10 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7feda4071bf0 con 0x7feda4072600 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 10 monclient(hunting): authenticate will time out at 1217.257892s 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda9187640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feda413ef60 0x7feda4141350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56660/0 (socket says 172.21.15.98:56660) 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda9187640 1 -- 172.21.15.98:0/4257483223 learned_addr learned my addr 172.21.15.98:0/4257483223 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:23.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 0x7feda413ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): get_auth_request con 0x7feda4072600 auth_method 0 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2171418720 0 0) 0x7feda413a600 con 0x7feda413ef60 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): _negotiate 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): my global_id is 34209 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed8c003530 con 0x7feda413ef60 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1405249760 0 0) 0x7fed8c003530 con 0x7feda413ef60 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 msgr2=0x7feda413ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 0x7feda413ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 >> v1:172.21.15.190:6789/0 conn(0x7feda4139b10 legacy=0x7feda4139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda4071350 con 0x7feda413ef60 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fed9c001cf0 con 0x7feda413ef60 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: dump: 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.393 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 >> v1:172.21.15.98:6789/0 conn(0x7feda413ef60 legacy=0x7feda4141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient: _add_conns ranks=[6,7,1,2,4,0,3,8,5] 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8c004240 0x7fed8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): picked mon.d con 0x7fed8c004240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:23.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 0x7fed8c004e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): picked mon.i con 0x7feda4072600 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c0099c0 0x7fed8c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): picked mon.f con 0x7fed8c0099c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feda4071bf0 con 0x7fed8c0099c0 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fed8c00c3f0 con 0x7fed8c004240 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fed8c00c5a0 con 0x7feda4072600 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda37fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8c004240 0x7fed8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): get_auth_request con 0x7fed8c004240 auth_method 0 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 0x7fed8c004e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda9187640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c0099c0 0x7fed8c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): get_auth_request con 0x7feda4072600 auth_method 0 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:23.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda9187640 10 monclient(hunting): get_auth_request con 0x7fed8c0099c0 auth_method 0 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda9187640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda9187640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda9187640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient(hunting): handle_auth_done global_id 34209 payload 293 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 1 -- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 msgr2=0x7fed8c004e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 0x7fed8c004e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 1 -- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c0099c0 msgr2=0x7fed8c00bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c0099c0 0x7fed8c00bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 1 monclient: found mon.d 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 1 -- 172.21.15.98:0/4257483223 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c003530 con 0x7fed8c004240 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda8986640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 5 monclient: authenticate success, global_id 34209 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7fedaabea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.373+0000 7feda3fff640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feda4072600 0x7fed8c004e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda8986640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8c004240 0x7fed8c004810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fed98001790 tx=0x7fed9801c100 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c29d372ab4b009e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed980013e0 con 0x7fed8c004240 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 10 monclient: dump: 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed98003600 con 0x7fed8c004240 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 4 set_mon_vals no callback set 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 1 -- 172.21.15.98:0/4257483223 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed98003c20 con 0x7fed8c004240 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda37fe640 10 monclient: dump: 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 10 monclient: shutdown 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 -- 172.21.15.98:0/4257483223 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8c004240 msgr2=0x7fed8c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 --2- 172.21.15.98:0/4257483223 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8c004240 0x7fed8c004810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fed98001790 tx=0x7fed9801c100 comp rx=0 tx=0).stop 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7feda9187640 1 -- 172.21.15.98:0/4257483223 reap_dead start 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 -- 172.21.15.98:0/4257483223 shutdown_connections 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 -- 172.21.15.98:0/4257483223 >> 172.21.15.98:0/4257483223 conn(0x7feda40ac790 msgr2=0x7feda40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 -- 172.21.15.98:0/4257483223 shutdown_connections 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 -- 172.21.15.98:0/4257483223 wait complete. 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 10 monclient: monmap: 2024-09-11T02:54:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.403 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 librados: starting msgr at 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 librados: starting objecter 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 Processor -- start 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 -- start start 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 librados: setting wanted keys 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 1 librados: calling monclient init 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 10 monclient: init 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.377+0000 7fedaabea640 10 monclient: _add_conns ranks=[3,5,1,8,6,0,4,7,2] 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feda4136800 0x7feda4136be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): picked mon.noname-g con 0x7feda4136800 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7feda40a1db0 0x7feda40a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): picked mon.noname-h con 0x7feda40a1db0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda40a2730 0x7feda40a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): picked mon.noname-f con 0x7feda40a2730 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feda4143280 con 0x7feda40a2730 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feda40517e0 con 0x7feda4136800 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7feda4143400 con 0x7feda40a1db0 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): authenticate will time out at 1217.264094s 2024-09-11T02:54:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feda4136800 0x7feda4136be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda40a2730 0x7feda40a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7feda40a1db0 0x7feda40a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda40a2730 0x7feda40a2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:59706/0 (socket says 172.21.15.98:59706) 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 -- 172.21.15.98:0/2979847525 learned_addr learned my addr 172.21.15.98:0/2979847525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): get_auth_request con 0x7feda4136800 auth_method 0 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): get_auth_request con 0x7feda40a2730 auth_method 0 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 10 monclient(hunting): get_auth_request con 0x7feda40a1db0 auth_method 0 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient(hunting): handle_auth_done global_id 24760 payload 824 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7feda40a1db0 msgr2=0x7feda40a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7feda40a1db0 0x7feda40a21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feda4136800 msgr2=0x7feda4136be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feda4136800 0x7feda4136be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 monclient: found mon.noname-f 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 -- 172.21.15.98:0/2979847525 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda40a6e80 con 0x7feda40a2730 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda9187640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda40a2730 0x7feda40a2b90 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7feda40a9ed0 tx=0x7fed9c00e530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 client.?.objecter ms_handle_connect 0x7feda40a2730 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed9c013890 con 0x7feda40a2730 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: dump: 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.409 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed9c013a70 con 0x7feda40a2730 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 5 monclient: authenticate success, global_id 24760 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed9c013c20 con 0x7feda40a2730 2024-09-11T02:54:23.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda2ffd640 4 set_mon_vals no callback set 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 monclient: dump: 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.411 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient: _renew_subs 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda40a58f0 con 0x7feda40a2730 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 client.24760.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient: _renew_subs 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda40a43d0 con 0x7feda40a2730 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 librados: init done 2024-09-11T02:54:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413a950 0x7feda413ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feda406ca10 con 0x7feda413a950 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413a950 0x7feda413ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient: get_auth_request con 0x7feda413a950 auth_method 0 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fed9c013420 con 0x7feda40a2730 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed9003df70 0x7fed90040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 6158+0+0 (secure 0 0 0) 0x7fed9c018020 con 0x7feda40a2730 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 10 client.24760.objecter ms_dispatch 0x7feda4000ba0 osd_map(190..190 src has 1..190) 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 3 client.24760.objecter handle_osd_map got epochs [190,190] > 0 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 3 client.24760.objecter handle_osd_map decoding full epoch 190 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.381+0000 7feda17fa640 20 client.24760.objecter dump_active .. 0 homeless 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda3fff640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed9003df70 0x7fed90040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda8986640 10 monclient(hunting): handle_auth_done global_id 24769 payload 824 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda8986640 1 -- 172.21.15.98:0/2979847525 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feda40a6a00 con 0x7feda413a950 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda3fff640 10 monclient: get_auth_request con 0x7fed9003df70 auth_method 0 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 client.24760.objecter ms_handle_connect 0x7feda413a950 2024-09-11T02:54:23.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 client.24760.objecter resend_mon_ops 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda8986640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413a950 0x7feda413ad30 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fed98007fd0 tx=0x7fed8c005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed980014c0 con 0x7feda413a950 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 monclient: dump: 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 client.24760.objecter ms_handle_connect 0x7fed9003df70 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda3fff640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed9003df70 0x7fed90040430 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7feda4137770 tx=0x7fed94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed980067e0 con 0x7feda413a950 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413a950 msgr2=0x7feda413ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413a950 0x7feda413ad30 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fed98007fd0 tx=0x7fed8c005840 comp rx=0 tx=0).stop 2024-09-11T02:54:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413a950 msgr2=0x7feda413ad30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.385+0000 7feda17fa640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413a950 0x7feda413ad30 secure :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fed98007fd0 tx=0x7fed8c005840 comp rx=0 tx=0).stop 2024-09-11T02:54:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.389+0000 7fedaabea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.389+0000 7fedaabea640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413b270 0x7feda41dc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.389+0000 7fedaabea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.389+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7feda406e870 con 0x7feda413b270 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413b270 0x7feda41dc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient: get_auth_request con 0x7feda413b270 auth_method 0 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 10 monclient(hunting): handle_auth_done global_id 24778 payload 824 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 1 -- 172.21.15.98:0/2979847525 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feda4142490 con 0x7feda413b270 2024-09-11T02:54:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda8986640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413b270 0x7feda41dc6d0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fed9801c9a0 tx=0x7fed980a6000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 10 client.24760.objecter ms_handle_connect 0x7feda413b270 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 10 client.24760.objecter resend_mon_ops 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed98003a70 con 0x7feda413b270 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 10 monclient: dump: 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9603 (secure 0 0 0) 0x7fed98003600 con 0x7feda413b270 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413b270 msgr2=0x7feda41dc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413b270 0x7feda41dc6d0 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fed9801c9a0 tx=0x7fed980a6000 comp rx=0 tx=0).stop 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413b270 msgr2=0x7feda41dc6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda17fa640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda413b270 0x7feda41dc6d0 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:23.422 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 279164, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:23.423 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.424 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.425 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.427 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:23.428 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.429 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.430 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:23.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.435 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:23.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:23.441 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.443 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:23.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.446 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.447 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.448 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:23.449 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 10 librados: watch_flush enter 2024-09-11T02:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 10 librados: watch_flush exit 2024-09-11T02:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 20 client.24760.objecter shutdown clearing up homeless session... 2024-09-11T02:54:23.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 10 client.24760.objecter successfully canceled tick 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed9003df70 msgr2=0x7fed90040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed9003df70 0x7fed90040430 secure :-1 s=READY pgs=3449 cs=0 l=1 rev1=1 crypto rx=0x7feda4137770 tx=0x7fed94008040 comp rx=0 tx=0).stop 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 10 monclient: shutdown 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda40a2730 msgr2=0x7feda40a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 --2- 172.21.15.98:0/2979847525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7feda40a2730 0x7feda40a2b90 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7feda40a9ed0 tx=0x7fed9c00e530 comp rx=0 tx=0).stop 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda9187640 1 -- 172.21.15.98:0/2979847525 reap_dead start 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7feda9187640 1 -- 172.21.15.98:0/2979847525 reap_dead start 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 shutdown_connections 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 >> 172.21.15.98:0/2979847525 conn(0x7feda40ac790 msgr2=0x7feda40a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 shutdown_connections 2024-09-11T02:54:23.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 -- 172.21.15.98:0/2979847525 wait complete. 2024-09-11T02:54:23.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.393+0000 7fedaabea640 1 librados: shutdown 2024-09-11T02:54:23.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:23.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:54:23.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:54:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.569+0000 7f3da7837640 10 monclient: get_monmap_and_config 2024-09-11T02:54:23.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.569+0000 7f3da7837640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.569+0000 7f3da7837640 10 monclient: monmap: 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:23.577130+0000 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:23.577130+0000 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.576 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:23.577 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:23.577 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 Processor -- start 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 -- start start 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient: init 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient: _add_conns ranks=[3,8,7,0,10,6,1,4,5,9,2] 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0139b10 0x7f3da0072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): picked mon.noname-g con 0x7f3da0139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:23.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3da0139ef0 0x7f3da00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): picked mon.noname-e con 0x7f3da0139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3da013a2d0 0x7f3da00a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): picked mon.noname-i con 0x7f3da013a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3da00702b0 con 0x7f3da0139b10 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3da0071bf0 con 0x7f3da013a2d0 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3da0071350 con 0x7f3da0139ef0 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da7837640 10 monclient(hunting): authenticate will time out at 1217.457564s 2024-09-11T02:54:23.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3da0139ef0 0x7f3da00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3da0139ef0 0x7f3da00a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:50776/0 (socket says 172.21.15.98:50776) 2024-09-11T02:54:23.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 -- 172.21.15.98:0/2002681427 learned_addr learned my addr 172.21.15.98:0/2002681427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): get_auth_request con 0x7f3da0139ef0 auth_method 0 2024-09-11T02:54:23.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient(hunting): handle_auth_done global_id 23903 payload 274 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 -- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3da013a2d0 msgr2=0x7f3da00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da5dd4640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3da013a2d0 0x7f3da00a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da55d3640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0139b10 0x7f3da0072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3da013a2d0 0x7f3da00a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 -- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0139b10 msgr2=0x7f3da0072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0139b10 0x7f3da0072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 monclient: found mon.noname-e 2024-09-11T02:54:23.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:23.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.573+0000 7f3da4dd2640 1 -- 172.21.15.98:0/2002681427 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da00a9620 con 0x7f3da0139ef0 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da4dd2640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3da0139ef0 0x7f3da00a7a10 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3d900098a0 tx=0x7f3d9001c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5e9d8da5f9084ba2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 1 -- 172.21.15.98:0/2002681427 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d900b0070 con 0x7f3da0139ef0 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: dump: 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.591 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 1 -- 172.21.15.98:0/2002681427 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d900aa810 con 0x7f3da0139ef0 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 5 monclient: authenticate success, global_id 23903 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 4 set_mon_vals no callback set 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 1 -- 172.21.15.98:0/2002681427 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d900a9300 con 0x7f3da0139ef0 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d97fff640 10 monclient: dump: 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: shutdown 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- 172.21.15.98:0/2002681427 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3da0139ef0 msgr2=0x7f3da00a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3da0139ef0 0x7f3da00a7a10 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3d900098a0 tx=0x7f3d9001c440 comp rx=0 tx=0).stop 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- 172.21.15.98:0/2002681427 shutdown_connections 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3da0139ef0 0x7f3da00a7a10 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3da013a2d0 0x7f3da00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 --2- 172.21.15.98:0/2002681427 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0139b10 0x7f3da0072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- 172.21.15.98:0/2002681427 >> 172.21.15.98:0/2002681427 conn(0x7f3da00ac790 msgr2=0x7f3da00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- 172.21.15.98:0/2002681427 shutdown_connections 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- 172.21.15.98:0/2002681427 wait complete. 2024-09-11T02:54:23.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: monmap: 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 librados: starting msgr at 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 librados: starting objecter 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 Processor -- start 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- start start 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 librados: setting wanted keys 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 librados: calling monclient init 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: init 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: _add_conns ranks=[2,1,4,5,7,0,6,8,3] 2024-09-11T02:54:23.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3da0139b10 0x7f3da013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): picked mon.noname-b con 0x7f3da0139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da0139ef0 0x7f3da013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): picked mon.noname-f con 0x7f3da0139ef0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3da013a2d0 0x7f3da013b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): picked mon.noname-c con 0x7f3da013a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3da0143460 con 0x7f3da0139ef0 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3da00a88a0 con 0x7f3da0139b10 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3da01435e0 con 0x7f3da013a2d0 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient(hunting): authenticate will time out at 1217.462598s 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3da0139b10 0x7f3da013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3da013a2d0 0x7f3da013b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3da0139b10 0x7f3da013a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50802/0 (socket says 172.21.15.98:50802) 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3da013a2d0 0x7f3da013b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34200/0 (socket says 172.21.15.98:34200) 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 1 -- 172.21.15.98:0/791802967 learned_addr learned my addr 172.21.15.98:0/791802967 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 10 monclient(hunting): get_auth_request con 0x7f3da0139b10 auth_method 0 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): get_auth_request con 0x7f3da013a2d0 auth_method 0 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient(hunting): handle_auth_done global_id 23413 payload 824 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da4dd2640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da0139ef0 0x7f3da013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3da0139b10 msgr2=0x7f3da013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3da0139b10 0x7f3da013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da0139ef0 msgr2=0x7f3da013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da0139ef0 0x7f3da013ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 monclient: found mon.noname-c 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 -- 172.21.15.98:0/791802967 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da00a9f20 con 0x7f3da013a2d0 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da4dd2640 10 monclient: get_auth_request con 0x7f3da0139ef0 auth_method 0 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da4dd2640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da0139ef0 0x7f3da013ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da5dd4640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3da013a2d0 0x7f3da013b250 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800cc50 tx=0x7f3d9800a5c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 client.?.objecter ms_handle_connect 0x7f3da013a2d0 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d9800ce00 con 0x7f3da013a2d0 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: dump: 2024-09-11T02:54:23.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.602 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d9800f860 con 0x7f3da013a2d0 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d980173b0 con 0x7f3da013a2d0 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d95ffb640 10 monclient: dump: 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.603 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3d977fe640 4 set_mon_vals no callback set 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 5 monclient: authenticate success, global_id 23413 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: _renew_subs 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da00a58c0 con 0x7f3da013a2d0 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 client.23413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: _renew_subs 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da01407b0 con 0x7f3da013a2d0 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.577+0000 7f3da7837640 1 librados: init done 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da7837640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da7837640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da01337f0 0x7f3da0133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3da006ca10 con 0x7f3da01337f0 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3d9800f3f0 con 0x7f3da013a2d0 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da01337f0 0x7f3da0133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient: get_auth_request con 0x7f3da01337f0 auth_method 0 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d8403df40 0x7f3d84040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(190..190 src has 1..190) ==== 6158+0+0 (secure 0 0 0) 0x7f3d98009030 con 0x7f3da013a2d0 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 client.23413.objecter ms_dispatch 0x7f3da0000ba0 osd_map(190..190 src has 1..190) 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 3 client.23413.objecter handle_osd_map got epochs [190,190] > 0 2024-09-11T02:54:23.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 3 client.23413.objecter handle_osd_map decoding full epoch 190 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 20 client.23413.objecter dump_active .. 0 homeless 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da4dd2640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d8403df40 0x7f3d84040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da4dd2640 10 monclient: get_auth_request con 0x7f3d8403df40 auth_method 0 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 10 monclient(hunting): handle_auth_done global_id 19902 payload 824 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 1 -- 172.21.15.98:0/791802967 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3da00aa470 con 0x7f3da01337f0 2024-09-11T02:54:23.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 client.23413.objecter ms_handle_connect 0x7f3da01337f0 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 client.23413.objecter resend_mon_ops 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da55d3640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da01337f0 0x7f3da0133bd0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00f950 tx=0x7f3d9c00fe10 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 client.23413.objecter ms_handle_connect 0x7f3d8403df40 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3da4dd2640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d8403df40 0x7f3d84040400 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f3d900ad000 tx=0x7f3d900a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d9c00de90 con 0x7f3da01337f0 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 monclient: dump: 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d9c00da20 con 0x7f3da01337f0 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da01337f0 msgr2=0x7f3da0133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da01337f0 0x7f3da0133bd0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00f950 tx=0x7f3d9c00fe10 comp rx=0 tx=0).stop 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da01337f0 msgr2=0x7f3da0133bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.581+0000 7f3d95ffb640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da01337f0 0x7f3da0133bd0 secure :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00f950 tx=0x7f3d9c00fe10 comp rx=0 tx=0).stop 2024-09-11T02:54:23.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:23.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0134110 0x7f3da0137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3da006e870 con 0x7f3da0134110 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0134110 0x7f3da0137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient: get_auth_request con 0x7f3da0134110 auth_method 0 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 10 monclient(hunting): handle_auth_done global_id 19911 payload 824 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 1 -- 172.21.15.98:0/791802967 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3da006e680 con 0x7f3da0134110 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 10 client.23413.objecter ms_handle_connect 0x7f3da0134110 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 10 client.23413.objecter resend_mon_ops 2024-09-11T02:54:23.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da55d3640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0134110 0x7f3da0137bc0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00f950 tx=0x7f3d9c0046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d9c00dc90 con 0x7f3da0134110 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 10 monclient: dump: 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f3d9c00dc90 con 0x7f3da0134110 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:23.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0134110 msgr2=0x7f3da0137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0134110 0x7f3da0137bc0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00f950 tx=0x7f3d9c0046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0134110 msgr2=0x7f3da0137bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3d95ffb640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3da0134110 0x7f3da0137bc0 secure :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00f950 tx=0x7f3d9c0046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:23.616 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 279340, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:23.617 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:23.619 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:23.621 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:23.622 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.623 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:23.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.640 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.641 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:23.642 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:23.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 10 librados: watch_flush enter 2024-09-11T02:54:23.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 10 librados: watch_flush exit 2024-09-11T02:54:23.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 20 client.23413.objecter shutdown clearing up homeless session... 2024-09-11T02:54:23.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 10 client.23413.objecter successfully canceled tick 2024-09-11T02:54:23.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d8403df40 msgr2=0x7f3d84040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d8403df40 0x7f3d84040400 secure :-1 s=READY pgs=3450 cs=0 l=1 rev1=1 crypto rx=0x7f3d900ad000 tx=0x7f3d900a6000 comp rx=0 tx=0).stop 2024-09-11T02:54:23.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 10 monclient: shutdown 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3da013a2d0 msgr2=0x7f3da013b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3da013a2d0 0x7f3da013b250 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800cc50 tx=0x7f3d9800a5c0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da5dd4640 1 -- 172.21.15.98:0/791802967 reap_dead start 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 shutdown_connections 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 --2- 172.21.15.98:0/791802967 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3da013a2d0 0x7f3da013b250 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 >> 172.21.15.98:0/791802967 conn(0x7f3da00ac790 msgr2=0x7f3da00aa6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 shutdown_connections 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 -- 172.21.15.98:0/791802967 wait complete. 2024-09-11T02:54:23.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.589+0000 7f3da7837640 1 librados: shutdown 2024-09-11T02:54:23.646 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:23.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:54:23.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:54:23.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.757+0000 7fc1986a0640 10 monclient: get_monmap_and_config 2024-09-11T02:54:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.757+0000 7fc1986a0640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.757+0000 7fc1986a0640 10 monclient: monmap: 2024-09-11T02:54:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:23.765111+0000 2024-09-11T02:54:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:23.765111+0000 2024-09-11T02:54:23.764 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:23.765 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:23.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 1 Processor -- start 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 1 -- start start 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient: init 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient: _add_conns ranks=[4,0,8,9,6,2,10,7,3,5,1] 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc1900a29d0 0x7fc1900a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient(hunting): picked mon.noname-c con 0x7fc1900a29d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1900a2090 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc1900a4db0 0x7fc1901418a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient(hunting): picked mon.noname-e con 0x7fc1900a4db0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:23.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1900a0e30 con 0x7fc1900a2090 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc190077740 con 0x7fc1900a29d0 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc190077480 con 0x7fc1900a4db0 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc1986a0640 10 monclient(hunting): authenticate will time out at 1217.645418s 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc1900a29d0 0x7fc1900a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc1900a4db0 0x7fc1901418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc1900a29d0 0x7fc1900a2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34204/0 (socket says 172.21.15.98:34204) 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc1900a4db0 0x7fc1901418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:50786/0 (socket says 172.21.15.98:50786) 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 1 -- 172.21.15.98:0/867706155 learned_addr learned my addr 172.21.15.98:0/867706155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 10 monclient(hunting): get_auth_request con 0x7fc1900a29d0 auth_method 0 2024-09-11T02:54:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): get_auth_request con 0x7fc1900a4db0 auth_method 0 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19543a640 1 -- 172.21.15.98:0/867706155 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339259541 0 0) 0x7fc18c0014a0 con 0x7fc1900a2090 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient(hunting): handle_auth_done global_id 23912 payload 274 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 1 -- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc1900a29d0 msgr2=0x7fc1900a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 1 --2- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc1900a29d0 0x7fc1900a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 1 -- 172.21.15.98:0/867706155 >> v1:172.21.15.98:6789/0 conn(0x7fc1900a2090 legacy=0x7fc1900a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 1 monclient: found mon.noname-e 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc196c3d640 1 -- 172.21.15.98:0/867706155 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1900a1110 con 0x7fc1900a4db0 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19543a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.761+0000 7fc19643c640 1 --2- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc1900a29d0 0x7fc1900a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 --2- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc1900a4db0 0x7fc1901418a0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fc1800097f0 tx=0x7fc18001c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=97f77892103df7a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 1 -- 172.21.15.98:0/867706155 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1800b0070 con 0x7fc1900a4db0 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: dump: 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 5 monclient: authenticate success, global_id 23912 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 1 -- 172.21.15.98:0/867706155 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1800aa810 con 0x7fc1900a4db0 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 4 set_mon_vals no callback set 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 1 -- 172.21.15.98:0/867706155 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1800aa3a0 con 0x7fc1900a4db0 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19543a640 10 monclient: dump: 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient: shutdown 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc1900a4db0 msgr2=0x7fc1901418a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 --2- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc1900a4db0 0x7fc1901418a0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fc1800097f0 tx=0x7fc18001c3a0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- 172.21.15.98:0/867706155 shutdown_connections 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 --2- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc1900a4db0 0x7fc1901418a0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 --2- 172.21.15.98:0/867706155 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc1900a29d0 0x7fc1900a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- 172.21.15.98:0/867706155 >> 172.21.15.98:0/867706155 conn(0x7fc19006eed0 msgr2=0x7fc19006f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- 172.21.15.98:0/867706155 shutdown_connections 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- 172.21.15.98:0/867706155 wait complete. 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient: monmap: 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 librados: starting msgr at 2024-09-11T02:54:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 librados: starting objecter 2024-09-11T02:54:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 Processor -- start 2024-09-11T02:54:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- start start 2024-09-11T02:54:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 librados: setting wanted keys 2024-09-11T02:54:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 librados: calling monclient init 2024-09-11T02:54:23.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient: init 2024-09-11T02:54:23.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient: _add_conns ranks=[3,7,0,5,1,4,8,2,6] 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1900a2090 0x7fc190137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): picked mon.noname-g con 0x7fc1900a2090 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc1900a29d0 0x7fc190138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): picked mon.noname-i con 0x7fc1900a29d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1900a4db0 0x7fc190138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): picked mon.noname-a con 0x7fc1900a4db0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc190144990 con 0x7fc1900a4db0 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc190144810 con 0x7fc1900a2090 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc190144b10 con 0x7fc1900a29d0 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 10 monclient(hunting): authenticate will time out at 1217.651086s 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1900a4db0 0x7fc190138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1900a4db0 0x7fc190138890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42768/0 (socket says 172.21.15.98:42768) 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 -- 172.21.15.98:0/2140008815 learned_addr learned my addr 172.21.15.98:0/2140008815 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): get_auth_request con 0x7fc1900a4db0 auth_method 0 2024-09-11T02:54:23.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19643c640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1900a2090 0x7fc190137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient(hunting): handle_auth_done global_id 34227 payload 824 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc195c3b640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc1900a29d0 0x7fc190138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc1900a29d0 msgr2=0x7fc190138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc1900a29d0 0x7fc190138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1900a2090 msgr2=0x7fc190137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1900a2090 0x7fc190137e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 monclient: found mon.noname-a 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 -- 172.21.15.98:0/2140008815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc190141160 con 0x7fc1900a4db0 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19643c640 10 monclient: get_auth_request con 0x7fc1900a2090 auth_method 0 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc19643c640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc1900a2090 0x7fc190137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc196c3d640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1900a4db0 0x7fc190138890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc1800ae040 tx=0x7fc180019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 10 client.?.objecter ms_handle_connect 0x7fc1900a4db0 2024-09-11T02:54:23.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1800b0050 con 0x7fc1900a4db0 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 10 monclient: dump: 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1800bc070 con 0x7fc1900a4db0 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.765+0000 7fc1986a0640 5 monclient: authenticate success, global_id 34227 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 10 monclient: _renew_subs 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc190141440 con 0x7fc1900a4db0 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1800044b0 con 0x7fc1900a4db0 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: dump: 2024-09-11T02:54:23.793 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.794 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc194c39640 4 set_mon_vals no callback set 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 10 client.34227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 10 monclient: _renew_subs 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1901408b0 con 0x7fc1900a4db0 2024-09-11T02:54:23.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 1 librados: init done 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013dac0 0x7fc19013dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc19006ca10 con 0x7fc19013dac0 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc180004040 con 0x7fc1900a4db0 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013dac0 0x7fc19013dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient: get_auth_request con 0x7fc19013dac0 auth_method 0 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc17c03df70 0x7fc17c040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 6158+0+0 (secure 0 0 0) 0x7fc1800e9e40 con 0x7fc1900a4db0 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 client.34227.objecter ms_dispatch 0x7fc190000ba0 osd_map(190..190 src has 1..190) 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 3 client.34227.objecter handle_osd_map got epochs [190,190] > 0 2024-09-11T02:54:23.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 3 client.34227.objecter handle_osd_map decoding full epoch 190 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 20 client.34227.objecter dump_active .. 0 homeless 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc195c3b640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc17c03df70 0x7fc17c040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc195c3b640 10 monclient: get_auth_request con 0x7fc17c03df70 auth_method 0 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 10 monclient(hunting): handle_auth_done global_id 19607 payload 824 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 1 -- 172.21.15.98:0/2140008815 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc19013cbf0 con 0x7fc19013dac0 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc19643c640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013dac0 0x7fc19013dea0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc188010790 tx=0x7fc188010c50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 client.34227.objecter ms_handle_connect 0x7fc19013dac0 2024-09-11T02:54:23.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 client.34227.objecter resend_mon_ops 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc18800f040 con 0x7fc19013dac0 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: dump: 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 client.34227.objecter ms_handle_connect 0x7fc17c03df70 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc195c3b640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc17c03df70 0x7fc17c040430 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0012f0 tx=0x7fc18c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc1880143f0 con 0x7fc19013dac0 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013dac0 msgr2=0x7fc19013dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013dac0 0x7fc19013dea0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc188010790 tx=0x7fc188010c50 comp rx=0 tx=0).stop 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013dac0 msgr2=0x7fc19013dea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.769+0000 7fc186ffd640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013dac0 0x7fc19013dea0 secure :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc188010790 tx=0x7fc188010c50 comp rx=0 tx=0).stop 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013e3e0 0x7fc1901397d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc19006e870 con 0x7fc19013e3e0 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013e3e0 0x7fc1901397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient: get_auth_request con 0x7fc19013e3e0 auth_method 0 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 10 monclient(hunting): handle_auth_done global_id 19616 payload 824 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 1 -- 172.21.15.98:0/2140008815 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc1900a2db0 con 0x7fc19013e3e0 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 10 client.34227.objecter ms_handle_connect 0x7fc19013e3e0 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 10 client.34227.objecter resend_mon_ops 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc19643c640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013e3e0 0x7fc1901397d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fc188010790 tx=0x7fc1880046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc188025070 con 0x7fc19013e3e0 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 10 monclient: dump: 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.803 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fc188006000 con 0x7fc19013e3e0 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013e3e0 msgr2=0x7fc1901397d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013e3e0 0x7fc1901397d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fc188010790 tx=0x7fc1880046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013e3e0 msgr2=0x7fc1901397d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc186ffd640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc19013e3e0 0x7fc1901397d0 secure :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fc188010790 tx=0x7fc1880046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.805 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:23.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:23.805 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:23.805 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:23.805 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:23.805 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 279528, 2024-09-11T02:54:23.805 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:23.806 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.807 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.808 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.810 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.811 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:23.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.818 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.820 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.822 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.824 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.826 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:23.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:23.828 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.829 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.830 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:23.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.830 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.830 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:23.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.831 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.833 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:23.833 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 10 librados: watch_flush enter 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 10 librados: watch_flush exit 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 20 client.34227.objecter shutdown clearing up homeless session... 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 10 client.34227.objecter successfully canceled tick 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc17c03df70 msgr2=0x7fc17c040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc17c03df70 0x7fc17c040430 secure :-1 s=READY pgs=3451 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0012f0 tx=0x7fc18c009040 comp rx=0 tx=0).stop 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 10 monclient: shutdown 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1900a4db0 msgr2=0x7fc190138890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 --2- 172.21.15.98:0/2140008815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1900a4db0 0x7fc190138890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc1800ae040 tx=0x7fc180019140 comp rx=0 tx=0).stop 2024-09-11T02:54:23.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc196c3d640 1 -- 172.21.15.98:0/2140008815 reap_dead start 2024-09-11T02:54:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc196c3d640 1 -- 172.21.15.98:0/2140008815 reap_dead start 2024-09-11T02:54:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 shutdown_connections 2024-09-11T02:54:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 >> 172.21.15.98:0/2140008815 conn(0x7fc19006eed0 msgr2=0x7fc19009e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 shutdown_connections 2024-09-11T02:54:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 -- 172.21.15.98:0/2140008815 wait complete. 2024-09-11T02:54:23.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.777+0000 7fc1986a0640 1 librados: shutdown 2024-09-11T02:54:23.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:23.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:54:23.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:54:23.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient: get_monmap_and_config 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient: monmap: 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:23.946135+0000 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:23.946135+0000 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:23.945 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:23.946 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 1 Processor -- start 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 1 -- start start 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient: init 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient: _add_conns ranks=[1,3,2,5,8,7,0,6,4,10,9] 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f21e40a3690 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21e40a1eb0 0x7f21e413c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient(hunting): picked mon.noname-g con 0x7f21e40a1eb0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21e40a4180 0x7f21e413ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient(hunting): picked mon.noname-b con 0x7f21e40a4180 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21e40a1ca0 con 0x7f21e40a3690 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f21e409ec40 con 0x7f21e40a4180 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f21e409dd20 con 0x7f21e40a1eb0 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e8b63640 10 monclient(hunting): authenticate will time out at 1217.826373s 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21e40a4180 0x7f21e413ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21e40a4180 0x7f21e413ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50814/0 (socket says 172.21.15.98:50814) 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2851219535 learned_addr learned my addr 172.21.15.98:0/2851219535 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request con 0x7f21e40a4180 auth_method 0 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e1d9b640 1 --2- 172.21.15.98:0/2851219535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21e40a1eb0 0x7f21e413c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_done global_id 23438 payload 274 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2851219535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21e40a1eb0 msgr2=0x7f21e413c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 --2- 172.21.15.98:0/2851219535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21e40a1eb0 0x7f21e413c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2851219535 >> v1:172.21.15.190:6789/0 conn(0x7f21e40a3690 legacy=0x7f21e40a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 monclient: found mon.noname-b 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:23.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2851219535 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e409d480 con 0x7f21e40a4180 2024-09-11T02:54:23.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e1d9b640 10 monclient: get_auth_request con 0x7f21e40a1eb0 auth_method 0 2024-09-11T02:54:23.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.941+0000 7f21e1d9b640 1 --2- 172.21.15.98:0/2851219535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21e40a1eb0 0x7f21e413c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 1 --2- 172.21.15.98:0/2851219535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21e40a4180 0x7f21e413ee00 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e80 tx=0x7f21dc01c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dc36a9afa0c7ec3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 1 -- 172.21.15.98:0/2851219535 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21dc0090d0 con 0x7f21e40a4180 2024-09-11T02:54:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: dump: 2024-09-11T02:54:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 5 monclient: authenticate success, global_id 23438 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 1 -- 172.21.15.98:0/2851219535 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21dc0a5730 con 0x7f21e40a4180 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 4 set_mon_vals no callback set 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 1 -- 172.21.15.98:0/2851219535 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21dc009270 con 0x7f21e40a4180 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:23.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e159a640 10 monclient: dump: 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.961 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient: shutdown 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- 172.21.15.98:0/2851219535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21e40a4180 msgr2=0x7f21e413ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 --2- 172.21.15.98:0/2851219535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21e40a4180 0x7f21e413ee00 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009e80 tx=0x7f21dc01c9a0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- 172.21.15.98:0/2851219535 shutdown_connections 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 --2- 172.21.15.98:0/2851219535 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21e40a4180 0x7f21e413ee00 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 --2- 172.21.15.98:0/2851219535 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f21e40a1eb0 0x7f21e413c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- 172.21.15.98:0/2851219535 >> 172.21.15.98:0/2851219535 conn(0x7f21e40a4780 msgr2=0x7f21e40a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- 172.21.15.98:0/2851219535 shutdown_connections 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- 172.21.15.98:0/2851219535 wait complete. 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient: build_initial_monmap 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient: monmap: 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:23.963 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 librados: starting msgr at 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 librados: starting objecter 2024-09-11T02:54:23.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 Processor -- start 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- start start 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 librados: setting wanted keys 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 librados: calling monclient init 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient: init 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient: _add_conns ranks=[1,0,4,2,6,3,8,5,7] 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e40a1eb0 0x7f21e4126760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): picked mon.noname-f con 0x7f21e40a1eb0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21e40a3690 0x7f21e41cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): picked mon.noname-a con 0x7f21e40a3690 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f21e40a4180 0x7f21e41241a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): picked mon.noname-c con 0x7f21e40a4180 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:23.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21e4141ef0 con 0x7f21e40a3690 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f21e4141d70 con 0x7f21e40a1eb0 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f21e4142070 con 0x7f21e40a4180 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e8b63640 10 monclient(hunting): authenticate will time out at 1217.831775s 2024-09-11T02:54:23.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f21e40a4180 0x7f21e41241a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21e40a3690 0x7f21e41cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f21e40a4180 0x7f21e41241a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34208/0 (socket says 172.21.15.98:34208) 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21e40a3690 0x7f21e41cf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42770/0 (socket says 172.21.15.98:42770) 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2992054470 learned_addr learned my addr 172.21.15.98:0/2992054470 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request con 0x7f21e40a4180 auth_method 0 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e1d9b640 10 monclient(hunting): get_auth_request con 0x7f21e40a3690 auth_method 0 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.945+0000 7f21e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e40a1eb0 0x7f21e4126760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): get_auth_request con 0x7f21e40a1eb0 auth_method 0 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 10 monclient(hunting): handle_auth_done global_id 23431 payload 824 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e40a1eb0 msgr2=0x7f21e4126760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e40a1eb0 0x7f21e4126760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21e40a3690 msgr2=0x7f21e41cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21e40a3690 0x7f21e41cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 1 monclient: found mon.noname-c 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2992054470 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e4127c80 con 0x7f21e40a4180 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e1d9b640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21e40a3690 0x7f21e41cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e2d9d640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f21e40a4180 0x7f21e41241a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009040 tx=0x7f21dc019740 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 client.?.objecter ms_handle_connect 0x7f21e40a4180 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21dc0bc020 con 0x7f21e40a4180 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: dump: 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21dc004040 con 0x7f21e40a4180 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21dc0046e0 con 0x7f21e40a4180 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 5 monclient: authenticate success, global_id 23431 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 10 monclient: _renew_subs 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e413f0d0 con 0x7f21e40a4180 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e0d99640 4 set_mon_vals no callback set 2024-09-11T02:54:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: dump: 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 10 client.23431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 10 monclient: _renew_subs 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e4127f60 con 0x7f21e40a4180 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 1 librados: init done 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e41cfb30 0x7f21e402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f21e41c6f00 con 0x7f21e41cfb30 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f21dc004270 con 0x7f21e40a4180 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e41cfb30 0x7f21e402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient: get_auth_request con 0x7f21e41cfb30 auth_method 0 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f21c403df70 0x7f21c4040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(190..190 src has 1..190) ==== 6158+0+0 (secure 0 0 0) 0x7f21dc0b3040 con 0x7f21e40a4180 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 client.23431.objecter ms_dispatch 0x7f21e4000ba0 osd_map(190..190 src has 1..190) 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 3 client.23431.objecter handle_osd_map got epochs [190,190] > 0 2024-09-11T02:54:23.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 3 client.23431.objecter handle_osd_map decoding full epoch 190 2024-09-11T02:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 20 client.23431.objecter dump_active .. 0 homeless 2024-09-11T02:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e1d9b640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f21c403df70 0x7f21c4040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e1d9b640 10 monclient: get_auth_request con 0x7f21c403df70 auth_method 0 2024-09-11T02:54:23.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 10 monclient(hunting): handle_auth_done global_id 19222 payload 824 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 1 -- 172.21.15.98:0/2992054470 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21e4068b40 con 0x7f21e41cfb30 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 client.23431.objecter ms_handle_connect 0x7f21e41cfb30 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e259c640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e41cfb30 0x7f21e402d450 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000f70 tx=0x7f21d400fc00 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 client.23431.objecter resend_mon_ops 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21d400d700 con 0x7f21e41cfb30 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21e1d9b640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f21c403df70 0x7f21c4040430 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7f21e41258c0 tx=0x7f21d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: dump: 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.978 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 client.23431.objecter ms_handle_connect 0x7f21c403df70 2024-09-11T02:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f21d40050d0 con 0x7f21e41cfb30 2024-09-11T02:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e41cfb30 msgr2=0x7f21e402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e41cfb30 0x7f21e402d450 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000f70 tx=0x7f21d400fc00 comp rx=0 tx=0).stop 2024-09-11T02:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e41cfb30 msgr2=0x7f21e402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.949+0000 7f21d2ffd640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e41cfb30 0x7f21e402d450 secure :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000f70 tx=0x7f21d400fc00 comp rx=0 tx=0).stop 2024-09-11T02:54:23.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:23.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e402d990 0x7f21e402dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:23.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:23.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f21e413eb80 con 0x7f21e402d990 2024-09-11T02:54:23.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e402d990 0x7f21e402dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:23.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient: get_auth_request con 0x7f21e402d990 auth_method 0 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 10 monclient(hunting): handle_auth_done global_id 19231 payload 824 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 1 -- 172.21.15.98:0/2992054470 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f21e40a3a70 con 0x7f21e402d990 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e259c640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e402d990 0x7f21e402dda0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000f70 tx=0x7f21d4001130 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 10 client.23431.objecter ms_handle_connect 0x7f21e402d990 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 10 client.23431.objecter resend_mon_ops 2024-09-11T02:54:23.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21d400de40 con 0x7f21e402d990 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 10 monclient: dump: 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:23.984 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f21d400d9d0 con 0x7f21e402d990 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e402d990 msgr2=0x7f21e402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e402d990 0x7f21e402dda0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000f70 tx=0x7f21d4001130 comp rx=0 tx=0).stop 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e402d990 msgr2=0x7f21e402dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:23.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21d2ffd640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21e402d990 0x7f21e402dda0 secure :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f21d4000f70 tx=0x7f21d4001130 comp rx=0 tx=0).stop 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 279728, 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:23.986 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:23.987 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:23.988 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.989 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:23.990 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:23.991 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.993 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:23.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.995 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.997 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:23.999 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:24.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:24.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:24.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.005 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:24.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:24.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.011 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:24.012 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:24.013 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:24.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 10 librados: watch_flush enter 2024-09-11T02:54:24.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 10 librados: watch_flush exit 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 20 client.23431.objecter shutdown clearing up homeless session... 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 10 client.23431.objecter successfully canceled tick 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f21c403df70 msgr2=0x7f21c4040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f21c403df70 0x7f21c4040430 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7f21e41258c0 tx=0x7f21d8008040 comp rx=0 tx=0).stop 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 10 monclient: shutdown 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f21e40a4180 msgr2=0x7f21e41241a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e8b63640 1 --2- 172.21.15.98:0/2992054470 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f21e40a4180 0x7f21e41241a0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009040 tx=0x7f21dc019740 comp rx=0 tx=0).stop 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2992054470 reap_dead start 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.957+0000 7f21e2d9d640 1 -- 172.21.15.98:0/2992054470 reap_dead start 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.961+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 shutdown_connections 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.961+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 >> 172.21.15.98:0/2992054470 conn(0x7f21e40a4780 msgr2=0x7f21e411e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:24.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.961+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 shutdown_connections 2024-09-11T02:54:24.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.961+0000 7f21e8b63640 1 -- 172.21.15.98:0/2992054470 wait complete. 2024-09-11T02:54:24.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:23.961+0000 7f21e8b63640 1 librados: shutdown 2024-09-11T02:54:24.016 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:24.016 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T02:54:24.016 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T02:54:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient: get_monmap_and_config 2024-09-11T02:54:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient: build_initial_monmap 2024-09-11T02:54:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient: monmap: 2024-09-11T02:54:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:24.143482+0000 2024-09-11T02:54:24.142 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:24.143482+0000 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:24.143 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:24.144 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:24.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:24.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 1 Processor -- start 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 1 -- start start 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient: init 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient: _add_conns ranks=[6,1,0,4,9,7,8,3,5,2,10] 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12e806f650 0x7f12e8072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient(hunting): picked mon.noname-d con 0x7f12e806f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f12e80a8380 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12e8070140 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:24.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e8071a80 con 0x7f12e8070140 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12e8071d60 con 0x7f12e80a8380 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f12e80a8800 con 0x7f12e806f650 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ef62e640 10 monclient(hunting): authenticate will time out at 1218.023643s 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12edbcb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f12e8070140 0x7f12e8140c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56668/0 (socket says 172.21.15.98:56668) 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12e806f650 0x7f12e8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:24.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12edbcb640 1 -- 172.21.15.98:0/1449142045 learned_addr learned my addr 172.21.15.98:0/1449142045 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 10 monclient(hunting): get_auth_request con 0x7f12e806f650 auth_method 0 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.137+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:24.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_done global_id 14685 payload 274 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 1 -- 172.21.15.98:0/1449142045 >> v1:172.21.15.190:6789/0 conn(0x7f12e80a8380 legacy=0x7f12e806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 1 -- 172.21.15.98:0/1449142045 >> v1:172.21.15.98:6789/0 conn(0x7f12e8070140 legacy=0x7f12e8140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 10 monclient: _finish_hunting 0 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 1 monclient: found mon.noname-d 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 1 -- 172.21.15.98:0/1449142045 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e80714c0 con 0x7f12e806f650 2024-09-11T02:54:24.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 1 -- 172.21.15.98:0/1449142045 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 94578183 0 0) 0x7f12e8071a80 con 0x7f12e8070140 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ed3ca640 1 --2- 172.21.15.98:0/1449142045 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12e806f650 0x7f12e8072640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097f0 tx=0x7f12d801c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=9e76d6159de99273 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 1 -- 172.21.15.98:0/1449142045 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12d80b2070 con 0x7f12e806f650 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: dump: 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:24.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: _finish_auth 0 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 1 -- 172.21.15.98:0/1449142045 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d80ab7f0 con 0x7f12e806f650 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 5 monclient: authenticate success, global_id 14685 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 1 -- 172.21.15.98:0/1449142045 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12d80abe10 con 0x7f12e806f650 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 4 set_mon_vals no callback set 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12dffff640 10 monclient: dump: 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:24.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:24.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 10 monclient: shutdown 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 -- 172.21.15.98:0/1449142045 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12e806f650 msgr2=0x7f12e8072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 --2- 172.21.15.98:0/1449142045 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12e806f650 0x7f12e8072640 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097f0 tx=0x7f12d801c3a0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 -- 172.21.15.98:0/1449142045 shutdown_connections 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 --2- 172.21.15.98:0/1449142045 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f12e806f650 0x7f12e8072640 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 -- 172.21.15.98:0/1449142045 >> 172.21.15.98:0/1449142045 conn(0x7f12e80ac8e0 msgr2=0x7f12e80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 -- 172.21.15.98:0/1449142045 shutdown_connections 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 -- 172.21.15.98:0/1449142045 wait complete. 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 10 monclient: build_initial_monmap 2024-09-11T02:54:24.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 10 monclient: monmap: 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:24.161 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 librados: starting msgr at 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 librados: starting objecter 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 Processor -- start 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 -- start start 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 librados: setting wanted keys 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 1 librados: calling monclient init 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.141+0000 7f12ef62e640 10 monclient: init 2024-09-11T02:54:24.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _add_conns ranks=[2,3,8,5,0,6,1,4,7] 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12e806f650 0x7f12e8140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): picked mon.noname-b con 0x7f12e806f650 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12e8070140 0x7f12e81d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): picked mon.noname-g con 0x7f12e8070140 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12e80a8380 0x7f12e81268d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): picked mon.noname-e con 0x7f12e80a8380 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f12e80517e0 con 0x7f12e806f650 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f12e8144d70 con 0x7f12e8070140 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:24.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f12e8144ef0 con 0x7f12e80a8380 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient(hunting): authenticate will time out at 1218.028233s 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12e806f650 0x7f12e8140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12e806f650 0x7f12e8140f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50830/0 (socket says 172.21.15.98:50830) 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 -- 172.21.15.98:0/2138303485 learned_addr learned my addr 172.21.15.98:0/2138303485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): get_auth_request con 0x7f12e806f650 auth_method 0 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12e80a8380 0x7f12e81268d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 10 monclient(hunting): get_auth_request con 0x7f12e80a8380 auth_method 0 2024-09-11T02:54:24.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12e8070140 0x7f12e81d14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 10 monclient(hunting): get_auth_request con 0x7f12e8070140 auth_method 0 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:24.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient(hunting): handle_auth_done global_id 23447 payload 824 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 -- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12e80a8380 msgr2=0x7f12e81268d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12e80a8380 0x7f12e81268d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 -- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12e8070140 msgr2=0x7f12e81d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12e8070140 0x7f12e81d14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient: _finish_hunting 0 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 monclient: found mon.noname-b 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 -- 172.21.15.98:0/2138303485 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e812a560 con 0x7f12e806f650 2024-09-11T02:54:24.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12edbcb640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12e80a8380 0x7f12e81268d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ed3ca640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12e806f650 0x7f12e8140f10 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f12d80b0040 tx=0x7f12e80a7f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 client.?.objecter ms_handle_connect 0x7f12e806f650 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 1 -- 172.21.15.98:0/2138303485 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12d80ba020 con 0x7f12e806f650 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: dump: 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.167 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:24.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 5 monclient: authenticate success, global_id 23447 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _renew_subs 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 1 -- 172.21.15.98:0/2138303485 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12d80ab9a0 con 0x7f12e806f650 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e8073530 con 0x7f12e806f650 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 1 -- 172.21.15.98:0/2138303485 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12d80b2040 con 0x7f12e806f650 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12df7fe640 4 set_mon_vals no callback set 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:24.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 client.23447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 monclient: dump: 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:24.170 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _renew_subs 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e812a280 con 0x7f12e806f650 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 librados: init done 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e8072620 con 0x7f12e806f650 2024-09-11T02:54:24.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 1 -- 172.21.15.98:0/2138303485 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f12d80b5050 con 0x7f12e806f650 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f12cc03df40 0x7f12cc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 1 -- 172.21.15.98:0/2138303485 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(191..191 src has 1..191) ==== 6817+0+0 (secure 0 0 0) 0x7f12d80e9760 con 0x7f12e806f650 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 client.23447.objecter ms_dispatch 0x7f12e8000ba0 osd_map(191..191 src has 1..191) 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 3 client.23447.objecter handle_osd_map got epochs [191,191] > 0 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 3 client.23447.objecter handle_osd_map decoding full epoch 191 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 20 client.23447.objecter dump_active .. 0 homeless 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f12cc03df40 0x7f12cc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 10 monclient: get_auth_request con 0x7f12cc03df40 auth_method 0 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ddffb640 10 client.23447.objecter ms_handle_connect 0x7f12cc03df40 2024-09-11T02:54:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.145+0000 7f12ecbc9640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f12cc03df40 0x7f12cc040400 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f12e8129d90 tx=0x7f12e40094d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.149+0000 7f12ddffb640 1 -- 172.21.15.98:0/2138303485 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f12d80c4030 con 0x7f12e806f650 2024-09-11T02:54:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.149+0000 7f12ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.149+0000 7f12ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:24.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.385+0000 7f12ef62e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T02:54:24.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.385+0000 7f12ef62e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T02:54:24.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.385+0000 7f12ef62e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:24.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.385+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f12e8068f80 con 0x7f12e806f650 2024-09-11T02:54:24.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.389+0000 7f12ddffb640 1 -- 172.21.15.98:0/2138303485 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f12d80160b0 con 0x7f12e806f650 2024-09-11T02:54:24.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.389+0000 7f12ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T02:54:24.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.389+0000 7f12ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=10,kv=8,logm=70} crc {auth=3362045076,config=692619927,health=33302854,kv=3205263351,logm=842709800}) 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=9,logm=68} crc {auth=1520300750,config=692619927,health=1995790572,kv=4292987402,logm=908436748}) 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.397 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=9,config=2,health=10,kv=10,logm=69} crc {auth=1427350800,config=692619927,health=2484213879,kv=3991532845,logm=1280564589}) 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=10,config=1,health=12,kv=10,logm=67} crc {auth=3362045076,config=1228700967,health=2236382428,kv=3991532845,logm=4033650124}) 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=9,config=2,health=11,kv=10,logm=68} crc {auth=4212714589,config=692619927,health=249806387,kv=3991532845,logm=2604694980}) 2024-09-11T02:54:24.398 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=1427350800,config=692619927,health=1995790572,kv=3991532845,logm=3121319439}) 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=3362045076,config=692619927,health=249806387,kv=3991532845,logm=1485304081}) 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=3362045076,config=692619927,health=1995790572,kv=1009120483,logm=258700320}) 2024-09-11T02:54:24.399 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.392+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=3362045076,config=692619927,health=1410898352,kv=3991532845,logm=926213858}) 2024-09-11T02:54:24.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 10 librados: watch_flush enter 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 10 librados: watch_flush exit 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 20 client.23447.objecter shutdown clearing up homeless session... 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 10 client.23447.objecter successfully canceled tick 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f12cc03df40 msgr2=0x7f12cc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f12cc03df40 0x7f12cc040400 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f12e8129d90 tx=0x7f12e40094d0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 10 monclient: shutdown 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12e806f650 msgr2=0x7f12e8140f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:24.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12e806f650 0x7f12e8140f10 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f12d80b0040 tx=0x7f12e80a7f40 comp rx=0 tx=0).stop 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 shutdown_connections 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f12e80a8380 0x7f12e81268d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f12e8070140 0x7f12e81d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f12cc03df40 0x7f12cc040400 unknown :-1 s=CLOSED pgs=3453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 --2- 172.21.15.98:0/2138303485 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f12e806f650 0x7f12e8140f10 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 >> 172.21.15.98:0/2138303485 conn(0x7f12e80ac8e0 msgr2=0x7f12e80a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 shutdown_connections 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 -- 172.21.15.98:0/2138303485 wait complete. 2024-09-11T02:54:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:24.393+0000 7f12ef62e640 1 librados: shutdown 2024-09-11T02:54:24.409 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967037795}) 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011668036}) 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1683452246}) 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.410 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2682797647}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2473247263}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3732136434}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=345056511}) 2024-09-11T02:54:24.411 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2015773850}) 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.396+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=717624694}) 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4078432719}) 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020133955}) 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.412 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.400+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=10679355}) 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2676873546}) 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2899783015}) 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.413 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1412830479}) 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2371201504}) 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3321836714}) 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=58878637}) 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.414 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3336889433}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205601482}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173747402}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1692540076}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.415 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3585699903}) 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1088014169}) 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3360405219}) 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771528106}) 2024-09-11T02:54:24.416 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.404+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3359567634}) 2024-09-11T02:54:24.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T02:54:24.420 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.420 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.420 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825657643}) 2024-09-11T02:54:24.420 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.420 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.420 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2851856263}) 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2350187643}) 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3988315909}) 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=653151523}) 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.421 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3418203659}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3498740854}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=198739609}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.412+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=453738750}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508090185}) 2024-09-11T02:54:24.422 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2287957643}) 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093203576}) 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1121022821}) 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.423 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2273463154}) 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2139792890}) 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1040525658}) 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714287806}) 2024-09-11T02:54:24.424 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.416+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1565755152}) 2024-09-11T02:54:24.428 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.428 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.428 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744223894}) 2024-09-11T02:54:24.428 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.428 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.428 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476980974}) 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3132629311}) 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738317523}) 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2325464571}) 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.429 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.430 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1121092680}) 2024-09-11T02:54:24.430 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.430 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.430 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=338219545}) 2024-09-11T02:54:24.430 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.430 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410879363}) 2024-09-11T02:54:24.430 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.420+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3698311598}) 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2066393678}) 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972710159}) 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=938003359}) 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.431 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=765144383}) 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=980917561}) 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2620486798}) 2024-09-11T02:54:24.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2435107948}) 2024-09-11T02:54:24.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763740043}) 2024-09-11T02:54:24.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.424+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2418912808}) 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072380334}) 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=323716470}) 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2763714617}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2984199573}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2550040316}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2193296689}) 2024-09-11T02:54:24.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=796608253}) 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347611910}) 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.428+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=712056730}) 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181533333}) 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248964968}) 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3099381192}) 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2796509862}) 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4046787251}) 2024-09-11T02:54:24.443 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2749082926}) 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=295706604}) 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862476917}) 2024-09-11T02:54:24.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.432+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4272476826}) 2024-09-11T02:54:24.452 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.452 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.452 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801186890}) 2024-09-11T02:54:24.452 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.452 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.452 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373721360}) 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1853187328}) 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3191986537}) 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3233863654}) 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.453 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1981908098}) 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3907348667}) 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874276845}) 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.436+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2126856693}) 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.454 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217607667}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959988991}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4129812108}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3314712985}) 2024-09-11T02:54:24.455 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3333799084}) 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=508768585}) 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3475987858}) 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2641687495}) 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.440+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=651734925}) 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680159003}) 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840094467}) 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2878438607}) 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2181264167}) 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=33681732}) 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=441483745}) 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3274412501}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578278225}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.444+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=963149180}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=576490098}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2817287316}) 2024-09-11T02:54:24.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3724844402}) 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=538155769}) 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3274401274}) 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1844737859}) 2024-09-11T02:54:24.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3661880201}) 2024-09-11T02:54:24.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129686390}) 2024-09-11T02:54:24.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.448+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4045786857}) 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1525366303}) 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626830902}) 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3376723294}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=784171522}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1648490261}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2135563368}) 2024-09-11T02:54:24.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=484641479}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2070844200}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.452+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3192294334}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149913165}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2274881400}) 2024-09-11T02:54:24.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=705457887}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1589938698}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1076270481}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4079789873}) 2024-09-11T02:54:24.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=817832324}) 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422720361}) 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.456+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1568736338}) 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1813718870}) 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827802261}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941846430}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3872287850}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2424748398}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=238492984}) 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1963979714}) 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264632336}) 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1061378197}) 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.460+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545587891}) 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1480360125}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1475176131}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1184905576}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3210114651}) 2024-09-11T02:54:24.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=963393446}) 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3947860504}) 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582078333}) 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3297482665}) 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859663797}) 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2937981923}) 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=946795572}) 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2688355193}) 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1547565884}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2327332864}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3769996710}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190090987}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.464+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=938158938}) 2024-09-11T02:54:24.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138262181}) 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2602338080}) 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1357800643}) 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3478346389}) 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3115105081}) 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2556423917}) 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1113647247}) 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081360417}) 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2092584299}) 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1922712050}) 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696763643}) 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.472+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1860972090}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250263100}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2967593748}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=426270088}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2690914523}) 2024-09-11T02:54:24.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274576708}) 2024-09-11T02:54:24.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2928289960}) 2024-09-11T02:54:24.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050584431}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=728172262}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3103623890}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4110212821}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1758678621}) 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1167956997}) 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.476+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3050735056}) 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931072593}) 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3391103037}) 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725496926}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=530956774}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1590627172}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2167023160}) 2024-09-11T02:54:24.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3150256503}) 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2759295934}) 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=485934934}) 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005804170}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.480+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1177561007}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=404362151}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553842614}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3689759368}) 2024-09-11T02:54:24.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2768867229}) 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=264193767}) 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=328250233}) 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159600762}) 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401854599}) 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.484+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3776429437}) 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=7701767}) 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2857806468}) 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=10348971}) 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3832057247}) 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=548980576}) 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2429815948}) 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1892234255}) 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828417537}) 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.488+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3511743487}) 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429669373}) 2024-09-11T02:54:24.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973473436}) 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1432465201}) 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=570734088}) 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2952870296}) 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1499473112}) 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2063662825}) 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292045604}) 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.492+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2964645798}) 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928183959}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705618489}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1283015722}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2736476820}) 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4238859740}) 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3216460888}) 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1892457973}) 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617696838}) 2024-09-11T02:54:24.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.496+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3232368093}) 2024-09-11T02:54:24.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071238052}) 2024-09-11T02:54:24.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481970828}) 2024-09-11T02:54:24.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2047737173}) 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3142140437}) 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2564045945}) 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4281242960}) 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3319376109}) 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899624663}) 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2655409071}) 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812429841}) 2024-09-11T02:54:24.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746909635}) 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=303307714}) 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3528946111}) 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1699342946}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=692716920}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1150977200}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705192691}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.500+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1207605944}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.537 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=319093418,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3817492598,mon_config_key=3492741104,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=833433874}) 2024-09-11T02:54:24.538 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.538 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.538 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=50,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=2730261384,mdsmap=3491474967,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=58979658,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=2374085391}) 2024-09-11T02:54:24.538 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.538 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.538 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=52,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=3} crc {logm=34091634,mdsmap=92030302,mgr=2802927022,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3394831202,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=2416972009}) 2024-09-11T02:54:24.538 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11} crc {logm=832697564,mdsmap=3491474967,mgr=4015380932,mgr_command_descs=2308730519,mgrstat=1160278827,mon_config_key=3163979437,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=412942451}) 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=3650969081,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=58979658,mon_config_key=1744168505,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=2859559657}) 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=3} crc {logm=630151865,mdsmap=3491474967,mgr=492734241,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=58979658,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2718501277,osd_snap=2416972009}) 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.539 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.540 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=47,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=151890974,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=58979658,mon_config_key=3200485260,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.540 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.540 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=49,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3843220343,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2956299639,mon_config_key=1408456615,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2718501277,osd_snap=1292118936}) 2024-09-11T02:54:24.540 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.504+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=2,osd_metadata=6,osd_snap=8} crc {logm=2373842091,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=421550639,mon_config_key=3782314913,monmap=2495409540,nvmeofgw=3140514009,osd_metadata=1477139452,osd_snap=4269828464}) 2024-09-11T02:54:24.540 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.540 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.540 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=2085011138,osdmap=3420251306}) 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=2085011138,osdmap=430000199}) 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=3549568526,osdmap=468546704}) 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=2548342122,osdmap=3422164976}) 2024-09-11T02:54:24.541 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.542 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.542 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=3856414640,osdmap=1881418271}) 2024-09-11T02:54:24.542 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.542 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.542 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=719896658,osdmap=2370393600}) 2024-09-11T02:54:24.542 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.542 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=2826428813,osdmap=48255775}) 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=137671534,osdmap=4244216429}) 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.508+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=2826428813,osdmap=2464014229}) 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1580690379}) 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.543 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=366626030}) 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=439487633}) 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1862494825}) 2024-09-11T02:54:24.544 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=339230826}) 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.512+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=936750278}) 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.545 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=643157122}) 2024-09-11T02:54:24.546 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.546 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2546420901}) 2024-09-11T02:54:24.546 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.516+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2571231086}) 2024-09-11T02:54:24.549 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.549 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=514055522}) 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1782002287}) 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1364167164}) 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.550 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=80354904}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=960477202}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3801853462}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2814723336}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=207940230}) 2024-09-11T02:54:24.551 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.520+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3696920169}) 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=11} crc {osdmap=2959144536}) 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=11} crc {osdmap=2959144536}) 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=10} crc {osdmap=2142478996}) 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.552 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=11} crc {osdmap=2959144536}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=11} crc {osdmap=2959144536}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=11} crc {osdmap=2959144536}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=11} crc {osdmap=2959144536}) 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:54:24.553 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=10} crc {osdmap=2829486585}) 2024-09-11T02:54:24.554 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:54:24.536+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=11} crc {osdmap=2959144536}) 2024-09-11T02:54:25.419 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:54:25.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:54:25.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient: get_monmap_and_config 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient: build_initial_monmap 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient: monmap: 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:25.551071+0000 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:25.551071+0000 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:25.550 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:25.551 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 1 Processor -- start 2024-09-11T02:54:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 1 -- start start 2024-09-11T02:54:25.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient: init 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient: _add_conns ranks=[2,8,1,3,9,7,10,5,0,4,6] 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f894806f650 0x7f8948072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient(hunting): picked mon.noname-b con 0x7f894806f650 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894806fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient(hunting): picked mon.noname-e con 0x7f89480a8380 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f894806ff70 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8948071a80 con 0x7f894806ff70 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f89480a8800 con 0x7f894806f650 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8948071d60 con 0x7f89480a8380 2024-09-11T02:54:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f894e9c3640 10 monclient(hunting): authenticate will time out at 1219.431548s 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894806fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f894806f650 0x7f8948072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f894806f650 0x7f8948072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50846/0 (socket says 172.21.15.98:50846) 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 1 -- 172.21.15.98:0/269515814 learned_addr learned my addr 172.21.15.98:0/269515814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894806fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:50812/0 (socket says 172.21.15.98:50812) 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): get_auth_request con 0x7f894806f650 auth_method 0 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 10 monclient(hunting): get_auth_request con 0x7f89480a8380 auth_method 0 2024-09-11T02:54:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f89477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 10 monclient(hunting): handle_auth_done global_id 23456 payload 274 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.545+0000 7f8947fff640 1 -- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 msgr2=0x7f894806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 1 --2- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894806fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 1 -- 172.21.15.98:0/269515814 >> v1:172.21.15.190:6789/0 conn(0x7f894806ff70 legacy=0x7f89481397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:25.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 1 monclient: found mon.noname-b 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 1 -- 172.21.15.98:0/269515814 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89480714c0 con 0x7f894806f650 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f89477fe640 1 --2- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894806fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8947fff640 1 --2- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f894806f650 0x7f8948072640 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8938009eb0 tx=0x7f893801d3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6766969b62e2cef server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 1 -- 172.21.15.98:0/269515814 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89380090d0 con 0x7f894806f650 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: dump: 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.559 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.560 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 1 -- 172.21.15.98:0/269515814 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89380a5980 con 0x7f894806f650 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 5 monclient: authenticate success, global_id 23456 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 1 -- 172.21.15.98:0/269515814 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89380ad3f0 con 0x7f894806f650 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 4 set_mon_vals no callback set 2024-09-11T02:54:25.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f8946ffd640 10 monclient: dump: 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.562 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 10 monclient: shutdown 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 -- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f894806f650 msgr2=0x7f8948072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 --2- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f894806f650 0x7f8948072640 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8938009eb0 tx=0x7f893801d3e0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 -- 172.21.15.98:0/269515814 shutdown_connections 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 --2- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894806fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 --2- 172.21.15.98:0/269515814 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f894806f650 0x7f8948072640 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 -- 172.21.15.98:0/269515814 >> 172.21.15.98:0/269515814 conn(0x7f89480ac8e0 msgr2=0x7f89480a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 -- 172.21.15.98:0/269515814 shutdown_connections 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 -- 172.21.15.98:0/269515814 wait complete. 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 10 monclient: build_initial_monmap 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 10 monclient: monmap: 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:25.564 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 librados: starting msgr at 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 librados: starting objecter 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.549+0000 7f894e9c3640 1 Processor -- start 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 -- start start 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 librados: setting wanted keys 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 librados: calling monclient init 2024-09-11T02:54:25.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: init 2024-09-11T02:54:25.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _add_conns ranks=[0,4,8,2,1,5,6,7,3] 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f894806f650 0x7f89481d1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): picked mon.noname-a con 0x7f894806f650 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f894806ff70 0x7f8948139110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): picked mon.noname-c con 0x7f894806ff70 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894813e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): picked mon.noname-e con 0x7f89480a8380 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8948143fd0 con 0x7f894806f650 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8948143e50 con 0x7f894806ff70 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8948144150 con 0x7f89480a8380 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient(hunting): authenticate will time out at 1219.436500s 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f894806f650 0x7f89481d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894813e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f894806ff70 0x7f8948139110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894813e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:50818/0 (socket says 172.21.15.98:50818) 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f894806f650 0x7f89481d1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42786/0 (socket says 172.21.15.98:42786) 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 1 -- 172.21.15.98:0/4070881971 learned_addr learned my addr 172.21.15.98:0/4070881971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient(hunting): get_auth_request con 0x7f894806ff70 auth_method 0 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): get_auth_request con 0x7f894806f650 auth_method 0 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 10 monclient(hunting): get_auth_request con 0x7f89480a8380 auth_method 0 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894cf60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient(hunting): handle_auth_done global_id 34254 payload 824 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 -- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 msgr2=0x7f894813e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894813e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 -- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f894806ff70 msgr2=0x7f8948139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f894806ff70 0x7f8948139110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 monclient: found mon.noname-a 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:25.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 -- 172.21.15.98:0/4070881971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89480a51d0 con 0x7f894806f650 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f894806ff70 0x7f8948139110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8947fff640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f894806f650 0x7f89481d1370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89380ab000 tx=0x7f8938019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 client.?.objecter ms_handle_connect 0x7f894806f650 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 1 -- 172.21.15.98:0/4070881971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89380bb020 con 0x7f894806f650 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: dump: 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.574 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.575 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: _finish_auth 0 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 1 -- 172.21.15.98:0/4070881971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8938009250 con 0x7f894806f650 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 5 monclient: authenticate success, global_id 34254 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _renew_subs 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89481237e0 con 0x7f894806f650 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 1 -- 172.21.15.98:0/4070881971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8938003e10 con 0x7f894806f650 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:25.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 monclient: dump: 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89467fc640 4 set_mon_vals no callback set 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 client.34254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _renew_subs 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89480a56b0 con 0x7f894806f650 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 librados: init done 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 1 -- 172.21.15.98:0/4070881971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f89380039a0 con 0x7f894806f650 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8948072620 con 0x7f894806f650 2024-09-11T02:54:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f892c03df40 0x7f892c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 1 -- 172.21.15.98:0/4070881971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 7028+0+0 (secure 0 0 0) 0x7f89380e9d60 con 0x7f894806f650 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 client.34254.objecter ms_dispatch 0x7f8948000ba0 osd_map(192..192 src has 1..192) 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 3 client.34254.objecter handle_osd_map got epochs [192,192] > 0 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 3 client.34254.objecter handle_osd_map decoding full epoch 192 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 20 client.34254.objecter dump_active .. 0 homeless 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f892c03df40 0x7f892c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 10 monclient: get_auth_request con 0x7f892c03df40 auth_method 0 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f8944ff9640 10 client.34254.objecter ms_handle_connect 0x7f892c03df40 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.553+0000 7f89477fe640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f892c03df40 0x7f892c040400 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f894813fb60 tx=0x7f8930009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.557+0000 7f8944ff9640 1 -- 172.21.15.98:0/4070881971 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f89480a51d0 con 0x7f894806f650 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.557+0000 7f8944ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:25.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.557+0000 7f8944ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:25.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.797+0000 7f894e9c3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:54:25.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.797+0000 7f894e9c3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:25.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.797+0000 7f894e9c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:54:25.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.797+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8948068f80 con 0x7f894806f650 2024-09-11T02:54:25.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.797+0000 7f8944ff9640 1 -- 172.21.15.98:0/4070881971 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f8938016600 con 0x7f894806f650 2024-09-11T02:54:25.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.797+0000 7f8944ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:25.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.797+0000 7f8944ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:25.802 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":42,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:54:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 10 librados: watch_flush enter 2024-09-11T02:54:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 10 librados: watch_flush exit 2024-09-11T02:54:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 20 client.34254.objecter shutdown clearing up homeless session... 2024-09-11T02:54:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 10 client.34254.objecter successfully canceled tick 2024-09-11T02:54:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f892c03df40 msgr2=0x7f892c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f892c03df40 0x7f892c040400 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f894813fb60 tx=0x7f8930009190 comp rx=0 tx=0).stop 2024-09-11T02:54:25.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 10 monclient: shutdown 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f894806f650 msgr2=0x7f89481d1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f894806f650 0x7f89481d1370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89380ab000 tx=0x7f8938019720 comp rx=0 tx=0).stop 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 shutdown_connections 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f892c03df40 0x7f892c040400 unknown :-1 s=CLOSED pgs=3454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f89480a8380 0x7f894813e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f894806ff70 0x7f8948139110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 --2- 172.21.15.98:0/4070881971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f894806f650 0x7f89481d1370 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 >> 172.21.15.98:0/4070881971 conn(0x7f89480ac8e0 msgr2=0x7f89480a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 shutdown_connections 2024-09-11T02:54:25.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 -- 172.21.15.98:0/4070881971 wait complete. 2024-09-11T02:54:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.801+0000 7f894e9c3640 1 librados: shutdown 2024-09-11T02:54:25.822 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:54:25.822 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T02:54:25.822 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:54:25.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:54:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.929+0000 7f9cd5654640 10 monclient: get_monmap_and_config 2024-09-11T02:54:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.929+0000 7f9cd5654640 10 monclient: build_initial_monmap 2024-09-11T02:54:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.929+0000 7f9cd5654640 10 monclient: monmap: 2024-09-11T02:54:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:25.936499+0000 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:25.936499+0000 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:25.937 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:25.937 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 1 Processor -- start 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 1 -- start start 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient: init 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient: _add_conns ranks=[2,1,5,8,4,0,3,10,6,9,7] 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cd00a46a0 0x7f9cd00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd00a46a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9cd00a13a0 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a4fc0 0x7f9cd0136b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient(hunting): picked mon.noname-h con 0x7f9cd00a4fc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd00a11a0 con 0x7f9cd00a13a0 2024-09-11T02:54:25.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd00a28a0 con 0x7f9cd00a46a0 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9cd00a25c0 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cd5654640 10 monclient(hunting): authenticate will time out at 1219.817614s 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cd00a46a0 0x7f9cd00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cd00a46a0 0x7f9cd00a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50856/0 (socket says 172.21.15.98:50856) 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 -- 172.21.15.98:0/772456634 learned_addr learned my addr 172.21.15.98:0/772456634 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): get_auth_request con 0x7f9cd00a46a0 auth_method 0 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient(hunting): handle_auth_done global_id 23465 payload 274 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9ccf7fe640 1 --2- 172.21.15.98:0/772456634 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a4fc0 0x7f9cd0136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 -- 172.21.15.98:0/772456634 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a4fc0 msgr2=0x7f9cd0136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 --2- 172.21.15.98:0/772456634 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a4fc0 0x7f9cd0136b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 -- 172.21.15.98:0/772456634 >> v1:172.21.15.190:6789/0 conn(0x7f9cd00a13a0 legacy=0x7f9cd0136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient: _finish_hunting 0 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 monclient: found mon.noname-b 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.933+0000 7f9cceffd640 1 -- 172.21.15.98:0/772456634 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd00a5440 con 0x7f9cd00a46a0 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cceffd640 1 --2- 172.21.15.98:0/772456634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cd00a46a0 0x7f9cd00a4a80 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00098a0 tx=0x7f9cc001c260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92dcee4081ca4255 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 1 -- 172.21.15.98:0/772456634 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cc00b0070 con 0x7f9cd00a46a0 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: dump: 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 1 -- 172.21.15.98:0/772456634 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc00aa810 con 0x7f9cd00a46a0 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 5 monclient: authenticate success, global_id 23465 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 4 set_mon_vals no callback set 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 1 -- 172.21.15.98:0/772456634 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cc00a9300 con 0x7f9cd00a46a0 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccdffb640 10 monclient: dump: 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient: shutdown 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- 172.21.15.98:0/772456634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cd00a46a0 msgr2=0x7f9cd00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 --2- 172.21.15.98:0/772456634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cd00a46a0 0x7f9cd00a4a80 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00098a0 tx=0x7f9cc001c260 comp rx=0 tx=0).stop 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- 172.21.15.98:0/772456634 shutdown_connections 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 --2- 172.21.15.98:0/772456634 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a4fc0 0x7f9cd0136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 --2- 172.21.15.98:0/772456634 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9cd00a46a0 0x7f9cd00a4a80 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- 172.21.15.98:0/772456634 >> 172.21.15.98:0/772456634 conn(0x7f9cd006ed00 msgr2=0x7f9cd006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- 172.21.15.98:0/772456634 shutdown_connections 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- 172.21.15.98:0/772456634 wait complete. 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient: build_initial_monmap 2024-09-11T02:54:25.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient: monmap: 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:25.954 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 librados: starting msgr at 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 librados: starting objecter 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 Processor -- start 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- start start 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 librados: setting wanted keys 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 librados: calling monclient init 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient: init 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient: _add_conns ranks=[5,1,8,6,7,4,0,3,2] 2024-09-11T02:54:25.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a13a0 0x7f9cd01cf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): picked mon.noname-h con 0x7f9cd00a13a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cd00a46a0 0x7f9cd01cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): picked mon.noname-f con 0x7f9cd00a46a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9cd00a4fc0 0x7f9cd013c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): picked mon.noname-e con 0x7f9cd00a4fc0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd0141150 con 0x7f9cd00a46a0 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9cd01372b0 con 0x7f9cd00a13a0 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9cd01412d0 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:25.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cd5654640 10 monclient(hunting): authenticate will time out at 1219.823077s 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9cd00a4fc0 0x7f9cd013c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9cd00a4fc0 0x7f9cd013c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:50824/0 (socket says 172.21.15.98:50824) 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 -- 172.21.15.98:0/2655978459 learned_addr learned my addr 172.21.15.98:0/2655978459 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): get_auth_request con 0x7f9cd00a4fc0 auth_method 0 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cceffd640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a13a0 0x7f9cd01cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cce7fc640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cd00a46a0 0x7f9cd01cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cceffd640 10 monclient(hunting): get_auth_request con 0x7f9cd00a13a0 auth_method 0 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cce7fc640 10 monclient(hunting): get_auth_request con 0x7f9cd00a46a0 auth_method 0 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient(hunting): handle_auth_done global_id 23948 payload 824 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a13a0 msgr2=0x7f9cd01cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a13a0 0x7f9cd01cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cd00a46a0 msgr2=0x7f9cd01cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cd00a46a0 0x7f9cd01cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 monclient: found mon.noname-e 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 -- 172.21.15.98:0/2655978459 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd011f370 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cceffd640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9cd00a13a0 0x7f9cd01cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9ccf7fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9cd00a4fc0 0x7f9cd013c250 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800cc30 tx=0x7f9cc800a5c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cb77fe640 10 client.?.objecter ms_handle_connect 0x7f9cd00a4fc0 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.937+0000 7f9cb77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cc800ce40 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: dump: 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.960 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: _finish_auth 0 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: _check_auth_tickets 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 5 monclient: authenticate success, global_id 23948 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc800f860 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cc80173b0 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9ccd7fa640 4 set_mon_vals no callback set 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: dump: 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 10 monclient: _renew_subs 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd00a4150 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 10 client.23948.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 10 monclient: _renew_subs 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd01216d0 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 1 librados: init done 2024-09-11T02:54:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd011fed0 0x7f9cd013c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd006ca10 con 0x7f9cd011fed0 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd011fed0 0x7f9cd013c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient: get_auth_request con 0x7f9cd011fed0 auth_method 0 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9cc800f3f0 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 10 monclient(hunting): handle_auth_done global_id 34263 payload 824 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 1 -- 172.21.15.98:0/2655978459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cd0120370 con 0x7f9cd011fed0 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9cbc03df40 0x7f9cbc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(192..192 src has 1..192) ==== 7028+0+0 (secure 0 0 0) 0x7f9cc8009030 con 0x7f9cd00a4fc0 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 client.23948.objecter ms_dispatch 0x7f9cd0000ba0 osd_map(192..192 src has 1..192) 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 3 client.23948.objecter handle_osd_map got epochs [192,192] > 0 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 3 client.23948.objecter handle_osd_map decoding full epoch 192 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cceffd640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd011fed0 0x7f9cd013c790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00700a0 tx=0x7f9cc0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 20 client.23948.objecter dump_active .. 0 homeless 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 client.23948.objecter ms_handle_connect 0x7f9cd011fed0 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 client.23948.objecter resend_mon_ops 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cc00b9030 con 0x7f9cd011fed0 2024-09-11T02:54:25.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: dump: 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.966 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cce7fc640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9cbc03df40 0x7f9cbc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cce7fc640 10 monclient: get_auth_request con 0x7f9cbc03df40 auth_method 0 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9cc00b1040 con 0x7f9cd011fed0 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd011fed0 msgr2=0x7f9cd013c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd011fed0 0x7f9cd013c790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00700a0 tx=0x7f9cc0019000 comp rx=0 tx=0).stop 2024-09-11T02:54:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd011fed0 msgr2=0x7f9cd013c790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd011fed0 0x7f9cd013c790 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00700a0 tx=0x7f9cc0019000 comp rx=0 tx=0).stop 2024-09-11T02:54:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cb77fe640 10 client.23948.objecter ms_handle_connect 0x7f9cbc03df40 2024-09-11T02:54:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.941+0000 7f9cce7fc640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9cbc03df40 0x7f9cbc040400 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f9cd013d930 tx=0x7f9cc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd013ccd0 0x7f9cd013d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd006e870 con 0x7f9cd013ccd0 2024-09-11T02:54:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd013ccd0 0x7f9cd013d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient: get_auth_request con 0x7f9cd013ccd0 auth_method 0 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 10 monclient(hunting): handle_auth_done global_id 34272 payload 824 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 1 -- 172.21.15.98:0/2655978459 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9cd00a4a80 con 0x7f9cd013ccd0 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cceffd640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd013ccd0 0x7f9cd013d0b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00097c0 tx=0x7f9cc00a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 10 client.23948.objecter ms_handle_connect 0x7f9cd013ccd0 2024-09-11T02:54:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 10 client.23948.objecter resend_mon_ops 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9cc00b9040 con 0x7f9cd013ccd0 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 10 monclient: dump: 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:25.972 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f9cc00be460 con 0x7f9cd013ccd0 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd013ccd0 msgr2=0x7f9cd013d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd013ccd0 0x7f9cd013d0b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00097c0 tx=0x7f9cc00a8000 comp rx=0 tx=0).stop 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd013ccd0 msgr2=0x7f9cd013d0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cb77fe640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9cd013ccd0 0x7f9cd013d0b0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00097c0 tx=0x7f9cc00a8000 comp rx=0 tx=0).stop 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 18947, 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:25.975 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:25.976 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:25.977 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:25.978 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:25.979 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:25.981 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:25.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.983 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.985 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.990 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:25.992 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:25.995 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:25.996 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:25.997 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:25.997 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:25.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:25.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:25.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:25.997 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:25.997 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:25.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 10 librados: watch_flush enter 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 10 librados: watch_flush exit 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 20 client.23948.objecter shutdown clearing up homeless session... 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 10 client.23948.objecter successfully canceled tick 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9cbc03df40 msgr2=0x7f9cbc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9cbc03df40 0x7f9cbc040400 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f9cd013d930 tx=0x7f9cc4009040 comp rx=0 tx=0).stop 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 10 monclient: shutdown 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9cd00a4fc0 msgr2=0x7f9cd013c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 --2- 172.21.15.98:0/2655978459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9cd00a4fc0 0x7f9cd013c250 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800cc30 tx=0x7f9cc800a5c0 comp rx=0 tx=0).stop 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9ccf7fe640 1 -- 172.21.15.98:0/2655978459 reap_dead start 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9ccf7fe640 1 -- 172.21.15.98:0/2655978459 reap_dead start 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 shutdown_connections 2024-09-11T02:54:25.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 >> 172.21.15.98:0/2655978459 conn(0x7f9cd006ed00 msgr2=0x7f9cd013e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 shutdown_connections 2024-09-11T02:54:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 -- 172.21.15.98:0/2655978459 wait complete. 2024-09-11T02:54:26.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:25.949+0000 7f9cd5654640 1 librados: shutdown 2024-09-11T02:54:26.000 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:26.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:54:26.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.125+0000 7f3a4c8b9640 10 monclient: get_monmap_and_config 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.125+0000 7f3a4c8b9640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.125+0000 7f3a4c8b9640 10 monclient: monmap: 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:26.132393+0000 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:26.132393+0000 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:26.132 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 1 Processor -- start 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 1 -- start start 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient: init 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient: _add_conns ranks=[0,7,10,6,9,4,2,8,3,1,5] 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a480a7aa0 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 0x7f3a480a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient(hunting): picked mon.noname-i con 0x7f3a480a12a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a4813ea90 0x7f3a48140e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient(hunting): picked mon.noname-f con 0x7f3a4813ea90 addr v2:172.21.15.190:3300/0 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a480a1720 con 0x7f3a480a7aa0 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3a480a2b30 con 0x7f3a4813ea90 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3a480a27c0 con 0x7f3a480a12a0 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 10 monclient(hunting): authenticate will time out at 1220.013206s 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3a480a7aa0 0x7f3a480a7e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56674/0 (socket says 172.21.15.98:56674) 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4659c640 1 -- 172.21.15.98:0/3582119516 learned_addr learned my addr 172.21.15.98:0/3582119516 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 1 --2- 172.21.15.98:0/3582119516 >> v2:172.21.15.190:3300/0 conn(0x7f3a4813ea90 0x7f3a48140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 0x7f3a480a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): get_auth_request con 0x7f3a4813ea90 auth_method 0 2024-09-11T02:54:26.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): get_auth_request con 0x7f3a480a12a0 auth_method 0 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3239829908 0 0) 0x7f3a480a1720 con 0x7f3a480a7aa0 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): _negotiate 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): my global_id is 34281 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a30003530 con 0x7f3a480a7aa0 2024-09-11T02:54:26.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1670311708 0 0) 0x7f3a30003530 con 0x7f3a480a7aa0 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 msgr2=0x7f3a480a8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 0x7f3a480a8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 >> v2:172.21.15.190:3300/0 conn(0x7f3a4813ea90 msgr2=0x7f3a48140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 --2- 172.21.15.98:0/3582119516 >> v2:172.21.15.190:3300/0 conn(0x7f3a4813ea90 0x7f3a48140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:54:26.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a480a9590 con 0x7f3a480a7aa0 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f3a38001970 con 0x7f3a480a7aa0 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 5 monclient: authenticate success, global_id 34281 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: dump: 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.147 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.148 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 >> v1:172.21.15.98:6789/0 conn(0x7f3a480a7aa0 legacy=0x7f3a480a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient: _add_conns ranks=[7,4,1,3,0,8,2,6,5] 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 0x7f3a30004210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): picked mon.i con 0x7f3a480a12a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a30004860 0x7f3a30004ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): picked mon.c con 0x7f3a30004860 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a300053f0 0x7f3a3000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): picked mon.f con 0x7f3a300053f0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3a480a2b30 con 0x7f3a300053f0 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3a480a27c0 con 0x7f3a30004860 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3a3000c340 con 0x7f3a480a12a0 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4559a640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4c8b9640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a30004860 0x7f3a30004ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): get_auth_request con 0x7f3a30004860 auth_method 0 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4659c640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 0x7f3a30004210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4659c640 10 monclient(hunting): get_auth_request con 0x7f3a480a12a0 auth_method 0 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a4659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a300053f0 0x7f3a3000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): get_auth_request con 0x7f3a300053f0 auth_method 0 2024-09-11T02:54:26.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a46d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_done global_id 34281 payload 293 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 -- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 msgr2=0x7f3a30004210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a480a12a0 0x7f3a30004210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 -- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a300053f0 msgr2=0x7f3a3000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a300053f0 0x7f3a3000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 monclient: found mon.c 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:26.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 1 -- 172.21.15.98:0/3582119516 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a30003530 con 0x7f3a30004860 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.129+0000 7f3a45d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a30004860 0x7f3a30004ce0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009ae0 tx=0x7f3a3c01ca50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1e3f43889dd90148 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a3c0096b0 con 0x7f3a30004860 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 10 monclient: dump: 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.153 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a3c0a68b0 con 0x7f3a30004860 2024-09-11T02:54:26.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 4 set_mon_vals no callback set 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 1 -- 172.21.15.98:0/3582119516 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a3c0a7420 con 0x7f3a30004860 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4559a640 10 monclient: dump: 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.155 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: shutdown 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3582119516 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a30004860 msgr2=0x7f3a30004ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 --2- 172.21.15.98:0/3582119516 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a30004860 0x7f3a30004ce0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009ae0 tx=0x7f3a3c01ca50 comp rx=0 tx=0).stop 2024-09-11T02:54:26.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a46d9d640 1 -- 172.21.15.98:0/3582119516 reap_dead start 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3582119516 shutdown_connections 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3582119516 >> 172.21.15.98:0/3582119516 conn(0x7f3a480ac690 msgr2=0x7f3a480a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3582119516 shutdown_connections 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3582119516 wait complete. 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: monmap: 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:26.157 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 librados: starting msgr at 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 librados: starting objecter 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 Processor -- start 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- start start 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 librados: setting wanted keys 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 librados: calling monclient init 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: init 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: _add_conns ranks=[7,6,5,2,3,8,4,1,0] 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a30004860 0x7f3a48072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): picked mon.noname-i con 0x7f3a30004860 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a48123dd0 0x7f3a48138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): picked mon.noname-d con 0x7f3a48123dd0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:26.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a48139160 0x7f3a481226e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): picked mon.noname-h con 0x7f3a48139160 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3a48141de0 con 0x7f3a48139160 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3a480517e0 con 0x7f3a48123dd0 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3a48141f60 con 0x7f3a30004860 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient(hunting): authenticate will time out at 1220.019118s 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a48123dd0 0x7f3a48138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a48123dd0 0x7f3a48138c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:51170/0 (socket says 172.21.15.98:51170) 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 -- 172.21.15.98:0/3638302554 learned_addr learned my addr 172.21.15.98:0/3638302554 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:26.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): get_auth_request con 0x7f3a48123dd0 auth_method 0 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4659c640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a30004860 0x7f3a48072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient(hunting): handle_auth_done global_id 14694 payload 824 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a46d9d640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a48139160 0x7f3a481226e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a30004860 msgr2=0x7f3a48072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3a30004860 0x7f3a48072960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a48139160 msgr2=0x7f3a481226e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a48139160 0x7f3a481226e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 monclient: found mon.noname-d 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 -- 172.21.15.98:0/3638302554 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a48073350 con 0x7f3a48123dd0 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a45d9b640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a48123dd0 0x7f3a48138c20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009a00 tx=0x7f3a30005840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 client.?.objecter ms_handle_connect 0x7f3a48123dd0 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a3c0a6600 con 0x7f3a48123dd0 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: dump: 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.162 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 5 monclient: authenticate success, global_id 14694 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a3c004040 con 0x7f3a48123dd0 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: _renew_subs 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a3c003620 con 0x7f3a48123dd0 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a481252f0 con 0x7f3a48123dd0 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a44d99640 4 set_mon_vals no callback set 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.133+0000 7f3a36ffd640 10 monclient: dump: 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.165 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 10 client.14694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 10 monclient: _renew_subs 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a48122c20 con 0x7f3a48123dd0 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 1 librados: init done 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802d450 0x7f3a4802d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a480517e0 con 0x7f3a4802d450 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3a3c0c1030 con 0x7f3a48123dd0 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802d450 0x7f3a4802d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient: get_auth_request con 0x7f3a4802d450 auth_method 0 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a2803df40 0x7f3a28040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 10 monclient(hunting): handle_auth_done global_id 23474 payload 824 2024-09-11T02:54:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(193..193 src has 1..193) ==== 7848+0+0 (secure 0 0 0) 0x7f3a3c00f040 con 0x7f3a48123dd0 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 1 -- 172.21.15.98:0/3638302554 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a480a5440 con 0x7f3a4802d450 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 client.14694.objecter ms_dispatch 0x7f3a48000ba0 osd_map(193..193 src has 1..193) 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 3 client.14694.objecter handle_osd_map got epochs [193,193] > 0 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 3 client.14694.objecter handle_osd_map decoding full epoch 193 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 20 client.14694.objecter dump_active .. 0 homeless 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 client.14694.objecter ms_handle_connect 0x7f3a4802d450 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 client.14694.objecter resend_mon_ops 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a4659c640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802d450 0x7f3a4802d830 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f3a38001ec0 tx=0x7f3a3800fba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a38010040 con 0x7f3a4802d450 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 monclient: dump: 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.168 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.169 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a38019070 con 0x7f3a4802d450 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802d450 msgr2=0x7f3a4802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802d450 0x7f3a4802d830 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f3a38001ec0 tx=0x7f3a3800fba0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802d450 msgr2=0x7f3a4802d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802d450 0x7f3a4802d830 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a46d9d640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a2803df40 0x7f3a28040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a46d9d640 10 monclient: get_auth_request con 0x7f3a2803df40 auth_method 0 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a36ffd640 10 client.14694.objecter ms_handle_connect 0x7f3a2803df40 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.137+0000 7f3a46d9d640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a2803df40 0x7f3a28040400 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f3a480a2cb0 tx=0x7f3a4000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4c8b9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4c8b9640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802dd70 0x7f3a4802e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4c8b9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a481d2200 con 0x7f3a4802dd70 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802dd70 0x7f3a4802e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient: get_auth_request con 0x7f3a4802dd70 auth_method 0 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 10 monclient(hunting): handle_auth_done global_id 23483 payload 824 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 1 -- 172.21.15.98:0/3638302554 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a480a7e80 con 0x7f3a4802dd70 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a36ffd640 10 client.14694.objecter ms_handle_connect 0x7f3a4802dd70 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a36ffd640 10 client.14694.objecter resend_mon_ops 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.145+0000 7f3a4659c640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802dd70 0x7f3a4802e180 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f3a38018000 tx=0x7f3a38001c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a38014ab0 con 0x7f3a4802dd70 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 10 monclient: dump: 2024-09-11T02:54:26.173 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.174 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f3a38014640 con 0x7f3a4802dd70 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802dd70 msgr2=0x7f3a4802e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802dd70 0x7f3a4802e180 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f3a38018000 tx=0x7f3a38001c70 comp rx=0 tx=0).stop 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802dd70 msgr2=0x7f3a4802e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a36ffd640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a4802dd70 0x7f3a4802e180 secure :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f3a38018000 tx=0x7f3a38001c70 comp rx=0 tx=0).stop 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:26.175 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 281938, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:26.176 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.177 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.178 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.179 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.180 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:26.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.184 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.186 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.187 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.189 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:26.190 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.192 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.193 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:26.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.196 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.197 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.198 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:26.198 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:26.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 10 librados: watch_flush enter 2024-09-11T02:54:26.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 10 librados: watch_flush exit 2024-09-11T02:54:26.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 20 client.14694.objecter shutdown clearing up homeless session... 2024-09-11T02:54:26.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 10 client.14694.objecter successfully canceled tick 2024-09-11T02:54:26.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a2803df40 msgr2=0x7f3a28040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a2803df40 0x7f3a28040400 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f3a480a2cb0 tx=0x7f3a4000b040 comp rx=0 tx=0).stop 2024-09-11T02:54:26.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 10 monclient: shutdown 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a48123dd0 msgr2=0x7f3a48138c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 --2- 172.21.15.98:0/3638302554 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3a48123dd0 0x7f3a48138c20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009a00 tx=0x7f3a30005840 comp rx=0 tx=0).stop 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a46d9d640 1 -- 172.21.15.98:0/3638302554 reap_dead start 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a46d9d640 1 -- 172.21.15.98:0/3638302554 reap_dead start 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 shutdown_connections 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 >> 172.21.15.98:0/3638302554 conn(0x7f3a480ac690 msgr2=0x7f3a480a9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 shutdown_connections 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 -- 172.21.15.98:0/3638302554 wait complete. 2024-09-11T02:54:26.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.153+0000 7f3a4c8b9640 1 librados: shutdown 2024-09-11T02:54:26.202 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:26.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:54:26.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient: get_monmap_and_config 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient: monmap: 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:26.318879+0000 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:26.318879+0000 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:26.318 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 1 Processor -- start 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 1 -- start start 2024-09-11T02:54:26.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient: init 2024-09-11T02:54:26.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient: _add_conns ranks=[0,1,7,8,4,2,10,9,6,3,5] 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f300a3690 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0f300a1eb0 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0f300a4180 0x7f0f3013ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient(hunting): picked mon.noname-i con 0x7f0f300a4180 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f300a1ca0 con 0x7f0f300a3690 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f3009dd20 con 0x7f0f300a1eb0 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0f3009ec40 con 0x7f0f300a4180 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 10 monclient(hunting): authenticate will time out at 1220.199120s 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0f300a3690 0x7f0f300a3a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56690/0 (socket says 172.21.15.98:56690) 2024-09-11T02:54:26.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2ffff640 1 -- 172.21.15.98:0/480114379 learned_addr learned my addr 172.21.15.98:0/480114379 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0f300a4180 0x7f0f3013ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3466678314 0 0) 0x7f0f20001120 con 0x7f0f300a3690 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 10 monclient(hunting): get_auth_request con 0x7f0f300a4180 auth_method 0 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient(hunting): _negotiate 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient(hunting): my global_id is 34290 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f18003530 con 0x7f0f300a3690 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f349fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1450523749 0 0) 0x7f0f18003530 con 0x7f0f300a3690 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0f300a4180 msgr2=0x7f0f3013ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0f300a4180 0x7f0f3013ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 >> v1:172.21.15.190:6789/0 conn(0x7f0f300a1eb0 legacy=0x7f0f3013c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3009d480 con 0x7f0f300a3690 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f0f20001b40 con 0x7f0f300a3690 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 5 monclient: authenticate success, global_id 34290 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f3645e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: dump: 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.333 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.334 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.313+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 >> v1:172.21.15.98:6789/0 conn(0x7f0f300a3690 legacy=0x7f0f300a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: _add_conns ranks=[6,0,8,5,7,1,3,4,2] 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f18004240 0x7f0f18004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): picked mon.d con 0x7f0f18004240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f18004d50 0x7f0f180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): picked mon.a con 0x7f0f18004d50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 0x7f0f1800c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): picked mon.e con 0x7f0f18009bc0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1800c900 con 0x7f0f18004d50 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0f1800cab0 con 0x7f0f18004240 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0f1800cc60 con 0x7f0f18009bc0 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f18004240 0x7f0f18004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f18004d50 0x7f0f180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request con 0x7f0f18004240 auth_method 0 2024-09-11T02:54:26.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request con 0x7f0f18004d50 auth_method 0 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 0x7f0f1800c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 10 monclient(hunting): get_auth_request con 0x7f0f18009bc0 auth_method 0 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 1 -- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 msgr2=0x7f0f1800c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:54:26.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 1 -- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 msgr2=0x7f0f1800c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 0x7f0f1800c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 0x7f0f1800c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_done global_id 34290 payload 293 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 -- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 msgr2=0x7f0f1800c050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f18009bc0 0x7f0f1800c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 -- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f18004d50 msgr2=0x7f0f180051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0f18004d50 0x7f0f180051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f349fb640 1 -- 172.21.15.98:0/480114379 reap_dead start 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 monclient: found mon.d 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 -- 172.21.15.98:0/480114379 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f18003530 con 0x7f0f18004240 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f18004240 0x7f0f18004810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f20001ec0 tx=0x7f0f2001c940 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8d52be5e31577dad server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f200ac070 con 0x7f0f18004240 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: dump: 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.340 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f20005d50 con 0x7f0f18004240 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 1 -- 172.21.15.98:0/480114379 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f20007d40 con 0x7f0f18004240 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 4 set_mon_vals no callback set 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f2effd640 10 monclient: dump: 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.341 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.342 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 10 monclient: shutdown 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 1 -- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f18004240 msgr2=0x7f0f18004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f18004240 0x7f0f18004810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0f20001ec0 tx=0x7f0f2001c940 comp rx=0 tx=0).stop 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 1 -- 172.21.15.98:0/480114379 shutdown_connections 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 1 --2- 172.21.15.98:0/480114379 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f18004240 0x7f0f18004810 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 1 -- 172.21.15.98:0/480114379 >> 172.21.15.98:0/480114379 conn(0x7f0f300a4780 msgr2=0x7f0f300a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 1 -- 172.21.15.98:0/480114379 shutdown_connections 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.317+0000 7f0f3645e640 1 -- 172.21.15.98:0/480114379 wait complete. 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: monmap: 2024-09-11T02:54:26.343 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.344 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 librados: starting msgr at 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 librados: starting objecter 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 Processor -- start 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 -- start start 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 librados: setting wanted keys 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 librados: calling monclient init 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: init 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: _add_conns ranks=[2,1,4,0,3,8,5,7,6] 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18004240 0x7f0f301cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): picked mon.noname-b con 0x7f0f18004240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:26.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f30136650 0x7f0f30136a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): picked mon.noname-f con 0x7f0f30136650 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f301201f0 0x7f0f30120650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): picked mon.noname-c con 0x7f0f301201f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f300517e0 con 0x7f0f30136650 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0f300a1aa0 con 0x7f0f18004240 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0f30137c70 con 0x7f0f301201f0 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): authenticate will time out at 1220.205277s 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f301201f0 0x7f0f30120650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18004240 0x7f0f301cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f301201f0 0x7f0f30120650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34228/0 (socket says 172.21.15.98:34228) 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request con 0x7f0f18004240 auth_method 0 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 1 -- 172.21.15.98:0/2746970838 learned_addr learned my addr 172.21.15.98:0/2746970838 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient(hunting): get_auth_request con 0x7f0f301201f0 auth_method 0 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f30136650 0x7f0f30136a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request con 0x7f0f30136650 auth_method 0 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_done global_id 23492 payload 824 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f301201f0 msgr2=0x7f0f30120650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f301201f0 0x7f0f30120650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f30136650 msgr2=0x7f0f30136a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f30136650 0x7f0f30136a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 monclient: found mon.noname-b 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 -- 172.21.15.98:0/2746970838 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f30122cc0 con 0x7f0f18004240 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f30136650 0x7f0f30136a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18004240 0x7f0f301cf000 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009000 tx=0x7f0f18005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 client.?.objecter ms_handle_connect 0x7f0f18004240 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f200ac040 con 0x7f0f18004240 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:54:26.349 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.350 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 5 monclient: authenticate success, global_id 23492 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f200b6070 con 0x7f0f18004240 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: _renew_subs 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f30124010 con 0x7f0f18004240 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f20007d20 con 0x7f0f18004240 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2e7fc640 4 set_mon_vals no callback set 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 client.23492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.352 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: _renew_subs 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f30136fd0 con 0x7f0f18004240 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 librados: init done 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f30120b90 0x7f0f30070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0f3006ca10 con 0x7f0f30120b90 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f30120b90 0x7f0f30070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient: get_auth_request con 0x7f0f30120b90 auth_method 0 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0f20007070 con 0x7f0f18004240 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_done global_id 23458 payload 824 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 1 -- 172.21.15.98:0/2746970838 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f30068b40 con 0x7f0f30120b90 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03dfa0 0x7f0f1c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(193..193 src has 1..193) ==== 7848+0+0 (secure 0 0 0) 0x7f0f200e92d0 con 0x7f0f18004240 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 client.23492.objecter ms_dispatch 0x7f0f30000ba0 osd_map(193..193 src has 1..193) 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 3 client.23492.objecter handle_osd_map got epochs [193,193] > 0 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 3 client.23492.objecter handle_osd_map decoding full epoch 193 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f30120b90 0x7f0f30070380 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f0f18004110 tx=0x7f0f240018a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 20 client.23492.objecter dump_active .. 0 homeless 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 client.23492.objecter ms_handle_connect 0x7f0f30120b90 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 client.23492.objecter resend_mon_ops 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f24019030 con 0x7f0f30120b90 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.355 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03dfa0 0x7f0f1c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f349fb640 10 monclient: get_auth_request con 0x7f0f1c03dfa0 auth_method 0 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0f24015680 con 0x7f0f30120b90 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.321+0000 7f0f2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.325+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f30120b90 msgr2=0x7f0f30070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.325+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f30120b90 0x7f0f30070380 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f0f18004110 tx=0x7f0f240018a0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.325+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f30120b90 msgr2=0x7f0f30070380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.325+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f30120b90 0x7f0f30070380 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.325+0000 7f0f349fb640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03dfa0 0x7f0f1c040460 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7f0f30121920 tx=0x7f0f2800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.325+0000 7f0f2cff9640 10 client.23492.objecter ms_handle_connect 0x7f0f1c03dfa0 2024-09-11T02:54:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f300708c0 0x7f0f30070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0f3006e870 con 0x7f0f300708c0 2024-09-11T02:54:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f300708c0 0x7f0f30070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient: get_auth_request con 0x7f0f300708c0 auth_method 0 2024-09-11T02:54:26.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_done global_id 23467 payload 824 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 1 -- 172.21.15.98:0/2746970838 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f30138000 con 0x7f0f300708c0 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 10 client.23492.objecter ms_handle_connect 0x7f0f300708c0 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f300708c0 0x7f0f30070cd0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f0f240055f0 tx=0x7f0f24001130 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 10 client.23492.objecter resend_mon_ops 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f240076d0 con 0x7f0f300708c0 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.361 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f0f24007260 con 0x7f0f300708c0 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f300708c0 msgr2=0x7f0f30070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f300708c0 0x7f0f30070cd0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f0f240055f0 tx=0x7f0f24001130 comp rx=0 tx=0).stop 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f300708c0 msgr2=0x7f0f30070cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f300708c0 0x7f0f30070cd0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 282114, 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:26.363 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.364 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:26.366 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:26.367 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.368 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:26.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:26.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.375 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.377 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.378 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.380 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.381 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.383 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.384 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.385 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.386 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:26.386 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.386 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.386 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:26.386 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 10 librados: watch_flush enter 2024-09-11T02:54:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 10 librados: watch_flush exit 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 20 client.23492.objecter shutdown clearing up homeless session... 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 10 client.23492.objecter successfully canceled tick 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03dfa0 msgr2=0x7f0f1c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03dfa0 0x7f0f1c040460 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7f0f30121920 tx=0x7f0f2800b040 comp rx=0 tx=0).stop 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 10 monclient: shutdown 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18004240 msgr2=0x7f0f301cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 --2- 172.21.15.98:0/2746970838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0f18004240 0x7f0f301cf000 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009000 tx=0x7f0f18005840 comp rx=0 tx=0).stop 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f349fb640 1 -- 172.21.15.98:0/2746970838 reap_dead start 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f349fb640 1 -- 172.21.15.98:0/2746970838 reap_dead start 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 shutdown_connections 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 >> 172.21.15.98:0/2746970838 conn(0x7f0f300a4780 msgr2=0x7f0f301bf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 shutdown_connections 2024-09-11T02:54:26.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 -- 172.21.15.98:0/2746970838 wait complete. 2024-09-11T02:54:26.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.329+0000 7f0f3645e640 1 librados: shutdown 2024-09-11T02:54:26.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:26.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:54:26.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:54:26.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.501+0000 7fee1ba51640 10 monclient: get_monmap_and_config 2024-09-11T02:54:26.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.501+0000 7fee1ba51640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.501+0000 7fee1ba51640 10 monclient: monmap: 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:26.508878+0000 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:26.508878+0000 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.508 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.509 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.509 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.509 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.509 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:26.509 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:26.509 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 1 Processor -- start 2024-09-11T02:54:26.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 1 -- start start 2024-09-11T02:54:26.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient: init 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient: _add_conns ranks=[0,3,6,10,7,2,8,9,1,4,5] 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee14139b80 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee1406fce0 0x7fee1413ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient(hunting): picked mon.noname-g con 0x7fee1406fce0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee140724b0 0x7fee14140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient(hunting): picked mon.noname-d con 0x7fee140724b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee140a94a0 con 0x7fee14139b80 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fee14071850 con 0x7fee1406fce0 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fee14071aa0 con 0x7fee140724b0 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee1ba51640 10 monclient(hunting): authenticate will time out at 1220.389039s 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee197ed640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fee14139b80 0x7fee14139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56700/0 (socket says 172.21.15.98:56700) 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee140724b0 0x7fee14140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee197ed640 1 -- 172.21.15.98:0/603358196 learned_addr learned my addr 172.21.15.98:0/603358196 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): get_auth_request con 0x7fee140724b0 auth_method 0 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee18fec640 1 --2- 172.21.15.98:0/603358196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee1406fce0 0x7fee1413ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee18fec640 10 monclient(hunting): get_auth_request con 0x7fee1406fce0 auth_method 0 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee18fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee18fec640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee18fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee13fff640 1 -- 172.21.15.98:0/603358196 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736457291 0 0) 0x7fee140a94a0 con 0x7fee14139b80 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee13fff640 10 monclient(hunting): _negotiate 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee13fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee13fff640 10 monclient(hunting): my global_id is 34299 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee13fff640 1 -- 172.21.15.98:0/603358196 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedfc003530 con 0x7fee14139b80 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient(hunting): handle_auth_done global_id 14703 payload 274 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 1 -- 172.21.15.98:0/603358196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee1406fce0 msgr2=0x7fee1413ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 1 --2- 172.21.15.98:0/603358196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee1406fce0 0x7fee1413ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 1 -- 172.21.15.98:0/603358196 >> v1:172.21.15.98:6789/0 conn(0x7fee14139b80 legacy=0x7fee14139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 1 monclient: found mon.noname-d 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 1 -- 172.21.15.98:0/603358196 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee141424a0 con 0x7fee140724b0 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee18fec640 1 --2- 172.21.15.98:0/603358196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee1406fce0 0x7fee1413ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee19fee640 1 --2- 172.21.15.98:0/603358196 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee140724b0 0x7fee14140f50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009dc0 tx=0x7fee0c01c850 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c2d7d675c52576d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.505+0000 7fee13fff640 1 -- 172.21.15.98:0/603358196 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee0c0a6950 con 0x7fee140724b0 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: dump: 2024-09-11T02:54:26.516 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.517 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 5 monclient: authenticate success, global_id 14703 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 1 -- 172.21.15.98:0/603358196 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee0c0a6b10 con 0x7fee140724b0 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 4 set_mon_vals no callback set 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 1 -- 172.21.15.98:0/603358196 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee0c0a6cc0 con 0x7fee140724b0 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee13fff640 10 monclient: dump: 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.519 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: shutdown 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- 172.21.15.98:0/603358196 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee140724b0 msgr2=0x7fee14140f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 --2- 172.21.15.98:0/603358196 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee140724b0 0x7fee14140f50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009dc0 tx=0x7fee0c01c850 comp rx=0 tx=0).stop 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- 172.21.15.98:0/603358196 shutdown_connections 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 --2- 172.21.15.98:0/603358196 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee140724b0 0x7fee14140f50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 --2- 172.21.15.98:0/603358196 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee1406fce0 0x7fee1413ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- 172.21.15.98:0/603358196 >> 172.21.15.98:0/603358196 conn(0x7fee140ac6b0 msgr2=0x7fee140a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- 172.21.15.98:0/603358196 shutdown_connections 2024-09-11T02:54:26.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- 172.21.15.98:0/603358196 wait complete. 2024-09-11T02:54:26.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: monmap: 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 librados: starting msgr at 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 librados: starting objecter 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 Processor -- start 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- start start 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 librados: setting wanted keys 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 librados: calling monclient init 2024-09-11T02:54:26.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: init 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: _add_conns ranks=[6,2,0,7,5,3,4,8,1] 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee1406fce0 0x7fee141d1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): picked mon.noname-d con 0x7fee1406fce0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee140724b0 0x7fee141d1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): picked mon.noname-b con 0x7fee140724b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee14139b80 0x7fee141d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): picked mon.noname-a con 0x7fee14139b80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee14142070 con 0x7fee14139b80 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fee14141ef0 con 0x7fee140724b0 2024-09-11T02:54:26.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fee141421f0 con 0x7fee1406fce0 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient(hunting): authenticate will time out at 1220.394364s 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee1406fce0 0x7fee141d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee1406fce0 0x7fee141d1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:51196/0 (socket says 172.21.15.98:51196) 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee14139b80 0x7fee141d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 -- 172.21.15.98:0/702089815 learned_addr learned my addr 172.21.15.98:0/702089815 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee140724b0 0x7fee141d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): get_auth_request con 0x7fee1406fce0 auth_method 0 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 10 monclient(hunting): get_auth_request con 0x7fee14139b80 auth_method 0 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 10 monclient(hunting): get_auth_request con 0x7fee140724b0 auth_method 0 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee19fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient(hunting): handle_auth_done global_id 14712 payload 824 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee140724b0 msgr2=0x7fee141d1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fee140724b0 0x7fee141d1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee14139b80 msgr2=0x7fee141d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee14139b80 0x7fee141d1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 monclient: found mon.noname-d 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 -- 172.21.15.98:0/702089815 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee140a2620 con 0x7fee1406fce0 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee197ed640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee1406fce0 0x7fee141d1410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c4d0 tx=0x7fee0400c990 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 client.?.objecter ms_handle_connect 0x7fee1406fce0 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee04017070 con 0x7fee1406fce0 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: dump: 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.534 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 5 monclient: authenticate success, global_id 14712 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee04007070 con 0x7fee1406fce0 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: _renew_subs 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee140a3500 con 0x7fee1406fce0 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee040076d0 con 0x7fee1406fce0 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 client.14712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee137fe640 4 set_mon_vals no callback set 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee11ffb640 10 monclient: dump: 2024-09-11T02:54:26.536 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.537 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: _renew_subs 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fee14073610 con 0x7fee1406fce0 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.509+0000 7fee1ba51640 1 librados: init done 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee1ba51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee1ba51640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14132d10 0x7fee141330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee1ba51640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fee1406ca10 con 0x7fee14132d10 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14132d10 0x7fee141330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient: get_auth_request con 0x7fee14132d10 auth_method 0 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fee04007260 con 0x7fee1406fce0 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 10 monclient(hunting): handle_auth_done global_id 14721 payload 824 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 1 -- 172.21.15.98:0/702089815 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee140a5e80 con 0x7fee14132d10 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee0003df40 0x7fee00040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(193..193 src has 1..193) ==== 7848+0+0 (secure 0 0 0) 0x7fee0404efb0 con 0x7fee1406fce0 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 client.14712.objecter ms_dispatch 0x7fee14000ba0 osd_map(193..193 src has 1..193) 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee18fec640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14132d10 0x7fee141330f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee14139220 tx=0x7fee08010c80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 3 client.14712.objecter handle_osd_map got epochs [193,193] > 0 2024-09-11T02:54:26.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 3 client.14712.objecter handle_osd_map decoding full epoch 193 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 20 client.14712.objecter dump_active .. 0 homeless 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 client.14712.objecter ms_handle_connect 0x7fee14132d10 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 client.14712.objecter resend_mon_ops 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee08018070 con 0x7fee14132d10 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 monclient: dump: 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.540 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee19fee640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee0003df40 0x7fee00040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee19fee640 10 monclient: get_auth_request con 0x7fee0003df40 auth_method 0 2024-09-11T02:54:26.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fee08010e70 con 0x7fee14132d10 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14132d10 msgr2=0x7fee141330f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14132d10 0x7fee141330f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee14139220 tx=0x7fee08010c80 comp rx=0 tx=0).stop 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14132d10 msgr2=0x7fee141330f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14132d10 0x7fee141330f0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee14139220 tx=0x7fee08010c80 comp rx=0 tx=0).stop 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee11ffb640 10 client.14712.objecter ms_handle_connect 0x7fee0003df40 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.513+0000 7fee19fee640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee0003df40 0x7fee00040400 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009dc0 tx=0x7fee0c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee1ba51640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee1ba51640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14133630 0x7fee141373e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee1ba51640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fee1406e870 con 0x7fee14133630 2024-09-11T02:54:26.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14133630 0x7fee141373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient: get_auth_request con 0x7fee14133630 auth_method 0 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 10 monclient(hunting): handle_auth_done global_id 14730 payload 824 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.517+0000 7fee18fec640 1 -- 172.21.15.98:0/702089815 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee14139f60 con 0x7fee14133630 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 10 client.14712.objecter ms_handle_connect 0x7fee14133630 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 10 client.14712.objecter resend_mon_ops 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee18fec640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14133630 0x7fee141373e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c10 tx=0x7fee080046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee0802b020 con 0x7fee14133630 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 10 monclient: dump: 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.544 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fee08014430 con 0x7fee14133630 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14133630 msgr2=0x7fee141373e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14133630 0x7fee141373e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c10 tx=0x7fee080046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14133630 msgr2=0x7fee141373e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee11ffb640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee14133630 0x7fee141373e0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee08000c10 tx=0x7fee080046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 146573, 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:26.546 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.547 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:26.548 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.549 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:26.550 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.551 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.552 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:26.554 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.557 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:26.558 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.559 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.561 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.564 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:26.567 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:26.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:26.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:26.570 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 10 librados: watch_flush enter 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 10 librados: watch_flush exit 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 20 client.14712.objecter shutdown clearing up homeless session... 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 10 client.14712.objecter successfully canceled tick 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee0003df40 msgr2=0x7fee00040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee0003df40 0x7fee00040400 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009dc0 tx=0x7fee0c0a7000 comp rx=0 tx=0).stop 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 10 monclient: shutdown 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee1406fce0 msgr2=0x7fee141d1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 --2- 172.21.15.98:0/702089815 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fee1406fce0 0x7fee141d1410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c4d0 tx=0x7fee0400c990 comp rx=0 tx=0).stop 2024-09-11T02:54:26.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee19fee640 1 -- 172.21.15.98:0/702089815 reap_dead start 2024-09-11T02:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee19fee640 1 -- 172.21.15.98:0/702089815 reap_dead start 2024-09-11T02:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 shutdown_connections 2024-09-11T02:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 >> 172.21.15.98:0/702089815 conn(0x7fee140ac6b0 msgr2=0x7fee140a7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 shutdown_connections 2024-09-11T02:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 -- 172.21.15.98:0/702089815 wait complete. 2024-09-11T02:54:26.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.521+0000 7fee1ba51640 1 librados: shutdown 2024-09-11T02:54:26.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:26.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:54:26.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:54:26.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.685+0000 7f07c715b640 10 monclient: get_monmap_and_config 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.685+0000 7f07c715b640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.685+0000 7f07c715b640 10 monclient: monmap: 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:26.693056+0000 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:26.693056+0000 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:26.692 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:26.693 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 1 Processor -- start 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 1 -- start start 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient: init 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient: _add_conns ranks=[0,10,5,9,2,1,6,4,3,7,8] 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07c0139b80 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 0x7f07c013ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient(hunting): picked mon.noname-f con 0x7f07c006fe50 addr v2:172.21.15.190:3300/0 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f07c0072640 0x7f07c0140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient(hunting): picked mon.noname-h con 0x7f07c0072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:26.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07c0071370 con 0x7f07c0139b80 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f07c00719c0 con 0x7f07c006fe50 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f07c0071c10 con 0x7f07c0072640 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 10 monclient(hunting): authenticate will time out at 1220.573362s 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c4ef7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07c0139b80 0x7f07c0139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56714/0 (socket says 172.21.15.98:56714) 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c4ef7640 1 -- 172.21.15.98:0/3590619308 learned_addr learned my addr 172.21.15.98:0/3590619308 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 1 --2- 172.21.15.98:0/3590619308 >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 0x7f07c013ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f07c0072640 0x7f07c0140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577854646 0 0) 0x7f07c0071370 con 0x7f07c0139b80 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): _negotiate 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): my global_id is 34317 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 1 -- 172.21.15.98:0/3590619308 >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 msgr2=0x7f07c013ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f07a8003530 con 0x7f07c0139b80 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 1 -- 172.21.15.98:0/3590619308 >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 msgr2=0x7f07c013ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): get_auth_request con 0x7f07c0072640 auth_method 0 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 1 --2- 172.21.15.98:0/3590619308 >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 0x7f07c013ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 1 --2- 172.21.15.98:0/3590619308 >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 0x7f07c013ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2366466928 0 0) 0x7f07a8003530 con 0x7f07c0139b80 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f07c0072640 msgr2=0x7f07c0140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f07c0072640 0x7f07c0140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 msgr2=0x7f07c013ea00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 --2- 172.21.15.98:0/3590619308 >> v2:172.21.15.190:3300/0 conn(0x7f07c006fe50 0x7f07c013ea00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c00a9600 con 0x7f07c0139b80 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f07b8001d10 con 0x7f07c0139b80 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 5 monclient: authenticate success, global_id 34317 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: dump: 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.711 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.712 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 >> v1:172.21.15.98:6789/0 conn(0x7f07c0139b80 legacy=0x7f07c0139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient: _add_conns ranks=[3,1,4,2,8,6,7,0,5] 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a8004210 0x7f07a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): picked mon.g con 0x7f07a8004210 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a8004da0 0x7f07a8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): picked mon.f con 0x7f07a8004da0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f07a8009c40 0x7f07a800c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): picked mon.c con 0x7f07a8009c40 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:26.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f07c00719c0 con 0x7f07a8004da0 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f07a800c9b0 con 0x7f07a8004210 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f07a800cb60 con 0x7f07a8009c40 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c715b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f07a8009c40 0x7f07a800c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): get_auth_request con 0x7f07a8009c40 auth_method 0 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c4ef7640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a8004210 0x7f07a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a8004da0 0x7f07a8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c4ef7640 10 monclient(hunting): get_auth_request con 0x7f07a8004210 auth_method 0 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c4ef7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c4ef7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c4ef7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 10 monclient(hunting): get_auth_request con 0x7f07a8004da0 auth_method 0 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient(hunting): handle_auth_done global_id 34317 payload 293 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 -- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a8004210 msgr2=0x7f07a8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f07a8004210 0x7f07a8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 -- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a8004da0 msgr2=0x7f07a8005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07a8004da0 0x7f07a8005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 monclient: found mon.c 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 1 -- 172.21.15.98:0/3590619308 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a8003530 con 0x7f07a8009c40 2024-09-11T02:54:26.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.689+0000 7f07c56f8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c56f8640 1 -- 172.21.15.98:0/3590619308 reap_dead start 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c56f8640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f07a8009c40 0x7f07a800c0d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009ab0 tx=0x7f07b001ca70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=19376446a253eec server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07b00096b0 con 0x7f07a8009c40 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 10 monclient: dump: 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.717 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.718 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b00a59c0 con 0x7f07a8009c40 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 1 -- 172.21.15.98:0/3590619308 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07b00ad480 con 0x7f07a8009c40 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 4 set_mon_vals no callback set 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bf7fe640 10 monclient: dump: 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.719 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.720 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient: shutdown 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- 172.21.15.98:0/3590619308 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f07a8009c40 msgr2=0x7f07a800c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f07a8009c40 0x7f07a800c0d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f07b0009ab0 tx=0x7f07b001ca70 comp rx=0 tx=0).stop 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- 172.21.15.98:0/3590619308 shutdown_connections 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 --2- 172.21.15.98:0/3590619308 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f07a8009c40 0x7f07a800c0d0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- 172.21.15.98:0/3590619308 >> 172.21.15.98:0/3590619308 conn(0x7f07c00ac730 msgr2=0x7f07c00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- 172.21.15.98:0/3590619308 shutdown_connections 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- 172.21.15.98:0/3590619308 wait complete. 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient: monmap: 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:26.722 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 librados: starting msgr at 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 librados: starting objecter 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 Processor -- start 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- start start 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 librados: setting wanted keys 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 librados: calling monclient init 2024-09-11T02:54:26.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient: init 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient: _add_conns ranks=[0,6,7,8,4,2,3,5,1] 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a8009c40 0x7f07c01d0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): picked mon.noname-a con 0x7f07a8009c40 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f07c0138940 0x7f07c0138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): picked mon.noname-d con 0x7f07c0138940 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f07c013d610 0x7f07c013da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): picked mon.noname-i con 0x7f07c013d610 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f07c013a2f0 con 0x7f07a8009c40 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f07c013a170 con 0x7f07c0138940 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f07c013a470 con 0x7f07c013d610 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c715b640 10 monclient(hunting): authenticate will time out at 1220.579240s 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a8009c40 0x7f07c01d0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f07c0138940 0x7f07c0138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a8009c40 0x7f07c01d0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42836/0 (socket says 172.21.15.98:42836) 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): get_auth_request con 0x7f07c0138940 auth_method 0 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 1 -- 172.21.15.98:0/168847974 learned_addr learned my addr 172.21.15.98:0/168847974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 10 monclient(hunting): get_auth_request con 0x7f07a8009c40 auth_method 0 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c56f8640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f07c013d610 0x7f07c013da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c56f8640 10 monclient(hunting): get_auth_request con 0x7f07c013d610 auth_method 0 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c56f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c56f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c56f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient(hunting): handle_auth_done global_id 14739 payload 824 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f07c013d610 msgr2=0x7f07c013da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f07c013d610 0x7f07c013da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a8009c40 msgr2=0x7f07c01d0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a8009c40 0x7f07c01d0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 monclient: found mon.noname-d 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 -- 172.21.15.98:0/168847974 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c013f0d0 con 0x7f07c0138940 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07c4ef7640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f07a8009c40 0x7f07c01d0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bffff640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f07c0138940 0x7f07c0138d40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f07b400e4d0 tx=0x7f07b400e990 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bd7fa640 10 client.?.objecter ms_handle_connect 0x7f07c0138940 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.693+0000 7f07bd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07b4019020 con 0x7f07c0138940 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: dump: 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.728 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.729 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b4015b00 con 0x7f07c0138940 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 5 monclient: authenticate success, global_id 14739 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 10 monclient: _renew_subs 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c01232d0 con 0x7f07c0138940 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07b4015cb0 con 0x7f07c0138940 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:26.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 10 client.14739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: dump: 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.731 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 10 monclient: _renew_subs 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07beffd640 4 set_mon_vals no callback set 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c013fc30 con 0x7f07c0138940 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 1 librados: init done 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c0122ea0 0x7f07c013dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f07c006ca10 con 0x7f07c0122ea0 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c0122ea0 0x7f07c013dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient: get_auth_request con 0x7f07c0122ea0 auth_method 0 2024-09-11T02:54:26.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f07b4015690 con 0x7f07c0138940 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f07ac03df70 0x7f07ac040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_done global_id 23957 payload 824 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 1 -- 172.21.15.98:0/168847974 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07c0122ba0 con 0x7f07c0122ea0 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(193..193 src has 1..193) ==== 7848+0+0 (secure 0 0 0) 0x7f07b400f040 con 0x7f07c0138940 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 client.14739.objecter ms_dispatch 0x7f07c0000ba0 osd_map(193..193 src has 1..193) 2024-09-11T02:54:26.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 3 client.14739.objecter handle_osd_map got epochs [193,193] > 0 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 3 client.14739.objecter handle_osd_map decoding full epoch 193 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 20 client.14739.objecter dump_active .. 0 homeless 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 client.14739.objecter ms_handle_connect 0x7f07c0122ea0 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 client.14739.objecter resend_mon_ops 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c4ef7640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c0122ea0 0x7f07c013dfd0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f07b80014f0 tx=0x7f07b8011980 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c56f8640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f07ac03df70 0x7f07ac040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07b8015bb0 con 0x7f07c0122ea0 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: dump: 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.734 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c56f8640 10 monclient: get_auth_request con 0x7f07ac03df70 auth_method 0 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07b8015740 con 0x7f07c0122ea0 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c0122ea0 msgr2=0x7f07c013dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c0122ea0 0x7f07c013dfd0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f07b80014f0 tx=0x7f07b8011980 comp rx=0 tx=0).stop 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c0122ea0 msgr2=0x7f07c013dfd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c0122ea0 0x7f07c013dfd0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07bd7fa640 10 client.14739.objecter ms_handle_connect 0x7f07ac03df70 2024-09-11T02:54:26.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.697+0000 7f07c56f8640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f07ac03df70 0x7f07ac040430 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f07a800cd30 tx=0x7f07b00046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c00a4540 0x7f07c00a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f07c006e870 con 0x7f07c00a4540 2024-09-11T02:54:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c00a4540 0x7f07c00a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient: get_auth_request con 0x7f07c00a4540 auth_method 0 2024-09-11T02:54:26.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 10 monclient(hunting): handle_auth_done global_id 23966 payload 824 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 1 -- 172.21.15.98:0/168847974 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07c0070ea0 con 0x7f07c00a4540 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 10 client.14739.objecter ms_handle_connect 0x7f07c00a4540 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 10 client.14739.objecter resend_mon_ops 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c4ef7640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c00a4540 0x7f07c00a4950 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f07b8018000 tx=0x7f07b8002050 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07b8013a70 con 0x7f07c00a4540 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 10 monclient: dump: 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.740 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f07b8013600 con 0x7f07c00a4540 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c00a4540 msgr2=0x7f07c00a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c00a4540 0x7f07c00a4950 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f07b8018000 tx=0x7f07b8002050 comp rx=0 tx=0).stop 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c00a4540 msgr2=0x7f07c00a4950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07bd7fa640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f07c00a4540 0x7f07c00a4950 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 282498, 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:26.742 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.743 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:26.744 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:26.746 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.747 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.749 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.751 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:26.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.753 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:26.754 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.755 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.756 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.757 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.758 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.759 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.760 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.762 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.764 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.765 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:26.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.766 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:26.766 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 10 librados: watch_flush enter 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 10 librados: watch_flush exit 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 20 client.14739.objecter shutdown clearing up homeless session... 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 10 client.14739.objecter successfully canceled tick 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f07ac03df70 msgr2=0x7f07ac040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f07ac03df70 0x7f07ac040430 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f07a800cd30 tx=0x7f07b00046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 10 monclient: shutdown 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f07c0138940 msgr2=0x7f07c0138d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 --2- 172.21.15.98:0/168847974 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f07c0138940 0x7f07c0138d40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f07b400e4d0 tx=0x7f07b400e990 comp rx=0 tx=0).stop 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c56f8640 1 -- 172.21.15.98:0/168847974 reap_dead start 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c56f8640 1 -- 172.21.15.98:0/168847974 reap_dead start 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 shutdown_connections 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 >> 172.21.15.98:0/168847974 conn(0x7f07c00ac730 msgr2=0x7f07c00a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 shutdown_connections 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 -- 172.21.15.98:0/168847974 wait complete. 2024-09-11T02:54:26.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.705+0000 7f07c715b640 1 librados: shutdown 2024-09-11T02:54:26.769 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:26.770 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:54:26.770 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:54:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient: get_monmap_and_config 2024-09-11T02:54:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient: monmap: 2024-09-11T02:54:26.884 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:26.885800+0000 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:26.885800+0000 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.885 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.886 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:26.886 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:26.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 Processor -- start 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 -- start start 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient: init 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient: _add_conns ranks=[5,9,4,0,2,10,1,3,7,8,6] 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e4072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): picked mon.noname-h con 0x7fe1e4139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1e4139ef0 0x7fe1e40a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): picked mon.noname-a con 0x7fe1e4139ef0 addr v2:172.21.15.98:3300/0 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe1e413a2d0 0x7fe1e40a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): picked mon.noname-c con 0x7fe1e413a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe1e40702b0 con 0x7fe1e4139ef0 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe1e4071bf0 con 0x7fe1e413a2d0 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fe1e4071350 con 0x7fe1e4139b10 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1eb1a5640 10 monclient(hunting): authenticate will time out at 1220.766130s 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1e4139ef0 0x7fe1e40a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe1e413a2d0 0x7fe1e40a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1e4139ef0 0x7fe1e40a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42838/0 (socket says 172.21.15.98:42838) 2024-09-11T02:54:26.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 -- 172.21.15.98:0/4232893722 learned_addr learned my addr 172.21.15.98:0/4232893722 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): get_auth_request con 0x7fe1e4139ef0 auth_method 0 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 10 monclient(hunting): get_auth_request con 0x7fe1e413a2d0 auth_method 0 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e8f41640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e4072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e8f41640 10 monclient(hunting): get_auth_request con 0x7fe1e4139b10 auth_method 0 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e8f41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e8f41640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e8f41640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient(hunting): handle_auth_done global_id 34335 payload 274 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 -- 172.21.15.98:0/4232893722 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 msgr2=0x7fe1e4072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e4072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 -- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe1e413a2d0 msgr2=0x7fe1e40a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe1e413a2d0 0x7fe1e40a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 monclient: found mon.noname-a 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1dbfff640 1 -- 172.21.15.98:0/4232893722 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e40a9620 con 0x7fe1e4139ef0 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.881+0000 7fe1e9742640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe1e413a2d0 0x7fe1e40a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1dbfff640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e4139ef0 0x7fe1e40a7a10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40098a0 tx=0x7fe1d401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ed929548b1f995 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 1 -- 172.21.15.98:0/4232893722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1d40b0070 con 0x7fe1e4139ef0 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: dump: 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.894 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.895 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 5 monclient: authenticate success, global_id 34335 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 1 -- 172.21.15.98:0/4232893722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1d40aa810 con 0x7fe1e4139ef0 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 4 set_mon_vals no callback set 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 1 -- 172.21.15.98:0/4232893722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1d40a9300 con 0x7fe1e4139ef0 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1db7fe640 10 monclient: dump: 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.897 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient: shutdown 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e4139ef0 msgr2=0x7fe1e40a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e4139ef0 0x7fe1e40a7a10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40098a0 tx=0x7fe1d401c490 comp rx=0 tx=0).stop 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/4232893722 shutdown_connections 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe1e413a2d0 0x7fe1e40a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1e4139ef0 0x7fe1e40a7a10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/4232893722 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e4072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/4232893722 >> 172.21.15.98:0/4232893722 conn(0x7fe1e40ac790 msgr2=0x7fe1e40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/4232893722 shutdown_connections 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/4232893722 wait complete. 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient: build_initial_monmap 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient: monmap: 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:26.899 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:26.900 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 librados: starting msgr at 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 librados: starting objecter 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 Processor -- start 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- start start 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 librados: setting wanted keys 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 librados: calling monclient init 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient: init 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient: _add_conns ranks=[5,7,8,2,6,3,0,4,1] 2024-09-11T02:54:26.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e40a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): picked mon.noname-h con 0x7fe1e4139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe1e4139ef0 0x7fe1e413f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): picked mon.noname-i con 0x7fe1e4139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1e413a2d0 0x7fe1e413fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): picked mon.noname-e con 0x7fe1e413a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fe1e4143460 con 0x7fe1e4139b10 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe1e40a88a0 con 0x7fe1e4139ef0 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe1e41435e0 con 0x7fe1e413a2d0 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:26.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1eb1a5640 10 monclient(hunting): authenticate will time out at 1220.771502s 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1e413a2d0 0x7fe1e413fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1e413a2d0 0x7fe1e413fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:50852/0 (socket says 172.21.15.98:50852) 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 1 -- 172.21.15.98:0/3450409484 learned_addr learned my addr 172.21.15.98:0/3450409484 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): get_auth_request con 0x7fe1e413a2d0 auth_method 0 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1dbfff640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe1e4139ef0 0x7fe1e413f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e8f41640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e40a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1dbfff640 10 monclient(hunting): get_auth_request con 0x7fe1e4139ef0 auth_method 0 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 10 monclient(hunting): handle_auth_done global_id 23975 payload 824 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe1e4139ef0 msgr2=0x7fe1e413f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe1e4139ef0 0x7fe1e413f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 msgr2=0x7fe1e40a5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.885+0000 7fe1e9742640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e40a5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e9742640 10 monclient: _finish_hunting 0 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e9742640 1 monclient: found mon.noname-e 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e9742640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e9742640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:26.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e9742640 1 -- 172.21.15.98:0/3450409484 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e40a9f70 con 0x7fe1e413a2d0 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient: get_auth_request con 0x7fe1e4139b10 auth_method 0 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fe1e4139b10 0x7fe1e40a5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e9742640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1e413a2d0 0x7fe1e413fe70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00efc0 tx=0x7fe1dc00c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 client.?.objecter ms_handle_connect 0x7fe1e413a2d0 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1dc018030 con 0x7fe1e413a2d0 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: dump: 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.911 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: _finish_auth 0 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: _check_auth_tickets 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1dc00cdf0 con 0x7fe1e413a2d0 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 5 monclient: authenticate success, global_id 23975 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 10 monclient: _renew_subs 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e40aa4a0 con 0x7fe1e413a2d0 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1dc014d60 con 0x7fe1e413a2d0 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:26.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 10 client.23975.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: dump: 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.914 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1daffd640 4 set_mon_vals no callback set 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 10 monclient: _renew_subs 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e413d410 con 0x7fe1e413a2d0 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 1 librados: init done 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1570 0x7fe1e41d1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e406ca10 con 0x7fe1e41d1570 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe1dc014480 con 0x7fe1e413a2d0 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1570 0x7fe1e41d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient: get_auth_request con 0x7fe1e41d1570 auth_method 0 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe1d003df40 0x7fe1d0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(193..193 src has 1..193) ==== 7848+0+0 (secure 0 0 0) 0x7fe1dc019070 con 0x7fe1e413a2d0 2024-09-11T02:54:26.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 client.23975.objecter ms_dispatch 0x7fe1e4000ba0 osd_map(193..193 src has 1..193) 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 3 client.23975.objecter handle_osd_map got epochs [193,193] > 0 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 3 client.23975.objecter handle_osd_map decoding full epoch 193 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 20 client.23975.objecter dump_active .. 0 homeless 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1dbfff640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe1d003df40 0x7fe1d0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1dbfff640 10 monclient: get_auth_request con 0x7fe1d003df40 auth_method 0 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_done global_id 24832 payload 824 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 1 -- 172.21.15.98:0/3450409484 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1e41403b0 con 0x7fe1e41d1570 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1e8f41640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1570 0x7fe1e41d1950 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e740 tx=0x7fe1e000ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 client.23975.objecter ms_handle_connect 0x7fe1e41d1570 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 client.23975.objecter resend_mon_ops 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 client.23975.objecter ms_handle_connect 0x7fe1d003df40 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1dbfff640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe1d003df40 0x7fe1d0040400 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40ad000 tx=0x7fe1d40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1e0017070 con 0x7fe1e41d1570 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: dump: 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.918 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe1e000ed30 con 0x7fe1e41d1570 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1570 msgr2=0x7fe1e41d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1570 0x7fe1e41d1950 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e740 tx=0x7fe1e000ec00 comp rx=0 tx=0).stop 2024-09-11T02:54:26.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1570 msgr2=0x7fe1e41d1950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.889+0000 7fe1d97fa640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1570 0x7fe1e41d1950 secure :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e740 tx=0x7fe1e000ec00 comp rx=0 tx=0).stop 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1eb1a5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1e90 0x7fe1e4137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1eb1a5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e406e870 con 0x7fe1e41d1e90 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1e90 0x7fe1e4137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient: get_auth_request con 0x7fe1e41d1e90 auth_method 0 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:26.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 10 monclient(hunting): handle_auth_done global_id 24841 payload 824 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 1 -- 172.21.15.98:0/3450409484 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1e406e680 con 0x7fe1e41d1e90 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1d97fa640 10 client.23975.objecter ms_handle_connect 0x7fe1e41d1e90 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1d97fa640 10 client.23975.objecter resend_mon_ops 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1e8f41640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1e90 0x7fe1e4137ba0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e740 tx=0x7fe1e00046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1e000f040 con 0x7fe1e41d1e90 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1d97fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.897+0000 7fe1d97fa640 10 monclient: dump: 2024-09-11T02:54:26.921 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:26.922 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9603 (secure 0 0 0) 0x7fe1e000ce40 con 0x7fe1e41d1e90 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1e90 msgr2=0x7fe1e4137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1d97fa640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1e90 0x7fe1e4137ba0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e740 tx=0x7fe1e00046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1d97fa640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1e90 msgr2=0x7fe1e4137ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1d97fa640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1e41d1e90 0x7fe1e4137ba0 secure :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000e740 tx=0x7fe1e00046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.923 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 282668, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:26.924 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.925 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.926 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:26.927 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:26.928 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.929 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:26.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:26.935 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.936 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.940 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:26.945 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.946 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.947 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:26.948 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 10 librados: watch_flush enter 2024-09-11T02:54:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 10 librados: watch_flush exit 2024-09-11T02:54:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 20 client.23975.objecter shutdown clearing up homeless session... 2024-09-11T02:54:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 10 client.23975.objecter successfully canceled tick 2024-09-11T02:54:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe1d003df40 msgr2=0x7fe1d0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe1d003df40 0x7fe1d0040400 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40ad000 tx=0x7fe1d40a6000 comp rx=0 tx=0).stop 2024-09-11T02:54:26.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 10 monclient: shutdown 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1e413a2d0 msgr2=0x7fe1e413fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1e413a2d0 0x7fe1e413fe70 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00efc0 tx=0x7fe1dc00c490 comp rx=0 tx=0).stop 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1e9742640 1 -- 172.21.15.98:0/3450409484 reap_dead start 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 shutdown_connections 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 --2- 172.21.15.98:0/3450409484 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1e413a2d0 0x7fe1e413fe70 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 >> 172.21.15.98:0/3450409484 conn(0x7fe1e40ac790 msgr2=0x7fe1e40a7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 shutdown_connections 2024-09-11T02:54:26.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 -- 172.21.15.98:0/3450409484 wait complete. 2024-09-11T02:54:26.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:26.901+0000 7fe1eb1a5640 1 librados: shutdown 2024-09-11T02:54:26.952 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:26.952 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:54:26.952 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.069+0000 7f98a838a640 10 monclient: get_monmap_and_config 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.069+0000 7f98a838a640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.069+0000 7f98a838a640 10 monclient: monmap: 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:27.076448+0000 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:27.076448+0000 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.075 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:27.076 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 Processor -- start 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 -- start start 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient: init 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient: _add_conns ranks=[2,7,8,4,3,0,5,1,9,6,10] 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a009f4a0 0x7f98a009f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): picked mon.noname-b con 0x7f98a009f4a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f98a0073860 0x7f98a00a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): picked mon.noname-i con 0x7f98a0073860 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f98a00a2bd0 0x7f98a00a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): picked mon.noname-e con 0x7f98a00a2bd0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f98a0077910 con 0x7f98a009f4a0 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f98a00a3050 con 0x7f98a0073860 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f98a009cf60 con 0x7f98a00a2bd0 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a838a640 10 monclient(hunting): authenticate will time out at 1220.957260s 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a009f4a0 0x7f98a009f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f98a00a2bd0 0x7f98a00a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a009f4a0 0x7f98a009f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:50924/0 (socket says 172.21.15.98:50924) 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 -- 172.21.15.98:0/3259155097 learned_addr learned my addr 172.21.15.98:0/3259155097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): get_auth_request con 0x7f98a009f4a0 auth_method 0 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 10 monclient(hunting): get_auth_request con 0x7f98a00a2bd0 auth_method 0 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a5925640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f98a0073860 0x7f98a00a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient(hunting): handle_auth_done global_id 23510 payload 274 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 -- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f98a00a2bd0 msgr2=0x7f98a00a4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f98a00a2bd0 0x7f98a00a4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 -- 172.21.15.98:0/3259155097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f98a0073860 msgr2=0x7f98a00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f98a0073860 0x7f98a00a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 monclient: found mon.noname-b 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 -- 172.21.15.98:0/3259155097 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a0137a50 con 0x7f98a009f4a0 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a5925640 10 monclient: get_auth_request con 0x7f98a0073860 auth_method 0 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6927640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f98a00a2bd0 0x7f98a00a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a5925640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f98a0073860 0x7f98a00a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:27.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.073+0000 7f98a6126640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a009f4a0 0x7f98a009f880 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098a0 tx=0x7f989c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ccc0238381bed06 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 1 -- 172.21.15.98:0/3259155097 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f989c0b0070 con 0x7f98a009f4a0 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: dump: 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.090 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 1 -- 172.21.15.98:0/3259155097 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f989c0aa810 con 0x7f98a009f4a0 2024-09-11T02:54:27.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 1 -- 172.21.15.98:0/3259155097 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f989c0a9300 con 0x7f98a009f4a0 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 5 monclient: authenticate success, global_id 23510 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 4 set_mon_vals no callback set 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5124640 10 monclient: dump: 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.092 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: shutdown 2024-09-11T02:54:27.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a009f4a0 msgr2=0x7f98a009f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a009f4a0 0x7f98a009f880 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f989c0098a0 tx=0x7f989c01c440 comp rx=0 tx=0).stop 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- 172.21.15.98:0/3259155097 shutdown_connections 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f98a00a2bd0 0x7f98a00a4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f98a0073860 0x7f98a00a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 --2- 172.21.15.98:0/3259155097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f98a009f4a0 0x7f98a009f880 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- 172.21.15.98:0/3259155097 >> 172.21.15.98:0/3259155097 conn(0x7f98a006eed0 msgr2=0x7f98a006f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- 172.21.15.98:0/3259155097 shutdown_connections 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- 172.21.15.98:0/3259155097 wait complete. 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: monmap: 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:27.094 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 librados: starting msgr at 2024-09-11T02:54:27.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 librados: starting objecter 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 Processor -- start 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- start start 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 librados: setting wanted keys 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 librados: calling monclient init 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: init 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: _add_conns ranks=[1,6,5,4,7,3,2,8,0] 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98a0073860 0x7f98a01d1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): picked mon.noname-f con 0x7f98a0073860 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f98a009f4a0 0x7f98a01d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): picked mon.noname-d con 0x7f98a009f4a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f98a00a2bd0 0x7f98a01d2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): picked mon.noname-h con 0x7f98a00a2bd0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f98a0141440 con 0x7f98a0073860 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f98a01412c0 con 0x7f98a00a2bd0 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f98a01415c0 con 0x7f98a009f4a0 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient(hunting): authenticate will time out at 1220.962450s 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f98a009f4a0 0x7f98a01d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f98a009f4a0 0x7f98a01d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:51240/0 (socket says 172.21.15.98:51240) 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 -- 172.21.15.98:0/2281883091 learned_addr learned my addr 172.21.15.98:0/2281883091 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): get_auth_request con 0x7f98a009f4a0 auth_method 0 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6126640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98a0073860 0x7f98a01d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6126640 10 monclient(hunting): get_auth_request con 0x7f98a0073860 auth_method 0 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6126640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6126640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6927640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f98a00a2bd0 0x7f98a01d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6927640 10 monclient(hunting): get_auth_request con 0x7f98a00a2bd0 auth_method 0 2024-09-11T02:54:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6927640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6927640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a6927640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient(hunting): handle_auth_done global_id 14748 payload 824 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f98a00a2bd0 msgr2=0x7f98a01d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f98a00a2bd0 0x7f98a01d2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98a0073860 msgr2=0x7f98a01d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98a0073860 0x7f98a01d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 monclient: found mon.noname-d 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 -- 172.21.15.98:0/2281883091 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a01201c0 con 0x7f98a009f4a0 2024-09-11T02:54:27.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a5925640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f98a009f4a0 0x7f98a01d1cc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f989000c7f0 tx=0x7f989000ccb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 client.?.objecter ms_handle_connect 0x7f98a009f4a0 2024-09-11T02:54:27.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:27.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9890015070 con 0x7f98a009f4a0 2024-09-11T02:54:27.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:27.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: dump: 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.104 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9890007ca0 con 0x7f98a009f4a0 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 5 monclient: authenticate success, global_id 14748 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: _renew_subs 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9890007e50 con 0x7f98a009f4a0 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98a009e970 con 0x7f98a009f4a0 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a4923640 4 set_mon_vals no callback set 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f9896ffd640 10 monclient: dump: 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.106 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 client.14748.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: _renew_subs 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f98a011fee0 con 0x7f98a009f4a0 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.077+0000 7f98a838a640 1 librados: init done 2024-09-11T02:54:27.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a838a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a838a640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0131fe0 0x7f98a01323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f98a011fb60 con 0x7f98a0131fe0 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9890022060 con 0x7f98a009f4a0 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0131fe0 0x7f98a01323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f988403e0e0 0x7f98840405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient: get_auth_request con 0x7f98a0131fe0 auth_method 0 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(193..193 src has 1..193) ==== 7848+0+0 (secure 0 0 0) 0x7f989004fa50 con 0x7f98a009f4a0 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 client.14748.objecter ms_dispatch 0x7f98a0000ba0 osd_map(193..193 src has 1..193) 2024-09-11T02:54:27.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 3 client.14748.objecter handle_osd_map got epochs [193,193] > 0 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 3 client.14748.objecter handle_osd_map decoding full epoch 193 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 20 client.14748.objecter dump_active .. 0 homeless 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6927640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f988403e0e0 0x7f98840405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6927640 10 monclient: get_auth_request con 0x7f988403e0e0 auth_method 0 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 10 monclient(hunting): handle_auth_done global_id 19938 payload 824 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 1 -- 172.21.15.98:0/2281883091 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98a009f880 con 0x7f98a0131fe0 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 client.14748.objecter ms_handle_connect 0x7f98a0131fe0 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 client.14748.objecter resend_mon_ops 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6126640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0131fe0 0x7f98a01323c0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f989c009870 tx=0x7f989c0191e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 client.14748.objecter ms_handle_connect 0x7f988403e0e0 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f98a6927640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f988403e0e0 0x7f98840405a0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f98a00a1a20 tx=0x7f989800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f989c0b9030 con 0x7f98a0131fe0 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 monclient: dump: 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.110 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f989c0b1040 con 0x7f98a0131fe0 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0131fe0 msgr2=0x7f98a01323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0131fe0 0x7f98a01323c0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f989c009870 tx=0x7f989c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0131fe0 msgr2=0x7f98a01323c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:27.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.081+0000 7f9896ffd640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0131fe0 0x7f98a01323c0 secure :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f989c009870 tx=0x7f989c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a838a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a838a640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0132900 0x7f98a0134760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a838a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f98a006ca10 con 0x7f98a0132900 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0132900 0x7f98a0134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient: get_auth_request con 0x7f98a0132900 auth_method 0 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 10 monclient(hunting): handle_auth_done global_id 19947 payload 824 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 1 -- 172.21.15.98:0/2281883091 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f98a00008e0 con 0x7f98a0132900 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f98a6126640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0132900 0x7f98a0134760 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f989c01c270 tx=0x7f989c0a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f9896ffd640 10 client.14748.objecter ms_handle_connect 0x7f98a0132900 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.089+0000 7f9896ffd640 10 client.14748.objecter resend_mon_ops 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f989c0b9040 con 0x7f98a0132900 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 10 monclient: dump: 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.115 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f989c0c8410 con 0x7f98a0132900 2024-09-11T02:54:27.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0132900 msgr2=0x7f98a0134760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0132900 0x7f98a0134760 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f989c01c270 tx=0x7f989c0a8000 comp rx=0 tx=0).stop 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0132900 msgr2=0x7f98a0134760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f9896ffd640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f98a0132900 0x7f98a0134760 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:27.117 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 282848, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.118 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.119 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.120 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:27.121 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.122 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.125 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.126 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.127 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.128 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:27.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:27.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.138 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.139 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:54:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.140 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:27.140 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 10 librados: watch_flush enter 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 10 librados: watch_flush exit 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 20 client.14748.objecter shutdown clearing up homeless session... 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 10 client.14748.objecter successfully canceled tick 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f988403e0e0 msgr2=0x7f98840405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f988403e0e0 0x7f98840405a0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f98a00a1a20 tx=0x7f989800b040 comp rx=0 tx=0).stop 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a6927640 1 -- 172.21.15.98:0/2281883091 reap_dead start 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 10 monclient: shutdown 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f98a009f4a0 msgr2=0x7f98a01d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.101+0000 7f98a838a640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f98a009f4a0 0x7f98a01d1cc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f989000c7f0 tx=0x7f989000ccb0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.105+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 shutdown_connections 2024-09-11T02:54:27.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.105+0000 7f98a838a640 1 --2- 172.21.15.98:0/2281883091 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f98a009f4a0 0x7f98a01d1cc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.105+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 >> 172.21.15.98:0/2281883091 conn(0x7f98a006eed0 msgr2=0x7f98a00a7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:27.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.105+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 shutdown_connections 2024-09-11T02:54:27.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.105+0000 7f98a838a640 1 -- 172.21.15.98:0/2281883091 wait complete. 2024-09-11T02:54:27.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.105+0000 7f98a838a640 1 librados: shutdown 2024-09-11T02:54:27.143 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:27.143 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:54:27.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient: get_monmap_and_config 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient: monmap: 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:27.265432+0000 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:27.265432+0000 2024-09-11T02:54:27.264 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:27.265 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 1 Processor -- start 2024-09-11T02:54:27.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 1 -- start start 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient: init 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient: _add_conns ranks=[3,1,4,2,7,9,8,0,10,5,6] 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c139b80 0x7f569c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient(hunting): picked mon.noname-g con 0x7f569c139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f569c06fe50 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f569c072640 0x7f569c0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient(hunting): picked mon.noname-c con 0x7f569c072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f569c071370 con 0x7f569c06fe50 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f569c0719c0 con 0x7f569c139b80 2024-09-11T02:54:27.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f569c071c10 con 0x7f569c072640 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a2cfd640 10 monclient(hunting): authenticate will time out at 1221.145761s 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f569c072640 0x7f569c0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f569c072640 0x7f569c0a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:33070/0 (socket says 172.21.15.98:33070) 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 -- 172.21.15.98:0/3479241612 learned_addr learned my addr 172.21.15.98:0/3479241612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): get_auth_request con 0x7f569c072640 auth_method 0 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f569bfff640 1 --1- 172.21.15.98:0/3479241612 >> v1:172.21.15.190:6789/0 conn(0x7f569c06fe50 0x7f569c0a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:49346/0 (socket says 172.21.15.98:49346) 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 1 --2- 172.21.15.98:0/3479241612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c139b80 0x7f569c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 10 monclient(hunting): get_auth_request con 0x7f569c139b80 auth_method 0 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient(hunting): handle_auth_done global_id 23485 payload 274 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 -- 172.21.15.98:0/3479241612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c139b80 msgr2=0x7f569c139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 --2- 172.21.15.98:0/3479241612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c139b80 0x7f569c139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 -- 172.21.15.98:0/3479241612 >> v1:172.21.15.190:6789/0 conn(0x7f569c06fe50 legacy=0x7f569c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 monclient: found mon.noname-c 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.261+0000 7f56a129a640 1 -- 172.21.15.98:0/3479241612 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569c0a9600 con 0x7f569c072640 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 --2- 172.21.15.98:0/3479241612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f569c072640 0x7f569c0a8120 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f568c0098a0 tx=0x7f568c01c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=423a642f06264d1e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 1 -- 172.21.15.98:0/3479241612 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c0b1070 con 0x7f569c072640 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: dump: 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.279 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 5 monclient: authenticate success, global_id 23485 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 1 -- 172.21.15.98:0/3479241612 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c0aa7b0 con 0x7f569c072640 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 4 set_mon_vals no callback set 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 1 -- 172.21.15.98:0/3479241612 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c0aae50 con 0x7f569c072640 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569b7fe640 10 monclient: dump: 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient: shutdown 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- 172.21.15.98:0/3479241612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f569c072640 msgr2=0x7f569c0a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 --2- 172.21.15.98:0/3479241612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f569c072640 0x7f569c0a8120 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f568c0098a0 tx=0x7f568c01c440 comp rx=0 tx=0).stop 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- 172.21.15.98:0/3479241612 shutdown_connections 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 --2- 172.21.15.98:0/3479241612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f569c072640 0x7f569c0a8120 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 --2- 172.21.15.98:0/3479241612 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c139b80 0x7f569c139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- 172.21.15.98:0/3479241612 >> 172.21.15.98:0/3479241612 conn(0x7f569c0ac730 msgr2=0x7f569c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- 172.21.15.98:0/3479241612 shutdown_connections 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- 172.21.15.98:0/3479241612 wait complete. 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient: monmap: 2024-09-11T02:54:27.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 librados: starting msgr at 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 librados: starting objecter 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 Processor -- start 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- start start 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 librados: setting wanted keys 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 librados: calling monclient init 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient: init 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient: _add_conns ranks=[3,5,6,1,7,0,8,2,4] 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c06fe50 0x7f569c13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): picked mon.noname-g con 0x7f569c06fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c072640 0x7f569c13ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): picked mon.noname-h con 0x7f569c072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f569c139b80 0x7f569c13b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): picked mon.noname-d con 0x7f569c139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f569c144450 con 0x7f569c06fe50 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f569c0a8870 con 0x7f569c072640 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f569c1445d0 con 0x7f569c139b80 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a2cfd640 10 monclient(hunting): authenticate will time out at 1221.151229s 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f569c139b80 0x7f569c13b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f569c139b80 0x7f569c13b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54960/0 (socket says 172.21.15.98:54960) 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 -- 172.21.15.98:0/1494637570 learned_addr learned my addr 172.21.15.98:0/1494637570 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): get_auth_request con 0x7f569c139b80 auth_method 0 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a0a99640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c06fe50 0x7f569c13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569bfff640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c072640 0x7f569c13ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a0a99640 10 monclient(hunting): get_auth_request con 0x7f569c06fe50 auth_method 0 2024-09-11T02:54:27.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a0a99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a0a99640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a0a99640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569bfff640 10 monclient(hunting): get_auth_request con 0x7f569c072640 auth_method 0 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f569bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient(hunting): handle_auth_done global_id 14757 payload 824 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c072640 msgr2=0x7f569c13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c072640 0x7f569c13ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c06fe50 msgr2=0x7f569c13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f569c06fe50 0x7f569c13a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 monclient: found mon.noname-d 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 -- 172.21.15.98:0/1494637570 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569c13ce90 con 0x7f569c139b80 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56a129a640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f569c139b80 0x7f569c13b250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f568c01aaf0 tx=0x7f568c0191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.265+0000 7f56997fa640 10 client.?.objecter ms_handle_connect 0x7f569c139b80 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c0ba070 con 0x7f569c139b80 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: dump: 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.289 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568c0aadd0 con 0x7f569c139b80 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 5 monclient: authenticate success, global_id 14757 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 10 monclient: _renew_subs 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569c13d170 con 0x7f569c139b80 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f568c0b1040 con 0x7f569c139b80 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f569affd640 4 set_mon_vals no callback set 2024-09-11T02:54:27.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: dump: 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.292 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 10 monclient: _renew_subs 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f569c140fc0 con 0x7f569c139b80 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 1 librados: init done 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f568c004310 con 0x7f569c139b80 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c133760 0x7f569c133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f569c0a8220 con 0x7f569c133760 2024-09-11T02:54:27.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f567c03df40 0x7f567c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(194..194 src has 1..194) ==== 7848+0+0 (secure 0 0 0) 0x7f568c0e8e10 con 0x7f569c139b80 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 client.14757.objecter ms_dispatch 0x7f569c000ba0 osd_map(194..194 src has 1..194) 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 3 client.14757.objecter handle_osd_map got epochs [194,194] > 0 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 3 client.14757.objecter handle_osd_map decoding full epoch 194 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 20 client.14757.objecter dump_active .. 0 homeless 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c133760 0x7f569c133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient: get_auth_request con 0x7f569c133760 auth_method 0 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f569bfff640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f567c03df40 0x7f567c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f569bfff640 10 monclient: get_auth_request con 0x7f567c03df40 auth_method 0 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 10 monclient(hunting): handle_auth_done global_id 19670 payload 824 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 1 -- 172.21.15.98:0/1494637570 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f569c139f60 con 0x7f569c133760 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 client.14757.objecter ms_handle_connect 0x7f569c133760 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 client.14757.objecter resend_mon_ops 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56a0a99640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c133760 0x7f569c133b40 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f5684011510 tx=0x7f56840119d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 client.14757.objecter ms_handle_connect 0x7f567c03df40 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f569bfff640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f567c03df40 0x7f567c040400 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f5694005fd0 tx=0x7f5694005ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f56840044a0 con 0x7f569c133760 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: dump: 2024-09-11T02:54:27.295 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5684004030 con 0x7f569c133760 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c133760 msgr2=0x7f569c133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c133760 0x7f569c133b40 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f5684011510 tx=0x7f56840119d0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c133760 msgr2=0x7f569c133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:27.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.269+0000 7f56997fa640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c133760 0x7f569c133b40 secure :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f5684011510 tx=0x7f56840119d0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a2cfd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a2cfd640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c134080 0x7f569c0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a2cfd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f569c06ca10 con 0x7f569c134080 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c134080 0x7f569c0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient: get_auth_request con 0x7f569c134080 auth_method 0 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 10 monclient(hunting): handle_auth_done global_id 19679 payload 824 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 1 -- 172.21.15.98:0/1494637570 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f569c069090 con 0x7f569c134080 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56a0a99640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c134080 0x7f569c0a0e60 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f5684011510 tx=0x7f56840046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56997fa640 10 client.14757.objecter ms_handle_connect 0x7f569c134080 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56997fa640 10 client.14757.objecter resend_mon_ops 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5684005c70 con 0x7f569c134080 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56997fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:27.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.277+0000 7f56997fa640 10 monclient: dump: 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.301 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f5684005c70 con 0x7f569c134080 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56997fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56997fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c134080 msgr2=0x7f569c0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56997fa640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c134080 0x7f569c0a0e60 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f5684011510 tx=0x7f56840046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56997fa640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c134080 msgr2=0x7f569c0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56997fa640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f569c134080 0x7f569c0a0e60 secure :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f5684011510 tx=0x7f56840046d0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.302 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 283028, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:27.303 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.305 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.306 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.306 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.306 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.306 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.306 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.306 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.307 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:27.307 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.309 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.313 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:27.315 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:27.316 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.318 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:27.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:27.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:27.325 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.326 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.327 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.328 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:27.328 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 10 librados: watch_flush enter 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 10 librados: watch_flush exit 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 20 client.14757.objecter shutdown clearing up homeless session... 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 10 client.14757.objecter successfully canceled tick 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f567c03df40 msgr2=0x7f567c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f567c03df40 0x7f567c040400 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f5694005fd0 tx=0x7f5694005ba0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 10 monclient: shutdown 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f569c139b80 msgr2=0x7f569c13b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 --2- 172.21.15.98:0/1494637570 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f569c139b80 0x7f569c13b250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f568c01aaf0 tx=0x7f568c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a129a640 1 -- 172.21.15.98:0/1494637570 reap_dead start 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a129a640 1 -- 172.21.15.98:0/1494637570 reap_dead start 2024-09-11T02:54:27.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 shutdown_connections 2024-09-11T02:54:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 >> 172.21.15.98:0/1494637570 conn(0x7f569c0ac730 msgr2=0x7f569c0aa5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 shutdown_connections 2024-09-11T02:54:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 -- 172.21.15.98:0/1494637570 wait complete. 2024-09-11T02:54:27.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.281+0000 7f56a2cfd640 1 librados: shutdown 2024-09-11T02:54:27.331 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:27.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:54:27.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient: get_monmap_and_config 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient: monmap: 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:27.449407+0000 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:27.449407+0000 2024-09-11T02:54:27.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:27.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 Processor -- start 2024-09-11T02:54:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 -- start start 2024-09-11T02:54:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient: init 2024-09-11T02:54:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient: _add_conns ranks=[7,8,2,4,1,0,6,10,9,5,3] 2024-09-11T02:54:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a5830 0x7fc7500a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): picked mon.noname-i con 0x7fc7500a5830 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a1770 0x7fc7500a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): picked mon.noname-e con 0x7fc7500a1770 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc750071f10 0x7fc7501375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): picked mon.noname-b con 0x7fc750071f10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc7500a6690 con 0x7fc750071f10 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc7500739f0 con 0x7fc7500a5830 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc750071cd0 con 0x7fc7500a1770 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc757098640 10 monclient(hunting): authenticate will time out at 1221.329603s 2024-09-11T02:54:27.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a1770 0x7fc7500a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a1770 0x7fc7500a6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37268/0 (socket says 172.21.15.98:37268) 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc750071f10 0x7fc7501375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 -- 172.21.15.98:0/1190354111 learned_addr learned my addr 172.21.15.98:0/1190354111 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): get_auth_request con 0x7fc7500a1770 auth_method 0 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 10 monclient(hunting): get_auth_request con 0x7fc750071f10 auth_method 0 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc754e34640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a5830 0x7fc7500a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient(hunting): handle_auth_done global_id 23993 payload 274 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 -- 172.21.15.98:0/1190354111 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a5830 msgr2=0x7fc7500a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a5830 0x7fc7500a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 -- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc750071f10 msgr2=0x7fc7501375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc750071f10 0x7fc7501375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 monclient: found mon.noname-e 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc74ffff640 1 -- 172.21.15.98:0/1190354111 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc750073cd0 con 0x7fc7500a1770 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc754e34640 10 monclient: get_auth_request con 0x7fc7500a5830 auth_method 0 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc754e34640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a5830 0x7fc7500a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.445+0000 7fc755635640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc750071f10 0x7fc7501375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a1770 0x7fc7500a6150 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc7380098d0 tx=0x7fc73801cbb0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ca479e2d9ea861a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 1 -- 172.21.15.98:0/1190354111 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7380b0070 con 0x7fc7500a1770 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: dump: 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 5 monclient: authenticate success, global_id 23993 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 1 -- 172.21.15.98:0/1190354111 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7380aa810 con 0x7fc7500a1770 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 4 set_mon_vals no callback set 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 1 -- 172.21.15.98:0/1190354111 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7380a9300 con 0x7fc7500a1770 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74f7fe640 10 monclient: dump: 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.467 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: shutdown 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a1770 msgr2=0x7fc7500a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a1770 0x7fc7500a6150 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fc7380098d0 tx=0x7fc73801cbb0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- 172.21.15.98:0/1190354111 shutdown_connections 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc750071f10 0x7fc7501375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a1770 0x7fc7500a6150 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 --2- 172.21.15.98:0/1190354111 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a5830 0x7fc7500a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- 172.21.15.98:0/1190354111 >> 172.21.15.98:0/1190354111 conn(0x7fc7500aa420 msgr2=0x7fc7500a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- 172.21.15.98:0/1190354111 shutdown_connections 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- 172.21.15.98:0/1190354111 wait complete. 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: monmap: 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:27.469 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 librados: starting msgr at 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 librados: starting objecter 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 Processor -- start 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- start start 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 librados: setting wanted keys 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 librados: calling monclient init 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: init 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: _add_conns ranks=[7,8,0,6,1,3,4,2,5] 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc750071f10 0x7fc75013b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): picked mon.noname-i con 0x7fc750071f10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a5830 0x7fc75013be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): picked mon.noname-e con 0x7fc7500a5830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc75013c350 0x7fc7501d10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): picked mon.noname-a con 0x7fc75013c350 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc750140de0 con 0x7fc75013c350 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc750140c60 con 0x7fc750071f10 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc750140f60 con 0x7fc7500a5830 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient(hunting): authenticate will time out at 1221.334709s 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a5830 0x7fc75013be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a5830 0x7fc75013be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37278/0 (socket says 172.21.15.98:37278) 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 -- 172.21.15.98:0/26379960 learned_addr learned my addr 172.21.15.98:0/26379960 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): get_auth_request con 0x7fc7500a5830 auth_method 0 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc75013c350 0x7fc7501d10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 10 monclient(hunting): get_auth_request con 0x7fc75013c350 auth_method 0 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc754e34640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc750071f10 0x7fc75013b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient(hunting): handle_auth_done global_id 24002 payload 824 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc750071f10 msgr2=0x7fc75013b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc750071f10 0x7fc75013b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc75013c350 msgr2=0x7fc7501d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc75013c350 0x7fc7501d10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 monclient: found mon.noname-e 2024-09-11T02:54:27.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 -- 172.21.15.98:0/26379960 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75013d6d0 con 0x7fc7500a5830 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc754e34640 10 monclient: get_auth_request con 0x7fc750071f10 auth_method 0 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc755635640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc75013c350 0x7fc7501d10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc754e34640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc750071f10 0x7fc75013b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74ffff640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a5830 0x7fc75013be10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc7380097c0 tx=0x7fc738018ef0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 client.?.objecter ms_handle_connect 0x7fc7500a5830 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7380b9070 con 0x7fc7500a5830 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: dump: 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7380044b0 con 0x7fc7500a5830 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7380b0040 con 0x7fc7500a5830 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74d7fa640 10 monclient: dump: 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.476 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 5 monclient: authenticate success, global_id 24002 2024-09-11T02:54:27.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc74effd640 4 set_mon_vals no callback set 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: _renew_subs 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc750070580 con 0x7fc7500a5830 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 client.24002.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: _renew_subs 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75006ede0 con 0x7fc7500a5830 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.449+0000 7fc757098640 1 librados: init done 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc757098640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc757098640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a1770 0x7fc75002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc75006ca10 con 0x7fc7500a1770 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc7380aacb0 con 0x7fc7500a5830 2024-09-11T02:54:27.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a1770 0x7fc75002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc73003df40 0x7fc730040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient: get_auth_request con 0x7fc7500a1770 auth_method 0 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(194..194 src has 1..194) ==== 7848+0+0 (secure 0 0 0) 0x7fc7380e9310 con 0x7fc7500a5830 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 client.24002.objecter ms_dispatch 0x7fc750000ba0 osd_map(194..194 src has 1..194) 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 3 client.24002.objecter handle_osd_map got epochs [194,194] > 0 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 3 client.24002.objecter handle_osd_map decoding full epoch 194 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 20 client.24002.objecter dump_active .. 0 homeless 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc755635640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc73003df40 0x7fc730040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc755635640 10 monclient: get_auth_request con 0x7fc73003df40 auth_method 0 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 10 monclient(hunting): handle_auth_done global_id 19276 payload 824 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 1 -- 172.21.15.98:0/26379960 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc750070d60 con 0x7fc7500a1770 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 client.24002.objecter ms_handle_connect 0x7fc7500a1770 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc754e34640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a1770 0x7fc75002d450 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc74800ddd0 tx=0x7fc74800b6c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 client.24002.objecter resend_mon_ops 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc74800f8c0 con 0x7fc7500a1770 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 monclient: dump: 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.480 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.481 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 client.24002.objecter ms_handle_connect 0x7fc73003df40 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc755635640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc73003df40 0x7fc730040400 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7fc740009580 tx=0x7fc740009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc74800f450 con 0x7fc7500a1770 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a1770 msgr2=0x7fc75002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a1770 0x7fc75002d450 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc74800ddd0 tx=0x7fc74800b6c0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a1770 msgr2=0x7fc75002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:27.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.453+0000 7fc74d7fa640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc7500a1770 0x7fc75002d450 secure :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc74800ddd0 tx=0x7fc74800b6c0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc75002d990 0x7fc75002dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc75006e870 con 0x7fc75002d990 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc75002d990 0x7fc75002dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient: get_auth_request con 0x7fc75002d990 auth_method 0 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 10 monclient(hunting): handle_auth_done global_id 19285 payload 824 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 1 -- 172.21.15.98:0/26379960 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc75013de20 con 0x7fc75002d990 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc754e34640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc75002d990 0x7fc75002dda0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc74800bfd0 tx=0x7fc748004a50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 10 client.24002.objecter ms_handle_connect 0x7fc75002d990 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 10 client.24002.objecter resend_mon_ops 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7480298b0 con 0x7fc75002d990 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 10 monclient: dump: 2024-09-11T02:54:27.485 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.486 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7fc748017420 con 0x7fc75002d990 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc75002d990 msgr2=0x7fc75002dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc75002d990 0x7fc75002dda0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc74800bfd0 tx=0x7fc748004a50 comp rx=0 tx=0).stop 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc75002d990 msgr2=0x7fc75002dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc74d7fa640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc75002d990 0x7fc75002dda0 secure :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc74800bfd0 tx=0x7fc748004a50 comp rx=0 tx=0).stop 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:27.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 42, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 283232, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:54:27.488 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.489 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.490 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:27.491 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:27.492 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.493 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:27.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:27.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:27.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.500 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.502 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:27.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:27.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:27.510 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:27.511 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:27.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 10 librados: watch_flush enter 2024-09-11T02:54:27.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 10 librados: watch_flush exit 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 20 client.24002.objecter shutdown clearing up homeless session... 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 10 client.24002.objecter successfully canceled tick 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc73003df40 msgr2=0x7fc730040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc73003df40 0x7fc730040400 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7fc740009580 tx=0x7fc740009190 comp rx=0 tx=0).stop 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 10 monclient: shutdown 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a5830 msgr2=0x7fc75013be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 --2- 172.21.15.98:0/26379960 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc7500a5830 0x7fc75013be10 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fc7380097c0 tx=0x7fc738018ef0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc755635640 1 -- 172.21.15.98:0/26379960 reap_dead start 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc755635640 1 -- 172.21.15.98:0/26379960 reap_dead start 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 shutdown_connections 2024-09-11T02:54:27.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 >> 172.21.15.98:0/26379960 conn(0x7fc7500aa420 msgr2=0x7fc7500a86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:27.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 shutdown_connections 2024-09-11T02:54:27.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 -- 172.21.15.98:0/26379960 wait complete. 2024-09-11T02:54:27.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.461+0000 7fc757098640 1 librados: shutdown 2024-09-11T02:54:27.514 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:54:27.514 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a', 'b', 'g', 'd'] 2024-09-11T02:54:27.514 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T02:54:27.514 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T02:54:27.514 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T02:54:27.515 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T02:54:27.515 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:54:27.526 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T02:54:27.526 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T02:54:27.526 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T02:54:27.526 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T02:54:27.526 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:54:27.581 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T02:54:27.581 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-09-11T02:54:27.581 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-09-11T02:54:27.581 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-11T02:54:27.581 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:54:27.607 INFO:tasks.ceph.mon.g:Stopped 2024-09-11T02:54:27.607 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-09-11T02:54:27.607 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-11T02:54:27.607 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-11T02:54:27.607 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:54:27.636 INFO:tasks.ceph.mon.d:Stopped 2024-09-11T02:54:27.636 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-09-11T02:54:27.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:54:27.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient: get_monmap_and_config 2024-09-11T02:54:27.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient: monmap: 2024-09-11T02:54:27.794 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.794 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.794 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:27.795616+0000 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:27.795616+0000 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.795 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:27.796 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:27.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 Processor -- start 2024-09-11T02:54:27.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 -- start start 2024-09-11T02:54:27.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient: init 2024-09-11T02:54:27.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient: _add_conns ranks=[10,7,5,8,3,2,4,1,6,9,0] 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d4c06f650 0x7f6d4c072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): picked mon.noname-f con 0x7f6d4c06f650 addr v2:172.21.15.190:3300/0 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6d4c0a8380 0x7f6d4c06fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): picked mon.noname-i con 0x7f6d4c0a8380 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c06ff70 0x7f6d4c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): picked mon.noname-h con 0x7f6d4c06ff70 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6d4c0a8800 con 0x7f6d4c06f650 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f6d4c071a80 con 0x7f6d4c06ff70 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f6d4c071d60 con 0x7f6d4c0a8380 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.789+0000 7f6d5359c640 10 monclient(hunting): authenticate will time out at 1221.675915s 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c06ff70 0x7f6d4c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d4c06f650 0x7f6d4c072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6d4c0a8380 0x7f6d4c06fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6d4c06f650 0x7f6d4c072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:59074/0 (socket says 172.21.15.98:59074) 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c06ff70 0x7f6d4c1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:39806/0 (socket says 172.21.15.98:39806) 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 1 -- 172.21.15.98:0/3965276659 learned_addr learned my addr 172.21.15.98:0/3965276659 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 --2- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6d4c0a8380 0x7f6d4c06fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:56664/0 (socket says 172.21.15.98:56664) 2024-09-11T02:54:27.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): get_auth_request con 0x7f6d4c0a8380 auth_method 0 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 10 monclient(hunting): get_auth_request con 0x7f6d4c06f650 auth_method 0 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 10 monclient(hunting): get_auth_request con 0x7f6d4c06ff70 auth_method 0 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient(hunting): handle_auth_done global_id 19294 payload 274 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 -- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c06ff70 msgr2=0x7f6d4c1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 --2- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c06ff70 0x7f6d4c1397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 -- 172.21.15.98:0/3965276659 >> v2:172.21.15.190:3300/0 conn(0x7f6d4c06f650 msgr2=0x7f6d4c072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 --2- 172.21.15.98:0/3965276659 >> v2:172.21.15.190:3300/0 conn(0x7f6d4c06f650 0x7f6d4c072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 monclient: found mon.noname-i 2024-09-11T02:54:27.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:54:27.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d50b37640 1 -- 172.21.15.98:0/3965276659 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c0714c0 con 0x7f6d4c0a8380 2024-09-11T02:54:27.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.793+0000 7f6d51338640 1 --2- 172.21.15.98:0/3965276659 >> v2:172.21.15.190:3300/0 conn(0x7f6d4c06f650 0x7f6d4c072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:27.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d50b37640 1 --2- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6d4c0a8380 0x7f6d4c06fa30 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f6d400097c0 tx=0x7f6d4001cad0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=14c33b1867acf588 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 1 -- 172.21.15.98:0/3965276659 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d400b0070 con 0x7f6d4c0a8380 2024-09-11T02:54:27.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: dump: 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.810 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 1 -- 172.21.15.98:0/3965276659 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d400aa810 con 0x7f6d4c0a8380 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 1 -- 172.21.15.98:0/3965276659 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d400a9300 con 0x7f6d4c0a8380 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:27.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d4bfff640 10 monclient: dump: 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.812 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 5 monclient: authenticate success, global_id 19294 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 4 set_mon_vals no callback set 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient: shutdown 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6d4c0a8380 msgr2=0x7f6d4c06fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 --2- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6d4c0a8380 0x7f6d4c06fa30 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f6d400097c0 tx=0x7f6d4001cad0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- 172.21.15.98:0/3965276659 shutdown_connections 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 --2- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c06ff70 0x7f6d4c1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 --2- 172.21.15.98:0/3965276659 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6d4c0a8380 0x7f6d4c06fa30 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 --2- 172.21.15.98:0/3965276659 >> v2:172.21.15.190:3300/0 conn(0x7f6d4c06f650 0x7f6d4c072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- 172.21.15.98:0/3965276659 >> 172.21.15.98:0/3965276659 conn(0x7f6d4c0ac8e0 msgr2=0x7f6d4c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- 172.21.15.98:0/3965276659 shutdown_connections 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- 172.21.15.98:0/3965276659 wait complete. 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient: build_initial_monmap 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient: monmap: 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:27.814 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 librados: starting msgr at 2024-09-11T02:54:27.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 librados: starting objecter 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 Processor -- start 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- start start 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 librados: setting wanted keys 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 librados: calling monclient init 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient: init 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient: _add_conns ranks=[0,3,5,6,1,4,7,8,2] 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d4c06f650 0x7f6d4c1d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): picked mon.noname-a con 0x7f6d4c06f650 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d4c06ff70 0x7f6d4c138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): picked mon.noname-g con 0x7f6d4c06ff70 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c0a8380 0x7f6d4c13da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): picked mon.noname-h con 0x7f6d4c0a8380 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c142fc0 con 0x7f6d4c06f650 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6d4c142e40 con 0x7f6d4c06ff70 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f6d4c143140 con 0x7f6d4c0a8380 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d5359c640 10 monclient(hunting): authenticate will time out at 1221.683080s 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51338640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d4c06f650 msgr2=0x7f6d4c1d11d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51338640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d4c06f650 0x7f6d4c1d11d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d50b37640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d4c06ff70 msgr2=0x7f6d4c138910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:54:27.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d50b37640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d4c06ff70 0x7f6d4c138910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c0a8380 0x7f6d4c13da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c0a8380 0x7f6d4c13da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:39818/0 (socket says 172.21.15.98:39818) 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 1 -- 172.21.15.98:0/2682902734 learned_addr learned my addr 172.21.15.98:0/2682902734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 10 monclient(hunting): get_auth_request con 0x7f6d4c0a8380 auth_method 0 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.797+0000 7f6d51b39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 10 monclient(hunting): handle_auth_done global_id 19697 payload 824 2024-09-11T02:54:27.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 1 -- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d4c06ff70 msgr2=0x7f6d4c138910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d4c06ff70 0x7f6d4c138910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 1 -- 172.21.15.98:0/2682902734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d4c06f650 msgr2=0x7f6d4c1d11d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d4c06f650 0x7f6d4c1d11d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 10 monclient: _finish_hunting 0 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 1 monclient: found mon.noname-h 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 1 -- 172.21.15.98:0/2682902734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c123fd0 con 0x7f6d4c0a8380 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51b39640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c0a8380 0x7f6d4c13da20 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00e930 tx=0x7f6d3c00edf0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 client.?.objecter ms_handle_connect 0x7f6d4c0a8380 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:27.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 1 -- 172.21.15.98:0/2682902734 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d3c017070 con 0x7f6d4c0a8380 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: dump: 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.820 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 1 -- 172.21.15.98:0/2682902734 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d3c00ce10 con 0x7f6d4c0a8380 2024-09-11T02:54:27.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 1 -- 172.21.15.98:0/2682902734 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6d3c0138e0 con 0x7f6d4c0a8380 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 5 monclient: authenticate success, global_id 19697 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 monclient: _renew_subs 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4c06fbc0 con 0x7f6d4c0a8380 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d4b7fe640 4 set_mon_vals no callback set 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 monclient: dump: 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:27.822 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:27.823 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 client.19697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 monclient: _renew_subs 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4c123cf0 con 0x7f6d4c0a8380 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 1 librados: init done 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d4c1397c0 con 0x7f6d4c0a8380 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 1 -- 172.21.15.98:0/2682902734 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6d3c024060 con 0x7f6d4c0a8380 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6d3803df40 0x7f6d38040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:27.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 1 -- 172.21.15.98:0/2682902734 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(194..194 src has 1..194) ==== 7848+0+0 (secure 0 0 0) 0x7f6d3c00f040 con 0x7f6d4c0a8380 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 client.19697.objecter ms_dispatch 0x7f6d4c000ba0 osd_map(194..194 src has 1..194) 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 3 client.19697.objecter handle_osd_map got epochs [194,194] > 0 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 3 client.19697.objecter handle_osd_map decoding full epoch 194 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 20 client.19697.objecter dump_active .. 0 homeless 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51338640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6d3803df40 0x7f6d38040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51338640 10 monclient: get_auth_request con 0x7f6d3803df40 auth_method 0 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d49ffb640 10 client.19697.objecter ms_handle_connect 0x7f6d3803df40 2024-09-11T02:54:27.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.801+0000 7f6d51338640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6d3803df40 0x7f6d38040400 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f6d44007240 tx=0x7f6d4400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.813+0000 7f6d49ffb640 1 -- 172.21.15.98:0/2682902734 <== mon.5 v2:172.21.15.190:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6d3c024380 con 0x7f6d4c0a8380 2024-09-11T02:54:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.813+0000 7f6d49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:27.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:27.813+0000 7f6d49ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.033+0000 7f6d5359c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:54:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.033+0000 7f6d5359c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.033+0000 7f6d5359c640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:54:28.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.033+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6d4c06e3d0 con 0x7f6d4c0a8380 2024-09-11T02:54:28.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.033+0000 7f6d49ffb640 1 -- 172.21.15.98:0/2682902734 <== mon.5 v2:172.21.15.190:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f6d3c00c9a0 con 0x7f6d4c0a8380 2024-09-11T02:54:28.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.033+0000 7f6d49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:28.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.033+0000 7f6d49ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:28.039 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":42,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:54:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 10 librados: watch_flush enter 2024-09-11T02:54:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 10 librados: watch_flush exit 2024-09-11T02:54:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 20 client.19697.objecter shutdown clearing up homeless session... 2024-09-11T02:54:28.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 10 client.19697.objecter successfully canceled tick 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6d3803df40 msgr2=0x7f6d38040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6d3803df40 0x7f6d38040400 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f6d44007240 tx=0x7f6d4400b040 comp rx=0 tx=0).stop 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 10 monclient: shutdown 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c0a8380 msgr2=0x7f6d4c13da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c0a8380 0x7f6d4c13da20 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00e930 tx=0x7f6d3c00edf0 comp rx=0 tx=0).stop 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 shutdown_connections 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6d3803df40 0x7f6d38040400 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f6d4c0a8380 0x7f6d4c13da20 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6d4c06ff70 0x7f6d4c138910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 --2- 172.21.15.98:0/2682902734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6d4c06f650 0x7f6d4c1d11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 >> 172.21.15.98:0/2682902734 conn(0x7f6d4c0ac8e0 msgr2=0x7f6d4c0a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:28.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 shutdown_connections 2024-09-11T02:54:28.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 -- 172.21.15.98:0/2682902734 wait complete. 2024-09-11T02:54:28.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:28.037+0000 7f6d5359c640 1 librados: shutdown 2024-09-11T02:54:31.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient: get_monmap_and_config 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient: build_initial_monmap 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient: monmap: 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:31.187334+0000 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:31.187334+0000 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:31.186 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:31.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:31.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 Processor -- start 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 -- start start 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient: init 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient: _add_conns ranks=[5,10,4,3,0,6,1,2,8,9,7] 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efea40a4030 0x7efea40a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): picked mon.noname-h con 0x7efea40a4030 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efea409ca50 0x7efea409ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): picked mon.noname-f con 0x7efea409ca50 addr v2:172.21.15.190:3300/0 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efea409d370 0x7efea413c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): picked mon.noname-c con 0x7efea409d370 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:31.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efea40a1470 con 0x7efea409ca50 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7efea409e320 con 0x7efea409d370 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7efea40a4410 con 0x7efea40a4030 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efeab47f640 10 monclient(hunting): authenticate will time out at 1225.067765s 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efea9a1c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efea409d370 0x7efea413c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.181+0000 7efea9a1c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efea409d370 0x7efea413c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:33092/0 (socket says 172.21.15.98:33092) 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 -- 172.21.15.98:0/3061241936 learned_addr learned my addr 172.21.15.98:0/3061241936 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): get_auth_request con 0x7efea409d370 auth_method 0 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 1 --2- 172.21.15.98:0/3061241936 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efea40a4030 0x7efea40a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea8a1a640 1 --2- 172.21.15.98:0/3061241936 >> v2:172.21.15.190:3300/0 conn(0x7efea409ca50 0x7efea409ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 10 monclient(hunting): get_auth_request con 0x7efea40a4030 auth_method 0 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea8a1a640 10 monclient(hunting): get_auth_request con 0x7efea409ca50 auth_method 0 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea8a1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea8a1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea8a1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea921b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient(hunting): handle_auth_done global_id 23494 payload 274 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 -- 172.21.15.98:0/3061241936 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efea40a4030 msgr2=0x7efea40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 --2- 172.21.15.98:0/3061241936 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efea40a4030 0x7efea40a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 -- 172.21.15.98:0/3061241936 >> v2:172.21.15.190:3300/0 conn(0x7efea409ca50 msgr2=0x7efea409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 --2- 172.21.15.98:0/3061241936 >> v2:172.21.15.190:3300/0 conn(0x7efea409ca50 0x7efea409ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient: _finish_hunting 0 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 monclient: found mon.noname-c 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 -- 172.21.15.98:0/3061241936 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea409ecc0 con 0x7efea409d370 2024-09-11T02:54:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea8a1a640 1 --2- 172.21.15.98:0/3061241936 >> v2:172.21.15.190:3300/0 conn(0x7efea409ca50 0x7efea409ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea9a1c640 1 --2- 172.21.15.98:0/3061241936 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efea409d370 0x7efea413c590 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00aa40 tx=0x7efe9c01d7d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ac0a44697bfc80a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 1 -- 172.21.15.98:0/3061241936 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe9c00b040 con 0x7efea409d370 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: dump: 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: _finish_auth 0 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 1 -- 172.21.15.98:0/3061241936 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9c0a9880 con 0x7efea409d370 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 1 -- 172.21.15.98:0/3061241936 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe9c0b1400 con 0x7efea409d370 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 5 monclient: authenticate success, global_id 23494 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 4 set_mon_vals no callback set 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efea3fff640 10 monclient: dump: 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 10 monclient: shutdown 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 1 -- 172.21.15.98:0/3061241936 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efea409d370 msgr2=0x7efea413c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 1 --2- 172.21.15.98:0/3061241936 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efea409d370 0x7efea413c590 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00aa40 tx=0x7efe9c01d7d0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 1 -- 172.21.15.98:0/3061241936 shutdown_connections 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 1 --2- 172.21.15.98:0/3061241936 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efea409d370 0x7efea413c590 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 1 --2- 172.21.15.98:0/3061241936 >> v2:172.21.15.190:3300/0 conn(0x7efea409ca50 0x7efea409ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 1 --2- 172.21.15.98:0/3061241936 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efea40a4030 0x7efea40a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.185+0000 7efeab47f640 1 -- 172.21.15.98:0/3061241936 >> 172.21.15.98:0/3061241936 conn(0x7efea40a4890 msgr2=0x7efea40a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- 172.21.15.98:0/3061241936 shutdown_connections 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- 172.21.15.98:0/3061241936 wait complete. 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: build_initial_monmap 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: monmap: 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:31.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 librados: starting msgr at 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 librados: starting objecter 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 Processor -- start 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- start start 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 librados: setting wanted keys 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 librados: calling monclient init 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: init 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _add_conns ranks=[6,8,0,4,2,5,1,3,7] 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efea409ca50 0x7efea41d1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): picked mon.noname-d con 0x7efea409ca50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efea409d370 0x7efea4138a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): picked mon.noname-e con 0x7efea409d370 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea40a4030 0x7efea4138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): picked mon.noname-a con 0x7efea40a4030 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea921b640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efea409ca50 msgr2=0x7efea41d1210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efea413ec60 con 0x7efea40a4030 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7efea413eae0 con 0x7efea409ca50 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea921b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efea409ca50 0x7efea41d1210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7efea413ede0 con 0x7efea409d370 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient(hunting): authenticate will time out at 1225.074000s 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea9a1c640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea40a4030 msgr2=0x7efea4138fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea9a1c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea40a4030 0x7efea4138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efea409d370 0x7efea4138a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efea409d370 0x7efea4138a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37282/0 (socket says 172.21.15.98:37282) 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 -- 172.21.15.98:0/73331106 learned_addr learned my addr 172.21.15.98:0/73331106 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): get_auth_request con 0x7efea409d370 auth_method 0 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient(hunting): handle_auth_done global_id 24011 payload 824 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 -- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efea409ca50 msgr2=0x7efea41d1210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efea409ca50 0x7efea41d1210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 -- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea40a4030 msgr2=0x7efea4138fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea40a4030 0x7efea4138fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient: _finish_hunting 0 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 monclient: found mon.noname-e 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 -- 172.21.15.98:0/73331106 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea41273e0 con 0x7efea409d370 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea8a1a640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efea409d370 0x7efea4138a70 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009fc0 tx=0x7efe8c00d560 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 client.?.objecter ms_handle_connect 0x7efea409d370 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 1 -- 172.21.15.98:0/73331106 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe8c01a070 con 0x7efea409d370 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: dump: 2024-09-11T02:54:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 5 monclient: authenticate success, global_id 24011 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 1 -- 172.21.15.98:0/73331106 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8c003e10 con 0x7efea409d370 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _renew_subs 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea4072100 con 0x7efea409d370 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 1 -- 172.21.15.98:0/73331106 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe8c0044a0 con 0x7efea409d370 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 client.24011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea37fe640 4 set_mon_vals no callback set 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 10 monclient: dump: 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _renew_subs 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7efea4128470 con 0x7efea409d370 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 librados: init done 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efea1ffb640 1 -- 172.21.15.98:0/73331106 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7efe8c004640 con 0x7efea409d370 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.189+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efea409ce30 con 0x7efea409d370 2024-09-11T02:54:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea1ffb640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efe8403df40 0x7efe84040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea1ffb640 1 -- 172.21.15.98:0/73331106 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(194..194 src has 1..194) ==== 7848+0+0 (secure 0 0 0) 0x7efe8c050130 con 0x7efea409d370 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea1ffb640 10 client.24011.objecter ms_dispatch 0x7efea4000ba0 osd_map(194..194 src has 1..194) 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea1ffb640 3 client.24011.objecter handle_osd_map got epochs [194,194] > 0 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea1ffb640 3 client.24011.objecter handle_osd_map decoding full epoch 194 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea1ffb640 20 client.24011.objecter dump_active .. 0 homeless 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea921b640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efe8403df40 0x7efe84040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea921b640 10 monclient: get_auth_request con 0x7efe8403df40 auth_method 0 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea1ffb640 10 client.24011.objecter ms_handle_connect 0x7efe8403df40 2024-09-11T02:54:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.193+0000 7efea921b640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efe8403df40 0x7efe84040400 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7efe94009470 tx=0x7efe94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.197+0000 7efea1ffb640 1 -- 172.21.15.98:0/73331106 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7efea4072100 con 0x7efea409d370 2024-09-11T02:54:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.197+0000 7efea1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.197+0000 7efea1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:31.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.429+0000 7efeab47f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:54:31.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.429+0000 7efeab47f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:31.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.429+0000 7efeab47f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:31.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.429+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efea40690d0 con 0x7efea409d370 2024-09-11T02:54:31.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efea1ffb640 1 -- 172.21.15.98:0/73331106 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7efe8c010070 con 0x7efea409d370 2024-09-11T02:54:31.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efea1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:31.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efea1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:31.437 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":42,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 10 librados: watch_flush enter 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 10 librados: watch_flush exit 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 20 client.24011.objecter shutdown clearing up homeless session... 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 10 client.24011.objecter successfully canceled tick 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efe8403df40 msgr2=0x7efe84040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efe8403df40 0x7efe84040400 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7efe94009470 tx=0x7efe94009190 comp rx=0 tx=0).stop 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 10 monclient: shutdown 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efea409d370 msgr2=0x7efea4138a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:31.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efea409d370 0x7efea4138a70 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009fc0 tx=0x7efe8c00d560 comp rx=0 tx=0).stop 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 shutdown_connections 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efe8403df40 0x7efe84040400 unknown :-1 s=CLOSED pgs=3465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efea40a4030 0x7efea4138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efea409d370 0x7efea4138a70 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 --2- 172.21.15.98:0/73331106 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7efea409ca50 0x7efea41d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 >> 172.21.15.98:0/73331106 conn(0x7efea40a4890 msgr2=0x7efea40a6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 shutdown_connections 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 -- 172.21.15.98:0/73331106 wait complete. 2024-09-11T02:54:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:31.433+0000 7efeab47f640 1 librados: shutdown 2024-09-11T02:54:34.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient: get_monmap_and_config 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient: build_initial_monmap 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient: monmap: 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:34.587078+0000 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:34.587078+0000 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:34.586 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:34.587 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 Processor -- start 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 -- start start 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient: init 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient: _add_conns ranks=[3,10,9,5,2,8,0,6,7,1,4] 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff85c139b10 0x7ff85c0725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): picked mon.noname-g con 0x7ff85c139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:34.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff85c06fe30 0x7ff85c139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): picked mon.noname-f con 0x7ff85c06fe30 addr v2:172.21.15.190:3300/0 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff85c0a7a10 0x7ff85c0a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c0a7a10 addr v2:172.21.15.98:3300/0 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff85c0a9650 con 0x7ff85c0a7a10 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff85c071350 con 0x7ff85c06fe30 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff85c071630 con 0x7ff85c139b10 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff8620d6640 10 monclient(hunting): authenticate will time out at 1228.467381s 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85b7fe640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff85c139b10 msgr2=0x7ff85c0725d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:54:34.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85bfff640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7ff85c0a7a10 msgr2=0x7ff85c0a7ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff85c139b10 0x7ff85c0725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff85c0a7a10 0x7ff85c0a7ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff85c06fe30 0x7ff85c139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff85c06fe30 0x7ff85c139ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:59084/0 (socket says 172.21.15.98:59084) 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 1 -- 172.21.15.98:0/3828543179 learned_addr learned my addr 172.21.15.98:0/3828543179 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 10 monclient(hunting): get_auth_request con 0x7ff85c06fe30 auth_method 0 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.581+0000 7ff85affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:34.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 10 monclient(hunting): handle_auth_done global_id 24886 payload 274 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 1 -- 172.21.15.98:0/3828543179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff85c139b10 msgr2=0x7ff85c0725d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 1 --2- 172.21.15.98:0/3828543179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff85c139b10 0x7ff85c0725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 1 -- 172.21.15.98:0/3828543179 >> v2:172.21.15.98:3300/0 conn(0x7ff85c0a7a10 msgr2=0x7ff85c0a7ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 1 --2- 172.21.15.98:0/3828543179 >> v2:172.21.15.98:3300/0 conn(0x7ff85c0a7a10 0x7ff85c0a7ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 10 monclient: _finish_hunting 0 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 1 monclient: found mon.noname-f 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:54:34.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 1 -- 172.21.15.98:0/3828543179 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c0702b0 con 0x7ff85c06fe30 2024-09-11T02:54:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85affd640 1 --2- 172.21.15.98:0/3828543179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff85c06fe30 0x7ff85c139ef0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0a95c0 tx=0x7ff84401c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6a438c57488d5d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 1 -- 172.21.15.98:0/3828543179 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8440b0070 con 0x7ff85c06fe30 2024-09-11T02:54:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:34.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: dump: 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:34.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 5 monclient: authenticate success, global_id 24886 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 1 -- 172.21.15.98:0/3828543179 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8440aa3a0 con 0x7ff85c06fe30 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:34.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 4 set_mon_vals no callback set 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 1 -- 172.21.15.98:0/3828543179 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8440aaa40 con 0x7ff85c06fe30 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff85a7fc640 10 monclient: dump: 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:34.601 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:34.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 10 monclient: shutdown 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 -- 172.21.15.98:0/3828543179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff85c06fe30 msgr2=0x7ff85c139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 --2- 172.21.15.98:0/3828543179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff85c06fe30 0x7ff85c139ef0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0a95c0 tx=0x7ff84401c420 comp rx=0 tx=0).stop 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 -- 172.21.15.98:0/3828543179 shutdown_connections 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 --2- 172.21.15.98:0/3828543179 >> v2:172.21.15.98:3300/0 conn(0x7ff85c0a7a10 0x7ff85c0a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 --2- 172.21.15.98:0/3828543179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff85c06fe30 0x7ff85c139ef0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 --2- 172.21.15.98:0/3828543179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff85c139b10 0x7ff85c0725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 -- 172.21.15.98:0/3828543179 >> 172.21.15.98:0/3828543179 conn(0x7ff85c0ac740 msgr2=0x7ff85c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 -- 172.21.15.98:0/3828543179 shutdown_connections 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.585+0000 7ff8620d6640 1 -- 172.21.15.98:0/3828543179 wait complete. 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: build_initial_monmap 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: monmap: 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:34.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 librados: starting msgr at 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 librados: starting objecter 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 Processor -- start 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 -- start start 2024-09-11T02:54:34.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 librados: setting wanted keys 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 librados: calling monclient init 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: init 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _add_conns ranks=[6,8,0,7,4,5,3,2,1] 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff85c0a7a10 0x7ff85c13f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): picked mon.noname-d con 0x7ff85c0a7a10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff85c139b10 0x7ff85c0a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): picked mon.noname-e con 0x7ff85c139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff85c13f6c0 0x7ff85c0a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c13f6c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:34.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85b7fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff85c0a7a10 msgr2=0x7ff85c13f180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1435c0 con 0x7ff85c13f6c0 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff85c143440 con 0x7ff85c0a7a10 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff85c0a7a10 0x7ff85c13f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff85c143740 con 0x7ff85c139b10 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient(hunting): authenticate will time out at 1228.473392s 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85bfff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff85c13f6c0 msgr2=0x7ff85c0a4bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:34.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff85c13f6c0 0x7ff85c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff85c139b10 0x7ff85c0a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff85c139b10 0x7ff85c0a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:37286/0 (socket says 172.21.15.98:37286) 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 -- 172.21.15.98:0/2135099608 learned_addr learned my addr 172.21.15.98:0/2135099608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): get_auth_request con 0x7ff85c139b10 auth_method 0 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient(hunting): handle_auth_done global_id 24020 payload 824 2024-09-11T02:54:34.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 -- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff85c0a7a10 msgr2=0x7ff85c13f180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff85c0a7a10 0x7ff85c13f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 -- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff85c13f6c0 msgr2=0x7ff85c0a4bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff85c13f6c0 0x7ff85c0a4bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient: _finish_hunting 0 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 monclient: found mon.noname-e 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 -- 172.21.15.98:0/2135099608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c13c980 con 0x7ff85c139b10 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85affd640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff85c139b10 0x7ff85c0a4680 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff84401ccc0 tx=0x7ff8440191c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 client.?.objecter ms_handle_connect 0x7ff85c139b10 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 1 -- 172.21.15.98:0/2135099608 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8440b0040 con 0x7ff85c139b10 2024-09-11T02:54:34.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: dump: 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:34.610 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:34.611 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:34.611 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:34.611 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:34.611 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:34.612 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:34.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: _finish_auth 0 2024-09-11T02:54:34.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:34.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:34.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 1 -- 172.21.15.98:0/2135099608 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff844003e40 con 0x7ff85c139b10 2024-09-11T02:54:34.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 1 -- 172.21.15.98:0/2135099608 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff8440aa9c0 con 0x7ff85c139b10 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 monclient: dump: 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:34.613 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:34.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8597fa640 4 set_mon_vals no callback set 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 5 monclient: authenticate success, global_id 24020 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _renew_subs 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85c135a60 con 0x7ff85c139b10 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 client.24020.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _renew_subs 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85c13c6a0 con 0x7ff85c139b10 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 librados: init done 2024-09-11T02:54:34.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff85c0a7ea0 con 0x7ff85c139b10 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 1 -- 172.21.15.98:0/2135099608 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ff8440b3070 con 0x7ff85c139b10 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff83c03dfa0 0x7ff83c040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 1 -- 172.21.15.98:0/2135099608 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(194..194 src has 1..194) ==== 7848+0+0 (secure 0 0 0) 0x7ff8440b9070 con 0x7ff85c139b10 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 10 client.24020.objecter ms_dispatch 0x7ff85c000ba0 osd_map(194..194 src has 1..194) 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 3 client.24020.objecter handle_osd_map got epochs [194,194] > 0 2024-09-11T02:54:34.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 3 client.24020.objecter handle_osd_map decoding full epoch 194 2024-09-11T02:54:34.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff853fff640 20 client.24020.objecter dump_active .. 0 homeless 2024-09-11T02:54:34.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85b7fe640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff83c03dfa0 0x7ff83c040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:34.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.589+0000 7ff85b7fe640 10 monclient: get_auth_request con 0x7ff83c03dfa0 auth_method 0 2024-09-11T02:54:34.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.593+0000 7ff853fff640 10 client.24020.objecter ms_handle_connect 0x7ff83c03dfa0 2024-09-11T02:54:34.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.593+0000 7ff85b7fe640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff83c03dfa0 0x7ff83c040460 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7ff84c003cc0 tx=0x7ff84c009270 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:34.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.597+0000 7ff853fff640 1 -- 172.21.15.98:0/2135099608 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7ff844016130 con 0x7ff85c139b10 2024-09-11T02:54:34.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.597+0000 7ff853fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:34.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.597+0000 7ff853fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:34.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.821+0000 7ff8620d6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:54:34.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.821+0000 7ff8620d6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:34.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.821+0000 7ff8620d6640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:34.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.821+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff85c13afe0 con 0x7ff85c139b10 2024-09-11T02:54:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.821+0000 7ff853fff640 1 -- 172.21.15.98:0/2135099608 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7ff8440aa7e0 con 0x7ff85c139b10 2024-09-11T02:54:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.821+0000 7ff853fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:34.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.821+0000 7ff853fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:34.827 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":42,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:54:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 10 librados: watch_flush enter 2024-09-11T02:54:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 10 librados: watch_flush exit 2024-09-11T02:54:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 20 client.24020.objecter shutdown clearing up homeless session... 2024-09-11T02:54:34.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 10 client.24020.objecter successfully canceled tick 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff83c03dfa0 msgr2=0x7ff83c040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff83c03dfa0 0x7ff83c040460 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7ff84c003cc0 tx=0x7ff84c009270 comp rx=0 tx=0).stop 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 10 monclient: shutdown 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff85c139b10 msgr2=0x7ff85c0a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff85c139b10 0x7ff85c0a4680 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff84401ccc0 tx=0x7ff8440191c0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 shutdown_connections 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff83c03dfa0 0x7ff83c040460 unknown :-1 s=CLOSED pgs=3466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff85c13f6c0 0x7ff85c0a4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff85c139b10 0x7ff85c0a4680 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 --2- 172.21.15.98:0/2135099608 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff85c0a7a10 0x7ff85c13f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 >> 172.21.15.98:0/2135099608 conn(0x7ff85c0ac740 msgr2=0x7ff85c0a9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 shutdown_connections 2024-09-11T02:54:34.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 -- 172.21.15.98:0/2135099608 wait complete. 2024-09-11T02:54:34.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:34.825+0000 7ff8620d6640 1 librados: shutdown 2024-09-11T02:54:37.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient: get_monmap_and_config 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient: build_initial_monmap 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient: monmap: 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:37.975360+0000 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:37.975360+0000 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:37.974 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:37.975 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 1 Processor -- start 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 1 -- start start 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient: init 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient: _add_conns ranks=[0,9,2,6,10,8,1,3,7,4,5] 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bfc06f650 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 0x7f4bfc072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient(hunting): picked mon.noname-a con 0x7f4bfc0a8380 addr v2:172.21.15.98:3300/0 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 0x7f4bfc140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:37.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient(hunting): picked mon.noname-b con 0x7f4bfc070140 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc071a80 con 0x7f4bfc06f650 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4bfc0a81b0 con 0x7f4bfc0a8380 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0a8800 con 0x7f4bfc070140 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c02922640 10 monclient(hunting): authenticate will time out at 1231.855620s 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4bfb7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 msgr2=0x7f4bfc072640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4bfbfff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4bfc06f650 legacy=0x7f4bfc06fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c00ebf640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 msgr2=0x7f4bfc140c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4bfb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 0x7f4bfc072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:37.969+0000 7f4c00ebf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 0x7f4bfc140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.173+0000 7f4bfb7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 msgr2=0x7f4bfc072640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.173+0000 7f4c00ebf640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 msgr2=0x7f4bfc140c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:54:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.173+0000 7f4bfbfff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4bfc06f650 legacy=0x7f4bfc06fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:54:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.173+0000 7f4bfb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 0x7f4bfc072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:54:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.173+0000 7f4c00ebf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 0x7f4bfc140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:54:38.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.577+0000 7f4bfbfff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4bfc06f650 legacy=0x7f4bfc06fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:54:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.577+0000 7f4c00ebf640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 msgr2=0x7f4bfc140c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:54:38.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.577+0000 7f4c00ebf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 0x7f4bfc140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:54:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.577+0000 7f4bfb7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 msgr2=0x7f4bfc072640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:38.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:38.577+0000 7f4bfb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 0x7f4bfc072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:54:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:39.381+0000 7f4bfb7fe640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 msgr2=0x7f4bfc072640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:39.381+0000 7f4bfb7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 0x7f4bfc072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:54:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:39.381+0000 7f4c00ebf640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 msgr2=0x7f4bfc140c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:54:39.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:39.381+0000 7f4bfbfff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4bfc06f650 legacy=0x7f4bfc06fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:54:39.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:39.381+0000 7f4c00ebf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 0x7f4bfc140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:54:40.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): tick 2024-09-11T02:54:40.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:54:40.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 monclient(hunting): continuing hunt 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 msgr2=0x7f4bfc140c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4bfc070140 0x7f4bfc140c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 msgr2=0x7f4bfc072640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4bfc0a8380 0x7f4bfc072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f4bfc06f650 legacy=0x7f4bfc06fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient: _add_conns ranks=[7,6,1,5,8,4,3,10] 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): picked mon.noname-i con 0x7f4be8003800 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 0x7f4be80086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): picked mon.noname-d con 0x7f4be8006250 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:40.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f4be8008db0 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc071d60 con 0x7f4be8008db0 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4bfc0a8800 con 0x7f4be8006250 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4be800dbf0 con 0x7f4be8003800 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.969+0000 7f4bf9ffb640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfb7fe640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 msgr2=0x7f4be80086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfb7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 0x7f4be80086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4c00ebf640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f4be8008db0 0x7f4be800b270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:45504/0 (socket says 172.21.15.98:45504) 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4c00ebf640 1 -- 172.21.15.98:0/1745064272 learned_addr learned my addr 172.21.15.98:0/1745064272 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:57180/0 (socket says 172.21.15.98:57180) 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request con 0x7f4be8003800 auth_method 0 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:40.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 msgr2=0x7f4be8005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:54:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 msgr2=0x7f4be8005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:54:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:54:40.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:40.973+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request con 0x7f4be8003800 auth_method 0 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfb7fe640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 msgr2=0x7f4be80086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfb7fe640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 0x7f4be80086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 msgr2=0x7f4be8005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:54:41.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 msgr2=0x7f4be8005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:54:41.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:54:41.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.177+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:54:41.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfb7fe640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 msgr2=0x7f4be80086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:54:41.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfb7fe640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 0x7f4be80086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:54:41.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:41.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request con 0x7f4be8003800 auth_method 0 2024-09-11T02:54:41.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:41.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:41.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:41.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 msgr2=0x7f4be8005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:54:41.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 msgr2=0x7f4be8005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:54:41.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:54:41.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:41.581+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:54:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.221+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284088515 0 0) 0x7f4bfc071d60 con 0x7f4be8008db0 2024-09-11T02:54:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.221+0000 7f4bfaffd640 10 monclient(hunting): _negotiate 2024-09-11T02:54:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.221+0000 7f4bfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.221+0000 7f4bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.221+0000 7f4bfaffd640 10 monclient(hunting): my global_id is 24895 2024-09-11T02:54:42.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.221+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be4003530 con 0x7f4be8008db0 2024-09-11T02:54:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2623740354 0 0) 0x7f4be4003530 con 0x7f4be8008db0 2024-09-11T02:54:42.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 msgr2=0x7f4be8005c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4be8003800 0x7f4be8005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 msgr2=0x7f4be80086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4be8006250 0x7f4be80086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: _finish_hunting 0 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 monclient: found mon.noname-f-legacy 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 -- 172.21.15.98:0/1745064272 reap_dead start 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.190:6789/0 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc071a80 con 0x7f4be8008db0 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:42.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f4bf4004610 con 0x7f4be8008db0 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: dump: 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.231 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] but i'm connected to v1:172.21.15.190:6789/0, reconnecting 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 >> v1:172.21.15.190:6789/0 conn(0x7f4be8008db0 legacy=0x7f4be800b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient: _add_conns ranks=[3,4,8,1,5,0,6,2,7] 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4be4004210 0x7f4be4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): picked mon.g con 0x7f4be4004210 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004e20 0x7f4be40052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): picked mon.c con 0x7f4be4004e20 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4be4009ce0 0x7f4be400c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): picked mon.e con 0x7f4be4009ce0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0a8800 con 0x7f4be4004210 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4be800dbf0 con 0x7f4be4004e20 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4be400ca80 con 0x7f4be4009ce0 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfaffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:54:42.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfbfff640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4be4004210 msgr2=0x7f4be4004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4be4009ce0 0x7f4be400c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): get_auth_request con 0x7f4be4009ce0 auth_method 0 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:54:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfb7fe640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004e20 0x7f4be40052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfb7fe640 10 monclient(hunting): get_auth_request con 0x7f4be4004e20 auth_method 0 2024-09-11T02:54:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient(hunting): handle_auth_done global_id 24895 payload 293 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004e20 msgr2=0x7f4be40052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004e20 0x7f4be40052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4be4004210 msgr2=0x7f4be4004810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4be4004210 0x7f4be4004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4be4004210 0x7f4be4004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient: _finish_hunting 0 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 monclient: found mon.e 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:54:42.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 1 -- 172.21.15.98:0/1745064272 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be4003530 con 0x7f4be4009ce0 2024-09-11T02:54:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient: _finish_auth 0 2024-09-11T02:54:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 10 monclient: _check_auth_tickets 2024-09-11T02:54:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c00ebf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c02922640 5 monclient: authenticate success, global_id 24895 2024-09-11T02:54:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.225+0000 7f4c02922640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:42.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.233+0000 7f4c00ebf640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4be4009ce0 0x7f4be400c170 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0acfc0 tx=0x7f4bf401c7f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b6fa73ec0dba0f19 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.233+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bf4007b70 con 0x7f4be4009ce0 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.233+0000 7f4bfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.233+0000 7f4bfaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.233+0000 7f4bfaffd640 10 monclient: dump: 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.239 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.233+0000 7f4c02922640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf4007030 con 0x7f4be4009ce0 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4bfaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 4 set_mon_vals no callback set 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4bfaffd640 1 -- 172.21.15.98:0/1745064272 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bf40047b0 con 0x7f4be4009ce0 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4bfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4bfaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4bfaffd640 10 monclient: dump: 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.242 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.243 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 10 monclient: shutdown 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 -- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4be4009ce0 msgr2=0x7f4be400c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4be4009ce0 0x7f4be400c170 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc0acfc0 tx=0x7f4bf401c7f0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 -- 172.21.15.98:0/1745064272 shutdown_connections 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4be4009ce0 0x7f4be400c170 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004e20 0x7f4be40052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 --2- 172.21.15.98:0/1745064272 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4be4004210 0x7f4be4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 -- 172.21.15.98:0/1745064272 >> 172.21.15.98:0/1745064272 conn(0x7f4bfc0ac8e0 msgr2=0x7f4bfc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 -- 172.21.15.98:0/1745064272 shutdown_connections 2024-09-11T02:54:42.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 -- 172.21.15.98:0/1745064272 wait complete. 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 10 monclient: build_initial_monmap 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 10 monclient: monmap: 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:42.249 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 librados: starting msgr at 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 librados: starting objecter 2024-09-11T02:54:42.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 Processor -- start 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 -- start start 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 librados: setting wanted keys 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 1 librados: calling monclient init 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.237+0000 7f4c02922640 10 monclient: init 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _add_conns ranks=[4,5,1,6,2,7,8,0,3] 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004210 0x7f4bfc138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): picked mon.noname-c con 0x7f4be4004210 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4be4004e20 0x7f4bfc138e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): picked mon.noname-h con 0x7f4be4004e20 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4be4009ce0 0x7f4bfc122650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): picked mon.noname-f con 0x7f4be4009ce0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:54:42.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc13a360 con 0x7f4be4009ce0 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4bfc13a1e0 con 0x7f4be4004210 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f4bfc13a4e0 con 0x7f4be4004e20 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient(hunting): authenticate will time out at 1236.124871s 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004210 0x7f4bfc138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004210 0x7f4bfc138910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:60444/0 (socket says 172.21.15.98:60444) 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 -- 172.21.15.98:0/2494347934 learned_addr learned my addr 172.21.15.98:0/2494347934 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request con 0x7f4be4004210 auth_method 0 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfb7fe640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4be4004e20 0x7f4bfc138e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfb7fe640 10 monclient(hunting): get_auth_request con 0x7f4be4004e20 auth_method 0 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c00ebf640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4be4009ce0 0x7f4bfc122650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient(hunting): handle_auth_done global_id 23503 payload 824 2024-09-11T02:54:42.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 -- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4be4004e20 msgr2=0x7f4bfc138e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4be4004e20 0x7f4bfc138e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 -- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4be4009ce0 msgr2=0x7f4bfc122650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4be4009ce0 0x7f4bfc122650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient: _finish_hunting 0 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 monclient: found mon.noname-c 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 -- 172.21.15.98:0/2494347934 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc0a5820 con 0x7f4be4004210 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c00ebf640 10 monclient: get_auth_request con 0x7f4be4009ce0 auth_method 0 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c00ebf640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4be4009ce0 0x7f4bfc122650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfbfff640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004210 0x7f4bfc138910 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000ea60 tx=0x7f4bf000eb50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 client.?.objecter ms_handle_connect 0x7f4be4004210 2024-09-11T02:54:42.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 1 -- 172.21.15.98:0/2494347934 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bf001c070 con 0x7f4be4004210 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: dump: 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.255 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: _finish_auth 0 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 1 -- 172.21.15.98:0/2494347934 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf000be60 con 0x7f4be4004210 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:42.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 1 -- 172.21.15.98:0/2494347934 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4bf000e0d0 con 0x7f4be4004210 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 monclient: dump: 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.257 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 5 monclient: authenticate success, global_id 23503 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _renew_subs 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfa7fc640 4 set_mon_vals no callback set 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc126010 con 0x7f4be4004210 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 client.23503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:42.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _renew_subs 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc1d14e0 con 0x7f4be4004210 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 librados: init done 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 1 -- 172.21.15.98:0/2494347934 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4bf000b9f0 con 0x7f4be4004210 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4be803ec70 0x7f4be8041130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 1 -- 172.21.15.98:0/2494347934 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(195..195 src has 1..195) ==== 7028+0+0 (secure 0 0 0) 0x7f4bf0014070 con 0x7f4be4004210 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 10 client.23503.objecter ms_dispatch 0x7f4bfc000ba0 osd_map(195..195 src has 1..195) 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 3 client.23503.objecter handle_osd_map got epochs [195,195] > 0 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 3 client.23503.objecter handle_osd_map decoding full epoch 195 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bfc125e10 con 0x7f4be4004210 2024-09-11T02:54:42.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bf8ff9640 20 client.23503.objecter dump_active .. 0 homeless 2024-09-11T02:54:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfb7fe640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4be803ec70 0x7f4be8041130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.241+0000 7f4bfb7fe640 10 monclient: get_auth_request con 0x7f4be803ec70 auth_method 0 2024-09-11T02:54:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.245+0000 7f4bf8ff9640 10 client.23503.objecter ms_handle_connect 0x7f4be803ec70 2024-09-11T02:54:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.245+0000 7f4bfb7fe640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4be803ec70 0x7f4be8041130 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f4bec003a50 tx=0x7f4bec0039c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.249+0000 7f4bf8ff9640 1 -- 172.21.15.98:0/2494347934 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f4bfc0a5820 con 0x7f4be4004210 2024-09-11T02:54:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.249+0000 7f4bf8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:42.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.249+0000 7f4bf8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.481+0000 7f4c02922640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:54:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.481+0000 7f4c02922640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.481+0000 7f4c02922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.481+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4bfc13cc00 con 0x7f4be4004210 2024-09-11T02:54:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4bf8ff9640 1 -- 172.21.15.98:0/2494347934 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3168 (secure 0 0 0) 0x7f4bf000cc20 con 0x7f4be4004210 2024-09-11T02:54:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4bf8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:54:42.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4bf8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:42.488 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":44,"quorum":[1,4,5,7,8],"quorum_names":["f","c","h","i","e"],"quorum_leader_name":"e","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:54:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 10 librados: watch_flush enter 2024-09-11T02:54:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 10 librados: watch_flush exit 2024-09-11T02:54:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 20 client.23503.objecter shutdown clearing up homeless session... 2024-09-11T02:54:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 10 client.23503.objecter successfully canceled tick 2024-09-11T02:54:42.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4be803ec70 msgr2=0x7f4be8041130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4be803ec70 0x7f4be8041130 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f4bec003a50 tx=0x7f4bec0039c0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 10 monclient: shutdown 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004210 msgr2=0x7f4bfc138910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004210 0x7f4bfc138910 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000ea60 tx=0x7f4bf000eb50 comp rx=0 tx=0).stop 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 shutdown_connections 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4be803ec70 0x7f4be8041130 unknown :-1 s=CLOSED pgs=3467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4be4009ce0 0x7f4bfc122650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4be4004e20 0x7f4bfc138e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 --2- 172.21.15.98:0/2494347934 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4be4004210 0x7f4bfc138910 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 >> 172.21.15.98:0/2494347934 conn(0x7f4bfc0ac8e0 msgr2=0x7f4bfc0aa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 shutdown_connections 2024-09-11T02:54:42.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 -- 172.21.15.98:0/2494347934 wait complete. 2024-09-11T02:54:42.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.485+0000 7f4c02922640 1 librados: shutdown 2024-09-11T02:54:42.508 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-09-11T02:54:42.508 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-09-11T02:54:42.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient: get_monmap_and_config 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient: build_initial_monmap 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient: monmap: 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:42.633322+0000 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:42.633322+0000 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.632 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:42.633 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 1 Processor -- start 2024-09-11T02:54:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 1 -- start start 2024-09-11T02:54:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient: init 2024-09-11T02:54:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient: _add_conns ranks=[6,10,0,4,8,3,9,2,1,7,5] 2024-09-11T02:54:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3c98139b80 0x7f3c98139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient(hunting): picked mon.noname-d con 0x7f3c98139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3c9806fe50 0x7f3c980a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient(hunting): picked mon.noname-f con 0x7f3c9806fe50 addr v2:172.21.15.190:3300/0 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c98072640 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c98071370 con 0x7f3c98072640 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3c980719c0 con 0x7f3c9806fe50 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3c98071c10 con 0x7f3c98139b80 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9ce9b640 10 monclient(hunting): authenticate will time out at 1236.513570s 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9659c640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3c98139b80 msgr2=0x7f3c98139f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:54:42.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c96d9d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f3c98072640 legacy=0x7f3c980a7f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c9659c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3c98139b80 0x7f3c98139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3c9806fe50 0x7f3c980a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3c9806fe50 0x7f3c980a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:59518/0 (socket says 172.21.15.98:59518) 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 -- 172.21.15.98:0/2218841235 learned_addr learned my addr 172.21.15.98:0/2218841235 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): get_auth_request con 0x7f3c9806fe50 auth_method 0 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_done global_id 24904 payload 274 2024-09-11T02:54:42.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 -- 172.21.15.98:0/2218841235 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3c98139b80 msgr2=0x7f3c98139f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:42.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 --2- 172.21.15.98:0/2218841235 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3c98139b80 0x7f3c98139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 -- 172.21.15.98:0/2218841235 >> v1:172.21.15.98:6789/0 conn(0x7f3c98072640 legacy=0x7f3c980a7f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:42.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:54:42.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 monclient: found mon.noname-f 2024-09-11T02:54:42.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:42.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:54:42.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.629+0000 7f3c95d9b640 1 -- 172.21.15.98:0/2218841235 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c980a9600 con 0x7f3c9806fe50 2024-09-11T02:54:42.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c95d9b640 1 --2- 172.21.15.98:0/2218841235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c9806fe50 0x7f3c980a7a10 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009870 tx=0x7f3c8c01c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eedc3bd9be5304d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 1 -- 172.21.15.98:0/2218841235 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3c8c0b0070 con 0x7f3c9806fe50 2024-09-11T02:54:42.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: dump: 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.647 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: _finish_auth 0 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: _check_auth_tickets 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 1 -- 172.21.15.98:0/2218841235 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c8c0aa810 con 0x7f3c9806fe50 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 1 -- 172.21.15.98:0/2218841235 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3c8c0a9300 con 0x7f3c9806fe50 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9559a640 10 monclient: dump: 2024-09-11T02:54:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 5 monclient: authenticate success, global_id 24904 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 4 set_mon_vals no callback set 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 10 monclient: shutdown 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/2218841235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c9806fe50 msgr2=0x7f3c980a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 --2- 172.21.15.98:0/2218841235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c9806fe50 0x7f3c980a7a10 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009870 tx=0x7f3c8c01c440 comp rx=0 tx=0).stop 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/2218841235 shutdown_connections 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 --2- 172.21.15.98:0/2218841235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3c9806fe50 0x7f3c980a7a10 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 --2- 172.21.15.98:0/2218841235 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3c98139b80 0x7f3c98139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/2218841235 >> 172.21.15.98:0/2218841235 conn(0x7f3c980ac730 msgr2=0x7f3c980a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/2218841235 shutdown_connections 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/2218841235 wait complete. 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 10 monclient: build_initial_monmap 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 10 monclient: monmap: 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 librados: starting msgr at 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 librados: starting objecter 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 Processor -- start 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 -- start start 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 librados: setting wanted keys 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 1 librados: calling monclient init 2024-09-11T02:54:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.633+0000 7f3c9ce9b640 10 monclient: init 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient: _add_conns ranks=[5,4,7,1,3,6,0,2,8] 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3c9806fe50 0x7f3c981329b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): picked mon.noname-h con 0x7f3c9806fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c98072640 0x7f3c98132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): picked mon.noname-c con 0x7f3c98072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3c98139b80 0x7f3c980a3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): picked mon.noname-i con 0x7f3c98139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3c98144450 con 0x7f3c98072640 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3c980a8870 con 0x7f3c9806fe50 2024-09-11T02:54:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3c981445d0 con 0x7f3c98139b80 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): authenticate will time out at 1236.520357s 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c98072640 0x7f3c98132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c98072640 0x7f3c98132ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:60452/0 (socket says 172.21.15.98:60452) 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 -- 172.21.15.98:0/3684539101 learned_addr learned my addr 172.21.15.98:0/3684539101 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): get_auth_request con 0x7f3c98072640 auth_method 0 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3c9806fe50 0x7f3c981329b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3c98139b80 0x7f3c980a3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): get_auth_request con 0x7f3c9806fe50 auth_method 0 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 10 monclient(hunting): get_auth_request con 0x7f3c98139b80 auth_method 0 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient(hunting): handle_auth_done global_id 23512 payload 824 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3c98139b80 msgr2=0x7f3c980a3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3c98139b80 0x7f3c980a3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3c9806fe50 msgr2=0x7f3c981329b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3c9806fe50 0x7f3c981329b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 monclient: found mon.noname-c 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 -- 172.21.15.98:0/3684539101 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c980a9f00 con 0x7f3c98072640 2024-09-11T02:54:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3c9806fe50 0x7f3c981329b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c95d9b640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c98072640 0x7f3c98132ef0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c01cce0 tx=0x7f3c8c0191e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 client.?.objecter ms_handle_connect 0x7f3c98072640 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3c8c0b9070 con 0x7f3c98072640 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: dump: 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.657 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: _finish_auth 0 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:54:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c8c0044a0 con 0x7f3c98072640 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 5 monclient: authenticate success, global_id 23512 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient: _renew_subs 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c9813c230 con 0x7f3c98072640 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3c8c0b0040 con 0x7f3c98072640 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c94d99640 4 set_mon_vals no callback set 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: dump: 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.659 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.660 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 client.23512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient: _renew_subs 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c980a5620 con 0x7f3c98072640 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 librados: init done 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c9813aa30 0x7f3c980a4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3c9806ca10 con 0x7f3c9813aa30 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c9813aa30 0x7f3c980a4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient: get_auth_request con 0x7f3c9813aa30 auth_method 0 2024-09-11T02:54:42.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3c8c0b3070 con 0x7f3c98072640 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 10 monclient(hunting): handle_auth_done global_id 23521 payload 824 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 1 -- 172.21.15.98:0/3684539101 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c98139f60 con 0x7f3c9813aa30 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c9659c640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c9813aa30 0x7f3c980a4200 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800d700 tx=0x7f3c8800dbc0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3c7803df40 0x7f3c78040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 client.23512.objecter ms_handle_connect 0x7f3c9813aa30 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 client.23512.objecter resend_mon_ops 2024-09-11T02:54:42.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(195..195 src has 1..195) ==== 7028+0+0 (secure 0 0 0) 0x7f3c8c0e8a50 con 0x7f3c98072640 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 client.23512.objecter ms_dispatch 0x7f3c98000ba0 osd_map(195..195 src has 1..195) 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 3 client.23512.objecter handle_osd_map got epochs [195,195] > 0 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 3 client.23512.objecter handle_osd_map decoding full epoch 195 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 20 client.23512.objecter dump_active .. 0 homeless 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3c880039c0 con 0x7f3c9813aa30 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: dump: 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3c88005a50 con 0x7f3c9813aa30 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:42.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c9813aa30 msgr2=0x7f3c980a4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c9813aa30 0x7f3c980a4200 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f3c8800d700 tx=0x7f3c8800dbc0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3c7803df40 0x7f3c78040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c9813aa30 msgr2=0x7f3c980a4200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c9813aa30 0x7f3c980a4200 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 10 monclient: get_auth_request con 0x7f3c7803df40 auth_method 0 2024-09-11T02:54:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c96d9d640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3c7803df40 0x7f3c78040400 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f3c980a5b00 tx=0x7f3c900050f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.637+0000 7f3c86ffd640 10 client.23512.objecter ms_handle_connect 0x7f3c7803df40 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c980a4740 0x7f3c980a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3c9806e870 con 0x7f3c980a4740 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c980a4740 0x7f3c980a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient: get_auth_request con 0x7f3c980a4740 auth_method 0 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 10 monclient(hunting): handle_auth_done global_id 23530 payload 824 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 1 -- 172.21.15.98:0/3684539101 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c980a5090 con 0x7f3c980a4740 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 10 client.23512.objecter ms_handle_connect 0x7f3c980a4740 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 10 client.23512.objecter resend_mon_ops 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9659c640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c980a4740 0x7f3c980a4b50 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f3c88015000 tx=0x7f3c880059e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3c8800b730 con 0x7f3c980a4740 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:42.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 10 monclient: dump: 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9390 (secure 0 0 0) 0x7f3c8801ba30 con 0x7f3c980a4740 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c980a4740 msgr2=0x7f3c980a4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c980a4740 0x7f3c980a4b50 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f3c88015000 tx=0x7f3c880059e0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c980a4740 msgr2=0x7f3c980a4b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:42.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c86ffd640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c980a4740 0x7f3c980a4b50 secure :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f3c88015000 tx=0x7f3c880059e0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 44, 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 298429, 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:42.671 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:42.672 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:42.673 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:42.674 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:42.675 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.676 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:42.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:42.683 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:42.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:42.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:42.690 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.692 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:42.693 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:42.694 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 10 librados: watch_flush enter 2024-09-11T02:54:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 10 librados: watch_flush exit 2024-09-11T02:54:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 20 client.23512.objecter shutdown clearing up homeless session... 2024-09-11T02:54:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 10 client.23512.objecter successfully canceled tick 2024-09-11T02:54:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3c7803df40 msgr2=0x7f3c78040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3c7803df40 0x7f3c78040400 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f3c980a5b00 tx=0x7f3c900050f0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 10 monclient: shutdown 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c98072640 msgr2=0x7f3c98132ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 --2- 172.21.15.98:0/3684539101 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3c98072640 0x7f3c98132ef0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c01cce0 tx=0x7f3c8c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c96d9d640 1 -- 172.21.15.98:0/3684539101 reap_dead start 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c96d9d640 1 -- 172.21.15.98:0/3684539101 reap_dead start 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 shutdown_connections 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 >> 172.21.15.98:0/3684539101 conn(0x7f3c980ac730 msgr2=0x7f3c980aa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 shutdown_connections 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 -- 172.21.15.98:0/3684539101 wait complete. 2024-09-11T02:54:42.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.645+0000 7f3c9ce9b640 1 librados: shutdown 2024-09-11T02:54:42.698 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-11T02:54:42.698 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon e 2024-09-11T02:54:42.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:54:42.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient: get_monmap_and_config 2024-09-11T02:54:42.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient: build_initial_monmap 2024-09-11T02:54:42.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient: monmap: 2024-09-11T02:54:42.813 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:42.813 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.813 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:42.813905+0000 2024-09-11T02:54:42.813 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:42.813905+0000 2024-09-11T02:54:42.814 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:42.814 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.814 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:42.814 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:42.814 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:42.814 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:42.815 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:42.815 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:42.815 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:42.815 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:42.815 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:42.815 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:42.816 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:42.816 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 1 Processor -- start 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 1 -- start start 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient: init 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient: _add_conns ranks=[4,1,0,5,3,2,8,6,9,7,10] 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63280a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient(hunting): picked mon.noname-c con 0x7f63280a3690 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f63280a1eb0 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63280a3fb0 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63280a1ca0 con 0x7f63280a3fb0 2024-09-11T02:54:42.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6328138280 con 0x7f63280a1eb0 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f632809ec40 con 0x7f63280a3690 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632f74f640 10 monclient(hunting): authenticate will time out at 1236.694148s 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632dcec640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f63280a3fb0 legacy=0x7f6328137b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63280a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63280a3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:60484/0 (socket says 172.21.15.98:60484) 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 1 -- 172.21.15.98:0/3303739312 learned_addr learned my addr 172.21.15.98:0/3303739312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): get_auth_request con 0x7f63280a3690 auth_method 0 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient(hunting): handle_auth_done global_id 23539 payload 274 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 1 -- 172.21.15.98:0/3303739312 >> v1:172.21.15.190:6789/0 conn(0x7f63280a1eb0 legacy=0x7f6328137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 1 -- 172.21.15.98:0/3303739312 >> v1:172.21.15.98:6789/0 conn(0x7f63280a3fb0 legacy=0x7f6328137b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient: _finish_hunting 0 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 1 monclient: found mon.noname-c 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.809+0000 7f632d4eb640 1 -- 172.21.15.98:0/3303739312 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f632809dd20 con 0x7f63280a3690 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632d4eb640 1 --2- 172.21.15.98:0/3303739312 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63280a3a70 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f63240098a0 tx=0x7f632401c260 comp rx=0 tx=0).ready entity=mon.4 client_cookie=bed3816de46184ba server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 1 -- 172.21.15.98:0/3303739312 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63240b0070 con 0x7f63280a3690 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: dump: 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.827 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: _finish_auth 0 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 1 -- 172.21.15.98:0/3303739312 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63240aa810 con 0x7f63280a3690 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 5 monclient: authenticate success, global_id 23539 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:42.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 1 -- 172.21.15.98:0/3303739312 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63240a9300 con 0x7f63280a3690 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 4 set_mon_vals no callback set 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f631bfff640 10 monclient: dump: 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient: shutdown 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- 172.21.15.98:0/3303739312 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 msgr2=0x7f63280a3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 --2- 172.21.15.98:0/3303739312 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63280a3a70 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f63240098a0 tx=0x7f632401c260 comp rx=0 tx=0).stop 2024-09-11T02:54:42.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- 172.21.15.98:0/3303739312 shutdown_connections 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 --2- 172.21.15.98:0/3303739312 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63280a3a70 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- 172.21.15.98:0/3303739312 >> 172.21.15.98:0/3303739312 conn(0x7f63280a4780 msgr2=0x7f63280a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- 172.21.15.98:0/3303739312 shutdown_connections 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- 172.21.15.98:0/3303739312 wait complete. 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient: build_initial_monmap 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient: monmap: 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:42.831 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 librados: starting msgr at 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 librados: starting objecter 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 Processor -- start 2024-09-11T02:54:42.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- start start 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 librados: setting wanted keys 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 librados: calling monclient init 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient: init 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient: _add_conns ranks=[3,4,7,2,6,8,5,1,0] 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63280a1eb0 0x7f63281cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): picked mon.noname-g con 0x7f63280a1eb0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63281cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): picked mon.noname-c con 0x7f63280a3690 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f63280a3fb0 0x7f63281cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): picked mon.noname-i con 0x7f63280a3fb0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f63280a1a10 con 0x7f63280a1eb0 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f63280a1890 con 0x7f63280a3690 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f6328143510 con 0x7f63280a3fb0 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632f74f640 10 monclient(hunting): authenticate will time out at 1236.699230s 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63281cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63281cf9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:60486/0 (socket says 172.21.15.98:60486) 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 -- 172.21.15.98:0/482955200 learned_addr learned my addr 172.21.15.98:0/482955200 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): get_auth_request con 0x7f63280a3690 auth_method 0 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632d4eb640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63280a1eb0 msgr2=0x7f63281cf480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:54:42.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632d4eb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63280a1eb0 0x7f63281cf480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632dcec640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f63280a3fb0 0x7f63281cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient(hunting): handle_auth_done global_id 23548 payload 824 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f63280a3fb0 msgr2=0x7f63281cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f63280a3fb0 0x7f63281cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63280a1eb0 msgr2=0x7f63281cf480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f63280a1eb0 0x7f63281cf480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient: _finish_hunting 0 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 monclient: found mon.noname-c 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 -- 172.21.15.98:0/482955200 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6328072da0 con 0x7f63280a3690 2024-09-11T02:54:42.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632dcec640 10 monclient: get_auth_request con 0x7f63280a3fb0 auth_method 0 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632dcec640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f63280a3fb0 0x7f63281cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f632ccea640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63281cf9c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c430 tx=0x7f631c00c8f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f6319ffb640 10 client.?.objecter ms_handle_connect 0x7f63280a3690 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.813+0000 7f6319ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f631c017070 con 0x7f63280a3690 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: dump: 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f631c003600 con 0x7f63280a3690 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 5 monclient: authenticate success, global_id 23548 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 10 monclient: _renew_subs 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63280a2f60 con 0x7f63280a3690 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f631c003c60 con 0x7f63280a3690 2024-09-11T02:54:42.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 10 client.23548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f631affd640 4 set_mon_vals no callback set 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: dump: 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.839 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 10 monclient: _renew_subs 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f63281366d0 con 0x7f63280a3690 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 1 librados: init done 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6328137950 0x7f632812c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f632806ca10 con 0x7f6328137950 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6328137950 0x7f632812c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient: get_auth_request con 0x7f6328137950 auth_method 0 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f631c0037f0 con 0x7f63280a3690 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 10 monclient(hunting): handle_auth_done global_id 24029 payload 824 2024-09-11T02:54:42.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 1 -- 172.21.15.98:0/482955200 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6328068b40 con 0x7f6328137950 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f630c03df40 0x7f630c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(195..195 src has 1..195) ==== 7028+0+0 (secure 0 0 0) 0x7f631c04ec30 con 0x7f63280a3690 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 client.23548.objecter ms_dispatch 0x7f6328000ba0 osd_map(195..195 src has 1..195) 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 3 client.23548.objecter handle_osd_map got epochs [195,195] > 0 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 3 client.23548.objecter handle_osd_map decoding full epoch 195 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 20 client.23548.objecter dump_active .. 0 homeless 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 client.23548.objecter ms_handle_connect 0x7f6328137950 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632d4eb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6328137950 0x7f632812c3b0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f63280a1590 tx=0x7f632401aaa0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 client.23548.objecter resend_mon_ops 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63240b0040 con 0x7f6328137950 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: dump: 2024-09-11T02:54:42.842 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.843 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632dcec640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f630c03df40 0x7f630c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632dcec640 10 monclient: get_auth_request con 0x7f630c03df40 auth_method 0 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f63240093d0 con 0x7f6328137950 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6328137950 msgr2=0x7f632812c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6328137950 0x7f632812c3b0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f63280a1590 tx=0x7f632401aaa0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6328137950 msgr2=0x7f632812c3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6328137950 0x7f632812c3b0 secure :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f63280a1590 tx=0x7f632401aaa0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f6319ffb640 10 client.23548.objecter ms_handle_connect 0x7f630c03df40 2024-09-11T02:54:42.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.817+0000 7f632dcec640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f630c03df40 0x7f630c040400 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f6328135280 tx=0x7f6320007320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:42.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f632812c8f0 0x7f632812cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f632812d240 con 0x7f632812c8f0 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f632812c8f0 0x7f632812cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient: get_auth_request con 0x7f632812c8f0 auth_method 0 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 10 monclient(hunting): handle_auth_done global_id 24038 payload 824 2024-09-11T02:54:42.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 1 -- 172.21.15.98:0/482955200 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6328135960 con 0x7f632812c8f0 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 10 client.23548.objecter ms_handle_connect 0x7f632812c8f0 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632d4eb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f632812c8f0 0x7f632812cd00 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f63240098a0 tx=0x7f6324019000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 10 client.23548.objecter resend_mon_ops 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63240b0040 con 0x7f632812c8f0 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 10 monclient: dump: 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:42.850 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9225 (secure 0 0 0) 0x7f63240aa840 con 0x7f632812c8f0 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f632812c8f0 msgr2=0x7f632812cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f632812c8f0 0x7f632812cd00 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f63240098a0 tx=0x7f6324019000 comp rx=0 tx=0).stop 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f632812c8f0 msgr2=0x7f632812cd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f6319ffb640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f632812c8f0 0x7f632812cd00 secure :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f63240098a0 tx=0x7f6324019000 comp rx=0 tx=0).stop 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 44, 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 298617, 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:42.852 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:42.853 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:42.854 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:42.855 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:42.856 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.857 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:42.861 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.863 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:42.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.866 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:42.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.869 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:42.873 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.874 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:42.875 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:42.876 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 10 librados: watch_flush enter 2024-09-11T02:54:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 10 librados: watch_flush exit 2024-09-11T02:54:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 20 client.23548.objecter shutdown clearing up homeless session... 2024-09-11T02:54:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 10 client.23548.objecter successfully canceled tick 2024-09-11T02:54:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f630c03df40 msgr2=0x7f630c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f630c03df40 0x7f630c040400 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f6328135280 tx=0x7f6320007320 comp rx=0 tx=0).stop 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 10 monclient: shutdown 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 msgr2=0x7f63281cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 --2- 172.21.15.98:0/482955200 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f63280a3690 0x7f63281cf9c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c430 tx=0x7f631c00c8f0 comp rx=0 tx=0).stop 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632dcec640 1 -- 172.21.15.98:0/482955200 reap_dead start 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632dcec640 1 -- 172.21.15.98:0/482955200 reap_dead start 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 shutdown_connections 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 >> 172.21.15.98:0/482955200 conn(0x7f63280a4780 msgr2=0x7f63280a6910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 shutdown_connections 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 -- 172.21.15.98:0/482955200 wait complete. 2024-09-11T02:54:42.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:42.825+0000 7f632f74f640 1 librados: shutdown 2024-09-11T02:54:42.880 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-11T02:54:42.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-09-11T02:54:42.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:54:43.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.001+0000 7f4ffbed5640 10 monclient: get_monmap_and_config 2024-09-11T02:54:43.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.001+0000 7f4ffbed5640 10 monclient: build_initial_monmap 2024-09-11T02:54:43.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.001+0000 7f4ffbed5640 10 monclient: monmap: 2024-09-11T02:54:43.007 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:43.008796+0000 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:43.008796+0000 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:43.008 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:43.009 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:43.009 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:43.009 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:43.009 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:43.009 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 Processor -- start 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 -- start start 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient: init 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient: _add_conns ranks=[3,4,8,7,9,1,5,2,0,10,6] 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 0x7f4ff4072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): picked mon.noname-g con 0x7f4ff4139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:43.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff4139ef0 0x7f4ff40a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff4139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4ff413a2d0 0x7f4ff40a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): picked mon.noname-e con 0x7f4ff413a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff40702b0 con 0x7f4ff4139b10 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4ff4071bf0 con 0x7f4ff4139ef0 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4ff4071350 con 0x7f4ff413a2d0 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffbed5640 10 monclient(hunting): authenticate will time out at 1236.889056s 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4ff413a2d0 0x7f4ff40a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff4139ef0 0x7f4ff40a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4ff413a2d0 0x7f4ff40a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:59682/0 (socket says 172.21.15.98:59682) 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9c71640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 msgr2=0x7f4ff4072600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 -- 172.21.15.98:0/3604210729 learned_addr learned my addr 172.21.15.98:0/3604210729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): get_auth_request con 0x7f4ff413a2d0 auth_method 0 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9c71640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 0x7f4ff4072600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 10 monclient(hunting): get_auth_request con 0x7f4ff4139ef0 auth_method 0 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient(hunting): handle_auth_done global_id 24047 payload 274 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 -- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff4139ef0 msgr2=0x7f4ff40a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff4139ef0 0x7f4ff40a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 -- 172.21.15.98:0/3604210729 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 msgr2=0x7f4ff4072600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 0x7f4ff4072600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient: _finish_hunting 0 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 monclient: found mon.noname-e 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:43.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 -- 172.21.15.98:0/3604210729 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff40a9620 con 0x7f4ff413a2d0 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff9470640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff4139ef0 0x7f4ff40a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ffa472640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4ff413a2d0 0x7f4ff40a7f50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009e80 tx=0x7f4fec01c930 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d73ae350595b5408 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff8c6f640 1 -- 172.21.15.98:0/3604210729 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fec0090d0 con 0x7f4ff413a2d0 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.005+0000 7f4ff8c6f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: dump: 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.017 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: _finish_auth 0 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: _check_auth_tickets 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:43.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 5 monclient: authenticate success, global_id 24047 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 1 -- 172.21.15.98:0/3604210729 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fec0a5980 con 0x7f4ff413a2d0 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 4 set_mon_vals no callback set 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 1 -- 172.21.15.98:0/3604210729 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fec0ad3f0 con 0x7f4ff413a2d0 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff8c6f640 10 monclient: dump: 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.019 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: shutdown 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4ff413a2d0 msgr2=0x7f4ff40a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4ff413a2d0 0x7f4ff40a7f50 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009e80 tx=0x7f4fec01c930 comp rx=0 tx=0).stop 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3604210729 shutdown_connections 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4ff413a2d0 0x7f4ff40a7f50 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff4139ef0 0x7f4ff40a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3604210729 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 0x7f4ff4072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3604210729 >> 172.21.15.98:0/3604210729 conn(0x7f4ff40ac790 msgr2=0x7f4ff40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3604210729 shutdown_connections 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3604210729 wait complete. 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: build_initial_monmap 2024-09-11T02:54:43.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: monmap: 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:43.022 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 librados: starting msgr at 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 librados: starting objecter 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 Processor -- start 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- start start 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 librados: setting wanted keys 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 librados: calling monclient init 2024-09-11T02:54:43.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: init 2024-09-11T02:54:43.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: _add_conns ranks=[3,5,4,0,1,8,2,7,6] 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 0x7f4ff41d1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): picked mon.noname-g con 0x7f4ff4139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4ff4139ef0 0x7f4ff41d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): picked mon.noname-h con 0x7f4ff4139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff413a2d0 0x7f4ff413a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff413a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff4143460 con 0x7f4ff4139b10 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4ff40a88a0 con 0x7f4ff413a2d0 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f4ff41435e0 con 0x7f4ff4139ef0 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient(hunting): authenticate will time out at 1236.894217s 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff413a2d0 0x7f4ff413a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff413a2d0 0x7f4ff413a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:60490/0 (socket says 172.21.15.98:60490) 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 -- 172.21.15.98:0/3030657431 learned_addr learned my addr 172.21.15.98:0/3030657431 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff9c71640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 msgr2=0x7f4ff41d1760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): get_auth_request con 0x7f4ff413a2d0 auth_method 0 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff9c71640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 0x7f4ff41d1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff9470640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4ff4139ef0 0x7f4ff41d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient(hunting): handle_auth_done global_id 23566 payload 824 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4ff4139ef0 msgr2=0x7f4ff41d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4ff4139ef0 0x7f4ff41d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 msgr2=0x7f4ff41d1760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4ff4139b10 0x7f4ff41d1760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient: _finish_hunting 0 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 monclient: found mon.noname-c 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 -- 172.21.15.98:0/3030657431 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff4140460 con 0x7f4ff413a2d0 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff9470640 10 monclient: get_auth_request con 0x7f4ff4139ef0 auth_method 0 2024-09-11T02:54:43.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ff9470640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4ff4139ef0 0x7f4ff41d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffa472640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff413a2d0 0x7f4ff413a6b0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009040 tx=0x7f4fec0196d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 client.?.objecter ms_handle_connect 0x7f4ff413a2d0 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fec0bc020 con 0x7f4ff413a2d0 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: dump: 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.035 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: _finish_auth 0 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: _check_auth_tickets 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fec003e10 con 0x7f4ff413a2d0 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:43.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fec0a5e00 con 0x7f4ff413a2d0 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 5 monclient: authenticate success, global_id 23566 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: _renew_subs 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff413dfe0 con 0x7f4ff413a2d0 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe7fff640 4 set_mon_vals no callback set 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4fe67fc640 10 monclient: dump: 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.037 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 client.23566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:43.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: _renew_subs 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff413bcf0 con 0x7f4ff413a2d0 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.009+0000 7f4ffbed5640 1 librados: init done 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ffbed5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413abf0 0x7f4ff413aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff406ca10 con 0x7f4ff413abf0 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4fec0c9060 con 0x7f4ff413a2d0 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4fd803df70 0x7f4fd8040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413abf0 0x7f4ff413aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(195..195 src has 1..195) ==== 7028+0+0 (secure 0 0 0) 0x7f4fec0b3040 con 0x7f4ff413a2d0 2024-09-11T02:54:43.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 client.23566.objecter ms_dispatch 0x7f4ff4000ba0 osd_map(195..195 src has 1..195) 2024-09-11T02:54:43.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 3 client.23566.objecter handle_osd_map got epochs [195,195] > 0 2024-09-11T02:54:43.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 3 client.23566.objecter handle_osd_map decoding full epoch 195 2024-09-11T02:54:43.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient: get_auth_request con 0x7f4ff413abf0 auth_method 0 2024-09-11T02:54:43.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 20 client.23566.objecter dump_active .. 0 homeless 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9470640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4fd803df70 0x7f4fd8040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9470640 10 monclient: get_auth_request con 0x7f4fd803df70 auth_method 0 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_done global_id 24913 payload 824 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 1 -- 172.21.15.98:0/3030657431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ff406e680 con 0x7f4ff413abf0 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 client.23566.objecter ms_handle_connect 0x7f4fd803df70 2024-09-11T02:54:43.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9470640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4fd803df70 0x7f4fd8040430 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f4ff413c2f0 tx=0x7f4ff0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 client.23566.objecter ms_handle_connect 0x7f4ff413abf0 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 client.23566.objecter resend_mon_ops 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4ff9c71640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413abf0 0x7f4ff413aff0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800b450 tx=0x7f4fe800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fe8003bd0 con 0x7f4ff413abf0 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 monclient: dump: 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.042 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.043 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.043 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.043 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.043 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.043 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.044 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.044 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.044 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.044 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.044 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.044 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4fe8015070 con 0x7f4ff413abf0 2024-09-11T02:54:43.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:43.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:43.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413abf0 msgr2=0x7f4ff413aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413abf0 0x7f4ff413aff0 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800b450 tx=0x7f4fe800b910 comp rx=0 tx=0).stop 2024-09-11T02:54:43.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413abf0 msgr2=0x7f4ff413aff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:43.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.013+0000 7f4fe67fc640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413abf0 0x7f4ff413aff0 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:43.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413b530 0x7f4ff40a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff406e870 con 0x7f4ff413b530 2024-09-11T02:54:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413b530 0x7f4ff40a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient: get_auth_request con 0x7f4ff413b530 auth_method 0 2024-09-11T02:54:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 10 monclient(hunting): handle_auth_done global_id 24922 payload 824 2024-09-11T02:54:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 1 -- 172.21.15.98:0/3030657431 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ff40a9ee0 con 0x7f4ff413b530 2024-09-11T02:54:43.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 10 client.23566.objecter ms_handle_connect 0x7f4ff413b530 2024-09-11T02:54:43.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 10 client.23566.objecter resend_mon_ops 2024-09-11T02:54:43.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ff9c71640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413b530 0x7f4ff40a0e60 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800b450 tx=0x7f4fe80059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fe8005580 con 0x7f4ff413b530 2024-09-11T02:54:43.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:54:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 10 monclient: dump: 2024-09-11T02:54:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.050 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.051 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.052 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.052 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9557 (secure 0 0 0) 0x7f4fe8005580 con 0x7f4ff413b530 2024-09-11T02:54:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413b530 msgr2=0x7f4ff40a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413b530 0x7f4ff40a0e60 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800b450 tx=0x7f4fe80059e0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413b530 msgr2=0x7f4ff40a0e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:43.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4fe67fc640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4ff413b530 0x7f4ff40a0e60 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.054 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:43.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:43.055 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:43.055 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:43.055 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 44, 2024-09-11T02:54:43.055 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 298791, 2024-09-11T02:54:43.055 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:43.056 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:43.056 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:43.056 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:43.056 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:43.056 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:43.057 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.057 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T02:54:43.057 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:43.057 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:43.057 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:43.058 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.058 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.058 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.058 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.059 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.059 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.059 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.059 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.059 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.059 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.061 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.061 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.061 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.061 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.061 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.062 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.062 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.062 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.062 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.062 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.063 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:43.063 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:43.063 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:43.063 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:43.063 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:43.064 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:43.064 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:43.064 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:43.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:43.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:43.064 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:43.065 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:43.065 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:43.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:43.065 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:43.065 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:43.065 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:43.066 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.066 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.066 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.066 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.066 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.067 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.067 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.067 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.067 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.067 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.067 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.068 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:43.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.068 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:43.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.068 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:43.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:43.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:43.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:43.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:43.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:43.072 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.072 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.072 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.073 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:43.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:43.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:43.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:43.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.075 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:43.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:43.076 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.076 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.076 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.077 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:43.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:43.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:43.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:43.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.079 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:43.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:43.080 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.080 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.081 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.081 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:43.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:43.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:43.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:43.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.084 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:43.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:43.085 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.085 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.085 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.086 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:43.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:43.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.087 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:43.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.092 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.095 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.097 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:43.097 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:43.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 10 librados: watch_flush enter 2024-09-11T02:54:43.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 10 librados: watch_flush exit 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 20 client.23566.objecter shutdown clearing up homeless session... 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 10 client.23566.objecter successfully canceled tick 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4fd803df70 msgr2=0x7f4fd8040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4fd803df70 0x7f4fd8040430 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f4ff413c2f0 tx=0x7f4ff0008040 comp rx=0 tx=0).stop 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 10 monclient: shutdown 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff413a2d0 msgr2=0x7f4ff413a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff413a2d0 0x7f4ff413a6b0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4fec009040 tx=0x7f4fec0196d0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffa472640 1 -- 172.21.15.98:0/3030657431 reap_dead start 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 shutdown_connections 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 --2- 172.21.15.98:0/3030657431 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4ff413a2d0 0x7f4ff413a6b0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 >> 172.21.15.98:0/3030657431 conn(0x7f4ff40ac790 msgr2=0x7f4ff4135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 shutdown_connections 2024-09-11T02:54:43.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 -- 172.21.15.98:0/3030657431 wait complete. 2024-09-11T02:54:43.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.021+0000 7f4ffbed5640 1 librados: shutdown 2024-09-11T02:54:43.100 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-11T02:54:43.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-09-11T02:54:43.100 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:54:43.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.221+0000 7fd1c53d2640 10 monclient: get_monmap_and_config 2024-09-11T02:54:43.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.221+0000 7fd1c53d2640 10 monclient: build_initial_monmap 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.221+0000 7fd1c53d2640 10 monclient: monmap: 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:43.229052+0000 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:43.229052+0000 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:43.228 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:43.229 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:43.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 Processor -- start 2024-09-11T02:54:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 -- start start 2024-09-11T02:54:43.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient: init 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient: _add_conns ranks=[8,5,2,4,0,9,6,1,10,7,3] 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1c0139b80 0x7fd1c0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): picked mon.noname-e con 0x7fd1c0139b80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c006fe50 0x7fd1c00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): picked mon.noname-h con 0x7fd1c006fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c0072640 0x7fd1c00a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): picked mon.noname-b con 0x7fd1c0072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c00719c0 con 0x7fd1c0072640 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd1c0071c10 con 0x7fd1c006fe50 2024-09-11T02:54:43.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd1c00702d0 con 0x7fd1c0139b80 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1c53d2640 10 monclient(hunting): authenticate will time out at 1237.109463s 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1bf7fe640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c0072640 msgr2=0x7fd1c00a7f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1bf7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c0072640 0x7fd1c00a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1c0139b80 0x7fd1c0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1c0139b80 0x7fd1c0139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:59698/0 (socket says 172.21.15.98:59698) 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 -- 172.21.15.98:0/2837379973 learned_addr learned my addr 172.21.15.98:0/2837379973 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): get_auth_request con 0x7fd1c0139b80 auth_method 0 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1be7fc640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c006fe50 0x7fd1c00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1be7fc640 10 monclient(hunting): get_auth_request con 0x7fd1c006fe50 auth_method 0 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient(hunting): handle_auth_done global_id 24056 payload 274 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 -- 172.21.15.98:0/2837379973 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c006fe50 msgr2=0x7fd1c00a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c006fe50 0x7fd1c00a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 -- 172.21.15.98:0/2837379973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c0072640 msgr2=0x7fd1c00a7f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c0072640 0x7fd1c00a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient: _finish_hunting 0 2024-09-11T02:54:43.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 monclient: found mon.noname-e 2024-09-11T02:54:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:54:43.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.225+0000 7fd1beffd640 1 -- 172.21.15.98:0/2837379973 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c0071370 con 0x7fd1c0139b80 2024-09-11T02:54:43.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1beffd640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1c0139b80 0x7fd1c0139f60 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009eb0 tx=0x7fd1b001c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=767ba80906bcef61 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 1 -- 172.21.15.98:0/2837379973 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b00090d0 con 0x7fd1c0139b80 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: dump: 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.242 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 1 -- 172.21.15.98:0/2837379973 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b00a5a70 con 0x7fd1c0139b80 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 5 monclient: authenticate success, global_id 24056 2024-09-11T02:54:43.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 1 -- 172.21.15.98:0/2837379973 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b00a5c60 con 0x7fd1c0139b80 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 4 set_mon_vals no callback set 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bdffb640 10 monclient: dump: 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.244 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient: shutdown 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:43.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2837379973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1c0139b80 msgr2=0x7fd1c0139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1c0139b80 0x7fd1c0139f60 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009eb0 tx=0x7fd1b001c910 comp rx=0 tx=0).stop 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2837379973 shutdown_connections 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd1c0072640 0x7fd1c00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c006fe50 0x7fd1c00a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2837379973 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd1c0139b80 0x7fd1c0139f60 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2837379973 >> 172.21.15.98:0/2837379973 conn(0x7fd1c00ac730 msgr2=0x7fd1c00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2837379973 shutdown_connections 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2837379973 wait complete. 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient: build_initial_monmap 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient: monmap: 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:43.246 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 librados: starting msgr at 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 librados: starting objecter 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 Processor -- start 2024-09-11T02:54:43.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- start start 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 librados: setting wanted keys 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 librados: calling monclient init 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient: init 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient: _add_conns ranks=[0,4,5,7,6,8,3,2,1] 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c006fe50 0x7fd1c013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): picked mon.noname-a con 0x7fd1c006fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c0072640 0x7fd1c013abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): picked mon.noname-c con 0x7fd1c0072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0139b80 0x7fd1c013b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): picked mon.noname-h con 0x7fd1c0139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c01435c0 con 0x7fd1c006fe50 2024-09-11T02:54:43.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd1c0143440 con 0x7fd1c0072640 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd1c0143740 con 0x7fd1c0139b80 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 10 monclient(hunting): authenticate will time out at 1237.115128s 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1beffd640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c006fe50 msgr2=0x7fd1c013a6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1beffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c006fe50 0x7fd1c013a6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c0072640 0x7fd1c013abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c0072640 0x7fd1c013abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:60500/0 (socket says 172.21.15.98:60500) 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 -- 172.21.15.98:0/2908379734 learned_addr learned my addr 172.21.15.98:0/2908379734 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): get_auth_request con 0x7fd1c0072640 auth_method 0 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1bf7fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0139b80 0x7fd1c013b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient(hunting): handle_auth_done global_id 23575 payload 824 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0139b80 msgr2=0x7fd1c013b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0139b80 0x7fd1c013b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c006fe50 msgr2=0x7fd1c013a6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1c006fe50 0x7fd1c013a6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 monclient: found mon.noname-c 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 -- 172.21.15.98:0/2908379734 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c00aa240 con 0x7fd1c0072640 2024-09-11T02:54:43.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1be7fc640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c0072640 0x7fd1c013abf0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800d830 tx=0x7fd1a800dcf0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 10 client.?.objecter ms_handle_connect 0x7fd1c0072640 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1a800bb30 con 0x7fd1c0072640 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 10 monclient: dump: 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 10 monclient: _finish_auth 0 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 10 monclient: _check_auth_tickets 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1a8019070 con 0x7fd1c0072640 2024-09-11T02:54:43.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.229+0000 7fd1c53d2640 5 monclient: authenticate success, global_id 23575 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 10 monclient: _renew_subs 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c0135aa0 con 0x7fd1c0072640 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1a800bcf0 con 0x7fd1c0072640 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: dump: 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1bd7fa640 4 set_mon_vals no callback set 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 10 client.23575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 10 monclient: _renew_subs 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c00a5850 con 0x7fd1c0072640 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 1 librados: init done 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0133760 0x7fd1c0133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd1c006ca10 con 0x7fd1c0133760 2024-09-11T02:54:43.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd1a8013070 con 0x7fd1c0072640 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0133760 0x7fd1c0133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd1a003df40 0x7fd1a0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient: get_auth_request con 0x7fd1c0133760 auth_method 0 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(196..196 src has 1..196) ==== 6377+0+0 (secure 0 0 0) 0x7fd1a800f040 con 0x7fd1c0072640 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 client.23575.objecter ms_dispatch 0x7fd1c0000ba0 osd_map(196..196 src has 1..196) 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 3 client.23575.objecter handle_osd_map got epochs [196,196] > 0 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 3 client.23575.objecter handle_osd_map decoding full epoch 196 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 20 client.23575.objecter dump_active .. 0 homeless 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1bf7fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd1a003df40 0x7fd1a0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1bf7fe640 10 monclient: get_auth_request con 0x7fd1a003df40 auth_method 0 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 10 monclient(hunting): handle_auth_done global_id 19742 payload 824 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 1 -- 172.21.15.98:0/2908379734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1c0139f60 con 0x7fd1c0133760 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 client.23575.objecter ms_handle_connect 0x7fd1c0133760 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 client.23575.objecter resend_mon_ops 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1beffd640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0133760 0x7fd1c0133b40 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009e80 tx=0x7fd1b00196b0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 client.23575.objecter ms_handle_connect 0x7fd1a003df40 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1bf7fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd1a003df40 0x7fd1a0040400 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fd1c01402c0 tx=0x7fd1b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b0009af0 con 0x7fd1c0133760 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: dump: 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.259 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd1b00094d0 con 0x7fd1c0133760 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0133760 msgr2=0x7fd1c0133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0133760 0x7fd1c0133b40 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009e80 tx=0x7fd1b00196b0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0133760 msgr2=0x7fd1c0133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:43.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.233+0000 7fd1b77fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0133760 0x7fd1c0133b40 secure :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0009e80 tx=0x7fd1b00196b0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1c53d2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0134080 0x7fd1c00a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1c53d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd1c006e870 con 0x7fd1c0134080 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0134080 0x7fd1c00a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient: get_auth_request con 0x7fd1c0134080 auth_method 0 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 10 monclient(hunting): handle_auth_done global_id 19751 payload 824 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 1 -- 172.21.15.98:0/2908379734 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1c0069090 con 0x7fd1c0134080 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1b77fe640 10 client.23575.objecter ms_handle_connect 0x7fd1c0134080 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1beffd640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0134080 0x7fd1c00a0e60 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd1b001c740 tx=0x7fd1b0009810 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.241+0000 7fd1b77fe640 10 client.23575.objecter resend_mon_ops 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1b00b9470 con 0x7fd1c0134080 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 10 monclient: dump: 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.263 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.264 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9390 (secure 0 0 0) 0x7fd1b00045a0 con 0x7fd1c0134080 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0134080 msgr2=0x7fd1c00a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0134080 0x7fd1c00a0e60 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd1b001c740 tx=0x7fd1b0009810 comp rx=0 tx=0).stop 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0134080 msgr2=0x7fd1c00a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1b77fe640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd1c0134080 0x7fd1c00a0e60 secure :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fd1b001c740 tx=0x7fd1b0009810 comp rx=0 tx=0).stop 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:43.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 44, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 298991, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:43.266 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.267 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.268 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:43.269 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.270 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:43.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:43.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.281 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.282 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:43.284 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:43.285 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:54:43.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.288 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:43.289 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:43.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 10 librados: watch_flush enter 2024-09-11T02:54:43.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 10 librados: watch_flush exit 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 20 client.23575.objecter shutdown clearing up homeless session... 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 10 client.23575.objecter successfully canceled tick 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd1a003df40 msgr2=0x7fd1a0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd1a003df40 0x7fd1a0040400 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7fd1c01402c0 tx=0x7fd1b8008040 comp rx=0 tx=0).stop 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 10 monclient: shutdown 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c0072640 msgr2=0x7fd1c013abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 --2- 172.21.15.98:0/2908379734 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd1c0072640 0x7fd1c013abf0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800d830 tx=0x7fd1a800dcf0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1bf7fe640 1 -- 172.21.15.98:0/2908379734 reap_dead start 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1bf7fe640 1 -- 172.21.15.98:0/2908379734 reap_dead start 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 shutdown_connections 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 >> 172.21.15.98:0/2908379734 conn(0x7fd1c00ac730 msgr2=0x7fd1c00a6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:43.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 shutdown_connections 2024-09-11T02:54:43.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 -- 172.21.15.98:0/2908379734 wait complete. 2024-09-11T02:54:43.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.245+0000 7fd1c53d2640 1 librados: shutdown 2024-09-11T02:54:43.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-11T02:54:43.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-09-11T02:54:43.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:54:43.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient: get_monmap_and_config 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient: build_initial_monmap 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient: monmap: 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:54:43.414187+0000 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:54:43.414187+0000 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.413 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:54:43.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 1 Processor -- start 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 1 -- start start 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient: init 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient: _add_conns ranks=[0,1,7,8,4,10,5,3,9,2,6] 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f413c0a3690 addr v1:172.21.15.98:6789/0 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f413c09df80 addr v1:172.21.15.190:6789/0 2024-09-11T02:54:43.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient(hunting): picked mon.noname-i con 0x7f413c0a4180 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c0a0870 con 0x7f413c0a3690 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c09fe90 con 0x7f413c09df80 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f413c073d70 con 0x7f413c0a4180 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f41415cd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7f413c0a3690 legacy=0x7f413c0a3a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4143831640 10 monclient(hunting): authenticate will time out at 1237.294420s 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4140dcc640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f413c09df80 0x7f413c13b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:45528/0 (socket says 172.21.15.98:45528) 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4140dcc640 1 -- 172.21.15.98:0/2875360830 learned_addr learned my addr 172.21.15.98:0/2875360830 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): get_auth_request con 0x7f413c0a4180 auth_method 0 2024-09-11T02:54:43.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient(hunting): handle_auth_done global_id 19312 payload 274 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 1 -- 172.21.15.98:0/2875360830 >> v1:172.21.15.190:6789/0 conn(0x7f413c09df80 legacy=0x7f413c13b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 1 -- 172.21.15.98:0/2875360830 >> v1:172.21.15.98:6789/0 conn(0x7f413c0a3690 legacy=0x7f413c0a3a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient: _finish_hunting 0 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 1 monclient: found mon.noname-i 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:54:43.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.409+0000 7f4141dce640 1 -- 172.21.15.98:0/2875360830 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c09fbb0 con 0x7f413c0a4180 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4141dce640 1 --2- 172.21.15.98:0/2875360830 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13efd0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f413c074a00 tx=0x7f413401c700 comp rx=0 tx=0).ready entity=mon.7 client_cookie=1c60bc71b4652db7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 1 -- 172.21.15.98:0/2875360830 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41340ac070 con 0x7f413c0a4180 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: dump: 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.420 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: _finish_auth 0 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: _check_auth_tickets 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 1 -- 172.21.15.98:0/2875360830 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41340a6910 con 0x7f413c0a4180 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 5 monclient: authenticate success, global_id 19312 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 1 -- 172.21.15.98:0/2875360830 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41340a7480 con 0x7f413c0a4180 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 10 monclient: get_monmap_and_config success 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 4 set_mon_vals no callback set 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4133fff640 10 monclient: dump: 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.422 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.423 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 10 monclient: shutdown 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 1 -- 172.21.15.98:0/2875360830 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 msgr2=0x7f413c13efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 1 --2- 172.21.15.98:0/2875360830 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13efd0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f413c074a00 tx=0x7f413401c700 comp rx=0 tx=0).stop 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 1 -- 172.21.15.98:0/2875360830 shutdown_connections 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 1 --2- 172.21.15.98:0/2875360830 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13efd0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 1 -- 172.21.15.98:0/2875360830 >> 172.21.15.98:0/2875360830 conn(0x7f413c0a4780 msgr2=0x7f413c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 1 -- 172.21.15.98:0/2875360830 shutdown_connections 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.413+0000 7f4143831640 1 -- 172.21.15.98:0/2875360830 wait complete. 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: build_initial_monmap 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: monmap: 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:54:43.424 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 librados: starting msgr at 2024-09-11T02:54:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 librados: starting objecter 2024-09-11T02:54:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 Processor -- start 2024-09-11T02:54:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 -- start start 2024-09-11T02:54:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 librados: setting wanted keys 2024-09-11T02:54:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 librados: calling monclient init 2024-09-11T02:54:43.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: init 2024-09-11T02:54:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: _reopen_session rank -1 2024-09-11T02:54:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: _add_conns ranks=[6,0,7,5,2,4,1,3,8] 2024-09-11T02:54:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f413c09df80 0x7f413c1ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): picked mon.noname-d con 0x7f413c09df80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413c0a3690 0x7f413c13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): picked mon.noname-a con 0x7f413c0a3690 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): picked mon.noname-i con 0x7f413c0a4180 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f413c09cde0 con 0x7f413c0a3690 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f413c09cc60 con 0x7f413c09df80 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f413c09cf60 con 0x7f413c0a4180 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): _renew_subs 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): authenticate will time out at 1237.301185s 2024-09-11T02:54:43.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4140dcc640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413c0a3690 msgr2=0x7f413c13d950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41415cd640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f413c09df80 msgr2=0x7f413c1ccdd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4140dcc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413c0a3690 0x7f413c13d950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41415cd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f413c09df80 0x7f413c1ccdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:57234/0 (socket says 172.21.15.98:57234) 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 -- 172.21.15.98:0/2913505831 learned_addr learned my addr 172.21.15.98:0/2913505831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): get_auth_request con 0x7f413c0a4180 auth_method 0 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient(hunting): handle_auth_done global_id 19321 payload 824 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f413c09df80 msgr2=0x7f413c1ccdd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f413c09df80 0x7f413c1ccdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413c0a3690 msgr2=0x7f413c13d950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f413c0a3690 0x7f413c13d950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient: _finish_hunting 0 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 monclient: found mon.noname-i 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 -- 172.21.15.98:0/2913505831 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c1380a0 con 0x7f413c0a4180 2024-09-11T02:54:43.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4141dce640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13de90 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f413401cfa0 tx=0x7f41340194a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 client.?.objecter ms_handle_connect 0x7f413c0a4180 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41340ac050 con 0x7f413c0a4180 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: dump: 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.434 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: _finish_auth 0 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4134003bb0 con 0x7f413c0a4180 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 5 monclient: authenticate success, global_id 19321 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:54:43.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4134003d60 con 0x7f413c0a4180 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: _renew_subs 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413c0a5d80 con 0x7f413c0a4180 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41337fe640 4 set_mon_vals no callback set 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 client.19321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4131ffb640 10 monclient: dump: 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.436 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: _renew_subs 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T02:54:43.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f413c13e3d0 con 0x7f413c0a4180 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 librados: init done 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136670 0x7f413c136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f413c06ca10 con 0x7f413c136670 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41415cd640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136670 0x7f413c136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41415cd640 10 monclient: get_auth_request con 0x7f413c136670 auth_method 0 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41415cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41415cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.417+0000 7f41415cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f41415cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f41415cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f41415cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4134004040 con 0x7f413c0a4180 2024-09-11T02:54:43.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f41415cd640 10 monclient(hunting): handle_auth_done global_id 19330 payload 824 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f41415cd640 1 -- 172.21.15.98:0/2913505831 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f413c068b40 con 0x7f413c136670 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f412003df40 0x7f4120040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 5 ==== osd_map(196..196 src has 1..196) ==== 6377+0+0 (secure 0 0 0) 0x7f41340b7070 con 0x7f413c0a4180 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 client.19321.objecter ms_dispatch 0x7f413c000ba0 osd_map(196..196 src has 1..196) 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 3 client.19321.objecter handle_osd_map got epochs [196,196] > 0 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 3 client.19321.objecter handle_osd_map decoding full epoch 196 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f41415cd640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136670 0x7f413c136a50 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f413800b310 tx=0x7f413800b7d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 20 client.19321.objecter dump_active .. 0 homeless 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 client.19321.objecter ms_handle_connect 0x7f413c136670 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 client.19321.objecter resend_mon_ops 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4140dcc640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f412003df40 0x7f4120040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4140dcc640 10 monclient: get_auth_request con 0x7f412003df40 auth_method 0 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4138016070 con 0x7f413c136670 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 monclient: dump: 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 client.19321.objecter ms_handle_connect 0x7f412003df40 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4140dcc640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f412003df40 0x7f4120040400 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7f413c137d50 tx=0x7f412c007b30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4138003bd0 con 0x7f413c136670 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136670 msgr2=0x7f413c136a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136670 0x7f413c136a50 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f413800b310 tx=0x7f413800b7d0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136670 msgr2=0x7f413c136a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:43.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.421+0000 7f4131ffb640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136670 0x7f413c136a50 secure :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f413800b310 tx=0x7f413800b7d0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.425+0000 7f4143831640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:54:43.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.425+0000 7f4143831640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136f90 0x7f413c1326b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:54:43.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.425+0000 7f4143831640 10 monclient(hunting): start opening mon connection 2024-09-11T02:54:43.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.425+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f413c06e870 con 0x7f413c136f90 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136f90 0x7f413c1326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient: get_auth_request con 0x7f413c136f90 auth_method 0 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 10 monclient(hunting): handle_auth_done global_id 19339 payload 824 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 1 -- 172.21.15.98:0/2913505831 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f413c138c00 con 0x7f413c136f90 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f41415cd640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136f90 0x7f413c1326b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4138013000 tx=0x7f4138003800 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 10 client.19321.objecter ms_handle_connect 0x7f413c136f90 2024-09-11T02:54:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 10 client.19321.objecter resend_mon_ops 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4138016040 con 0x7f413c136f90 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 10 monclient: dump: 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:54:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9223 (secure 0 0 0) 0x7f4138011400 con 0x7f413c136f90 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136f90 msgr2=0x7f413c1326b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136f90 0x7f413c1326b0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4138013000 tx=0x7f4138003800 comp rx=0 tx=0).stop 2024-09-11T02:54:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136f90 msgr2=0x7f413c1326b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4131ffb640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c136f90 0x7f413c1326b0 secure :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f4138013000 tx=0x7f4138003800 comp rx=0 tx=0).stop 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 44, 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 299199, 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:54:43.447 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.448 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:54:43.450 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:54:43.451 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:54:43.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:54:43.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:54:43.455 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:54:43.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:54:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:54:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:54:43.459 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.460 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:54:43.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.465 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:54:43.467 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.468 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:54:43.470 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:54:43.471 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:54:43.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 10 librados: watch_flush enter 2024-09-11T02:54:43.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 10 librados: watch_flush exit 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 20 client.19321.objecter shutdown clearing up homeless session... 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 10 client.19321.objecter successfully canceled tick 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f412003df40 msgr2=0x7f4120040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f412003df40 0x7f4120040400 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7f413c137d50 tx=0x7f412c007b30 comp rx=0 tx=0).stop 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 10 monclient: shutdown 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 msgr2=0x7f413c13de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 --2- 172.21.15.98:0/2913505831 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f413c0a4180 0x7f413c13de90 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f413401cfa0 tx=0x7f41340194a0 comp rx=0 tx=0).stop 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4141dce640 1 -- 172.21.15.98:0/2913505831 reap_dead start 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4141dce640 1 -- 172.21.15.98:0/2913505831 reap_dead start 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 shutdown_connections 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 >> 172.21.15.98:0/2913505831 conn(0x7f413c0a4780 msgr2=0x7f413c13ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:54:43.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 shutdown_connections 2024-09-11T02:54:43.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 -- 172.21.15.98:0/2913505831 wait complete. 2024-09-11T02:54:43.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:54:43.429+0000 7f4143831640 1 librados: shutdown 2024-09-11T02:54:43.474 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-09-11T02:54:43.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T02:54:58.308 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAi 2024-09-11T02:54:58.308 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6952 ms) 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6995 ms) 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7035 ms) 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7045 ms) 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6939 ms) 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-11T02:54:58.309 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7033 ms) 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6977 ms) 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7008 ms) 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6995 ms) 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7008 ms) 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7017 ms) 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-11T02:54:58.310 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6986 ms) 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (17701 ms) 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (21110 ms) 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7118 ms) 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6968 ms) 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (136888 ms total) 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [----------] Global test environment tear-down 2024-09-11T02:54:58.311 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (237313 ms total) 2024-09-11T02:54:58.312 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-11T02:55:05.285 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-11T02:55:05.285 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-11T02:55:05.285 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-11T02:55:05.285 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: seed 28350 2024-09-11T02:55:05.285 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-11T02:55:05.285 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (191 ms) 2024-09-11T02:55:05.285 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10929 ms) 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (25 ms) 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11145 ms total) 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7083 ms) 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7926 ms) 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10016 ms) 2024-09-11T02:55:05.286 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [3] 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [4,3] 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting some heads 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: waiting for scrubs... 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: done waiting 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (58057 ms) 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-11T02:55:05.287 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (9980 ms) 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8005 ms) 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7967 ms) 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8120 ms) 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9938 ms) 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9008 ms) 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10097 ms) 2024-09-11T02:55:05.288 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12871 ms) 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8049 ms) 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (17942 ms) 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (27920 ms) 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7987 ms) 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-11T02:55:05.289 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8009 ms) 2024-09-11T02:55:20.999 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-11T02:55:20.999 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-11T02:55:20.999 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: ok, hit_set contains 268:602f83fe:::foo:head 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9001 ms) 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg_num = 32 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 0 ls 1726023321,0 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 1 ls 1726023321,0 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 2 ls 1726023321,0 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 3 ls 1726023321,0 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 4 ls 1726023321,0 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 5 ls 1726023321,0 2024-09-11T02:55:21.000 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 6 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 7 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 8 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 9 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 10 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 11 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 12 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 13 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 14 ls 1726023321,0 2024-09-11T02:55:21.001 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 15 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 16 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 17 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 18 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 19 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 20 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 21 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 22 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 23 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 24 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 25 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 26 ls 1726023321,0 2024-09-11T02:55:21.002 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 27 ls 1726023321,0 2024-09-11T02:55:21.003 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 28 ls 1726023321,0 2024-09-11T02:55:21.003 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 29 ls 1726023321,0 2024-09-11T02:55:21.003 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 30 ls 1726023321,0 2024-09-11T02:55:21.003 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg 31 ls 1726023321,0 2024-09-11T02:55:21.003 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: pg_num = 32 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6cac518f:::0:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:02547ec2:::1:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f905c69b:::2:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cfc208b3:::3:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d83876eb:::4:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b29083e3:::5:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c4fdafeb:::6:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5c6b0b28:::7:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bd63b0f1:::8:head 2024-09-11T02:55:21.005 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e960b815:::9:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:52ea6a34:::10:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:89d3ae78:::11:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:de5d7c5f:::12:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:566253c9:::13:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:62a1935d:::14:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:863748b0:::15:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3958e169:::16:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4d4dabf9:::17:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:8391935d:::18:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:28883081:::19:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:69259c59:::20:head 2024-09-11T02:55:21.006 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4bdb80b7:::21:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a11c5d71:::22:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:271af37b:::23:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:95b121be:::24:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:58d1031b:::25:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0a050783:::26:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c709704c:::27:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cbe56eaf:::28:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:86b4b162:::29:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:70d89383:::30:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:dd450c7c:::31:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6d5729b1:::32:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c388f3fb:::33:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:56cfea31:::34:head 2024-09-11T02:55:21.007 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9dbc1bf7:::35:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:40b74ccd:::36:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4d5aaf42:::37:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:920f362c:::38:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6cc53222:::39:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9cad833f:::40:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1ea84d41:::41:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c4480ef6:::42:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a694361e:::43:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d1bd33e9:::44:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ddc2cd5d:::45:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:2b782207:::46:head 2024-09-11T02:55:21.008 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7b187fca:::47:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:90ecdf6f:::48:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a5ed95fe:::49:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ea0eaa55:::50:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f33ef17b:::51:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a0d1b2f6:::52:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:60c5229e:::53:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:edcbc575:::54:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:102cf253:::55:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:efb7fb0b:::56:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:50d0a326:::57:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d4dc5daf:::58:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3a130462:::59:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ec87ed71:::60:head 2024-09-11T02:55:21.009 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d5bc9454:::61:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3ddfe313:::62:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7c2816b9:::63:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:47e00e4d:::64:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c6410c18:::65:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b48ed237:::66:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cd63ad31:::67:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b179e92b:::68:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0d9f741a:::69:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6d3352ae:::70:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c6d5c19e:::71:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bc4729c3:::72:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:77e930b9:::73:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0abeecfd:::74:head 2024-09-11T02:55:21.010 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b7c37e15:::75:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b6378398:::76:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:02bd68de:::77:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:cc795d2d:::78:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:630d4fea:::79:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e0d29ef5:::80:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:fd6f13d2:::81:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:606461d5:::82:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:eadbdc43:::83:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:8761d0bb:::84:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9ef0186f:::85:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e0d41294:::86:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:961de695:::87:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1423148f:::88:head 2024-09-11T02:55:21.011 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:633a8fa2:::89:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a8653809:::90:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3dac8b33:::91:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:35aad435:::92:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f6dcc343:::93:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:dbbdad87:::94:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1cb48ce0:::95:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:03cd461c:::96:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:17a4ea99:::97:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9993c9a7:::98:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6394211c:::99:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:94c7ae57:::100:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6fdee5bb:::101:head 2024-09-11T02:55:21.012 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9a477fd1:::102:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:eb850916:::103:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:affc56b9:::104:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b42dc814:::105:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f319f8f0:::106:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9a40b9de:::107:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:8b524f28:::108:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e3de589f:::109:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:90f90a5b:::110:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a7b4f1d7:::111:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:af51766e:::112:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b6f90bd1:::113:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e0261208:::114:head 2024-09-11T02:55:21.013 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c9569ef7:::115:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:61bebe50:::116:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:fe93412b:::117:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d3d38bee:::118:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3100ba0c:::119:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d0560ada:::120:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f0ea8b35:::121:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:766f231a:::122:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a07a2582:::123:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bd7c6b3a:::124:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:fb2ddaff:::125:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4408e1fe:::126:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ee1df7a7:::127:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c3002909:::128:head 2024-09-11T02:55:21.014 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4f48ffa9:::129:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:edf38733:::130:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c08425c0:::131:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5f902d98:::132:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:41ea2c93:::133:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:813cee13:::134:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0131818d:::135:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:26ba5a85:::136:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:381b8a5a:::137:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:28797e47:::138:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:bfca7f22:::139:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:36807075:::140:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:80b03975:::141:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5c15709b:::142:head 2024-09-11T02:55:21.015 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f39ea15e:::143:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ea992956:::144:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:48887b1c:::145:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:9f24a9dd:::146:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:987f100b:::147:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d2dd3581:::148:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7fed1808:::149:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c80b70e9:::150:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:85ed90f9:::151:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:36428b24:::152:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d044c34a:::153:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7c18bf58:::154:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d1c21232:::155:head 2024-09-11T02:55:21.016 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a7a3c575:::156:head 2024-09-11T02:55:21.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:87da0633:::157:head 2024-09-11T02:55:21.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d5ac3822:::158:head 2024-09-11T02:55:21.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3f20522d:::159:head 2024-09-11T02:55:21.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6ca26563:::160:head 2024-09-11T02:55:21.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:532ce135:::161:head 2024-09-11T02:55:21.017 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:c78863e6:::162:head 2024-09-11T02:55:53.340 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7462ddf6:::o.RoundTripAppendPP (2994 ms) 2024-09-11T02:55:53.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-11T02:55:53.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2958 ms) 2024-09-11T02:55:53.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-11T02:55:53.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3055 ms) 2024-09-11T02:55:53.341 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-11T02:55:53.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3063 ms) 2024-09-11T02:55:53.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-11T02:55:53.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: setting pool EIO 2024-09-11T02:55:53.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: max_success 88, min_failed 89 2024-09-11T02:55:53.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3903 ms) 2024-09-11T02:55:53.342 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-11T02:55:53.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3021 ms) 2024-09-11T02:55:53.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (137403 ms total) 2024-09-11T02:55:53.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T02:55:53.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-11T02:55:53.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-11T02:55:53.343 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3002 ms) 2024-09-11T02:55:53.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-11T02:55:53.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8990 ms) 2024-09-11T02:55:53.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-11T02:55:53.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15002 ms) 2024-09-11T02:55:53.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-11T02:55:53.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3018 ms) 2024-09-11T02:55:53.344 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30012 ms total) 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2984 ms) 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2984 ms total) 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14009 ms) 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (17703 ms) 2024-09-11T02:55:53.345 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (21111 ms) 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7103 ms) 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2966 ms) 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7065 ms) 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6952 ms) 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-11T02:55:53.346 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (10001 ms) 2024-09-11T02:55:53.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-11T02:55:53.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6996 ms) 2024-09-11T02:55:53.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-11T02:55:53.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7030 ms) 2024-09-11T02:55:53.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-11T02:55:53.347 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7010 ms) 2024-09-11T02:55:53.348 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-11T02:55:53.348 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6975 ms) 2024-09-11T02:55:53.348 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-11T02:55:53.348 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7030 ms) 2024-09-11T02:56:13.474 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T02:56:13.475 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T02:56:13.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T02:56:13.480 INFO:tasks.ceph.mon.a:Started 2024-09-11T02:56:13.480 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T02:56:13.480 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T02:56:13.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T02:56:13.482 INFO:tasks.ceph.mon.b:Started 2024-09-11T02:56:13.482 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-09-11T02:56:13.483 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-11T02:56:13.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-11T02:56:13.485 INFO:tasks.ceph.mon.g:Started 2024-09-11T02:56:13.485 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-09-11T02:56:13.486 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-11T02:56:13.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-11T02:56:13.488 INFO:tasks.ceph.mon.d:Started 2024-09-11T02:56:13.489 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:56:13.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:56:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.608+0000 7fe1c22bd640 10 monclient: get_monmap_and_config 2024-09-11T02:56:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.608+0000 7fe1c22bd640 10 monclient: build_initial_monmap 2024-09-11T02:56:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.608+0000 7fe1c22bd640 10 monclient: monmap: 2024-09-11T02:56:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:13.612662+0000 2024-09-11T02:56:13.612 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:13.612662+0000 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:13.613 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:13.614 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.608+0000 7fe1c22bd640 1 Processor -- start 2024-09-11T02:56:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- start start 2024-09-11T02:56:13.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient: init 2024-09-11T02:56:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient: _add_conns ranks=[9,1,8,4,3,5,6,10,0,7,2] 2024-09-11T02:56:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 0x7fe1bc072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient(hunting): picked mon.noname-a con 0x7fe1bc06f6c0 addr v2:172.21.15.98:3300/0 2024-09-11T02:56:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe1bc06faa0 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc13ec80 0x7fe1bc141070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient(hunting): picked mon.noname-e con 0x7fe1bc13ec80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1bc0a8750 con 0x7fe1bc06faa0 2024-09-11T02:56:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe1bc071ff0 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe1bc0716c0 con 0x7fe1bc13ec80 2024-09-11T02:56:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient(hunting): authenticate will time out at 1327.492586s 2024-09-11T02:56:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 0x7fe1bc072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc13ec80 0x7fe1bc141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 0x7fe1bc072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39112/0 (socket says 172.21.15.98:39112) 2024-09-11T02:56:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc13ec80 0x7fe1bc141070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45694/0 (socket says 172.21.15.98:45694) 2024-09-11T02:56:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 1 -- 172.21.15.98:0/4201758844 learned_addr learned my addr 172.21.15.98:0/4201758844 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 10 monclient(hunting): get_auth_request con 0x7fe1bc06f6c0 auth_method 0 2024-09-11T02:56:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): get_auth_request con 0x7fe1bc13ec80 auth_method 0 2024-09-11T02:56:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 1 -- 172.21.15.98:0/4201758844 >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 msgr2=0x7fe1bc072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 1 -- 172.21.15.98:0/4201758844 >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 msgr2=0x7fe1bc072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 1 --2- 172.21.15.98:0/4201758844 >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 0x7fe1bc072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bbfff640 1 --2- 172.21.15.98:0/4201758844 >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 0x7fe1bc072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient(hunting): handle_auth_done global_id 24155 payload 274 2024-09-11T02:56:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 -- 172.21.15.98:0/4201758844 >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 msgr2=0x7fe1bc072940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 --2- 172.21.15.98:0/4201758844 >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 0x7fe1bc072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 -- 172.21.15.98:0/4201758844 >> v1:172.21.15.190:6789/0 conn(0x7fe1bc06faa0 legacy=0x7fe1bc06fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient: _finish_hunting 0 2024-09-11T02:56:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 monclient: found mon.noname-e 2024-09-11T02:56:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 -- 172.21.15.98:0/4201758844 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc0a64a0 con 0x7fe1bc13ec80 2024-09-11T02:56:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c085a640 1 --2- 172.21.15.98:0/4201758844 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc13ec80 0x7fe1bc141070 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009d90 tx=0x7fe1b401c830 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4420f5dc0f57f213 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 1 -- 172.21.15.98:0/4201758844 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1b40b2070 con 0x7fe1bc13ec80 2024-09-11T02:56:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: dump: 2024-09-11T02:56:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:13.639 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 1 -- 172.21.15.98:0/4201758844 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1b40a5b80 con 0x7fe1bc13ec80 2024-09-11T02:56:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 1 -- 172.21.15.98:0/4201758844 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1b40a5db0 con 0x7fe1bc13ec80 2024-09-11T02:56:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 5 monclient: authenticate success, global_id 24155 2024-09-11T02:56:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 4 set_mon_vals no callback set 2024-09-11T02:56:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1bb7fe640 10 monclient: dump: 2024-09-11T02:56:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:13.643 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:13.644 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:13.645 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:13.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 10 monclient: shutdown 2024-09-11T02:56:13.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:13.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- 172.21.15.98:0/4201758844 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc13ec80 msgr2=0x7fe1bc141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:13.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/4201758844 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc13ec80 0x7fe1bc141070 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009d90 tx=0x7fe1b401c830 comp rx=0 tx=0).stop 2024-09-11T02:56:13.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- 172.21.15.98:0/4201758844 shutdown_connections 2024-09-11T02:56:13.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/4201758844 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc13ec80 0x7fe1bc141070 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/4201758844 >> v2:172.21.15.98:3300/0 conn(0x7fe1bc06f6c0 0x7fe1bc072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- 172.21.15.98:0/4201758844 >> 172.21.15.98:0/4201758844 conn(0x7fe1bc0acac0 msgr2=0x7fe1bc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:13.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- 172.21.15.98:0/4201758844 shutdown_connections 2024-09-11T02:56:13.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.612+0000 7fe1c22bd640 1 -- 172.21.15.98:0/4201758844 wait complete. 2024-09-11T02:56:13.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: build_initial_monmap 2024-09-11T02:56:13.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: monmap: 2024-09-11T02:56:13.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:13.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:13.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:13.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 librados: starting msgr at 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 librados: starting objecter 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 Processor -- start 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 -- start start 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 librados: setting wanted keys 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 librados: calling monclient init 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: init 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _add_conns ranks=[8,6,0,3,1,2,7,5,4] 2024-09-11T02:56:13.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc06f6c0 0x7fe1bc136790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): picked mon.noname-e con 0x7fe1bc06f6c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 0x7fe1bc136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): picked mon.noname-d con 0x7fe1bc06faa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 0x7fe1bc137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): picked mon.noname-a con 0x7fe1bc13ec80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc144160 con 0x7fe1bc13ec80 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:13.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe1bc143fe0 con 0x7fe1bc06faa0 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe1bc1442e0 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient(hunting): authenticate will time out at 1327.497352s 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc06f6c0 0x7fe1bc136790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc06f6c0 0x7fe1bc136790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:45708/0 (socket says 172.21.15.98:45708) 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 -- 172.21.15.98:0/2279785345 learned_addr learned my addr 172.21.15.98:0/2279785345 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): get_auth_request con 0x7fe1bc06f6c0 auth_method 0 2024-09-11T02:56:13.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 0x7fe1bc136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 0x7fe1bc137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 10 monclient(hunting): get_auth_request con 0x7fe1bc13ec80 auth_method 0 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 10 monclient(hunting): get_auth_request con 0x7fe1bc06faa0 auth_method 0 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:13.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 msgr2=0x7fe1bc137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 msgr2=0x7fe1bc136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 msgr2=0x7fe1bc137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 msgr2=0x7fe1bc136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 0x7fe1bc137210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 0x7fe1bc136cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c085a640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 0x7fe1bc137210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 0x7fe1bc136cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:13.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient(hunting): handle_auth_done global_id 24164 payload 824 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 msgr2=0x7fe1bc136cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 0x7fe1bc136cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 msgr2=0x7fe1bc137210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 0x7fe1bc137210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient: _finish_hunting 0 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 monclient: found mon.noname-e 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 -- 172.21.15.98:0/2279785345 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc13c6f0 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1bbfff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc06f6c0 0x7fe1bc136790 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00e870 tx=0x7fe1ac00ed30 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 client.?.objecter ms_handle_connect 0x7fe1bc06f6c0 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 1 -- 172.21.15.98:0/2279785345 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1ac017070 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: dump: 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:13.655 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: _finish_auth 0 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: _check_auth_tickets 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 5 monclient: authenticate success, global_id 24164 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 1 -- 172.21.15.98:0/2279785345 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1ac00ee60 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _renew_subs 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:13.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc0aa5f0 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 1 -- 172.21.15.98:0/2279785345 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe1ac00f040 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1baffd640 4 set_mon_vals no callback set 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 client.24164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 monclient: dump: 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:13.657 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _renew_subs 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:13.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc13c410 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 librados: init done 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1bc072940 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 1 -- 172.21.15.98:0/2279785345 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe1ac012050 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe19c03df40 0x7fe19c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 1 -- 172.21.15.98:0/2279785345 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(286..286 src has 1..286) ==== 6194+0+0 (secure 0 0 0) 0x7fe1ac04ed20 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 10 client.24164.objecter ms_dispatch 0x7fe1bc000ba0 osd_map(286..286 src has 1..286) 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 3 client.24164.objecter handle_osd_map got epochs [286,286] > 0 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 3 client.24164.objecter handle_osd_map decoding full epoch 286 2024-09-11T02:56:13.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b97fa640 20 client.24164.objecter dump_active .. 0 homeless 2024-09-11T02:56:13.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe19c03df40 0x7fe19c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:13.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.616+0000 7fe1b3fff640 10 monclient: get_auth_request con 0x7fe19c03df40 auth_method 0 2024-09-11T02:56:13.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.620+0000 7fe1b97fa640 10 client.24164.objecter ms_handle_connect 0x7fe19c03df40 2024-09-11T02:56:13.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.620+0000 7fe1b3fff640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe19c03df40 0x7fe19c040400 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc13a0a0 tx=0x7fe1a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:13.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.624+0000 7fe1b97fa640 1 -- 172.21.15.98:0/2279785345 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe1ac053070 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.624+0000 7fe1b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:13.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.624+0000 7fe1b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:13.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.872+0000 7fe1c22bd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:56:13.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.872+0000 7fe1c22bd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:13.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.872+0000 7fe1c22bd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:13.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.872+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe1bc13aef0 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.872+0000 7fe1b97fa640 1 -- 172.21.15.98:0/2279785345 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3169 (secure 0 0 0) 0x7fe1ac020070 con 0x7fe1bc06f6c0 2024-09-11T02:56:13.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.872+0000 7fe1b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:13.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.872+0000 7fe1b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:13.875 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":44,"quorum":[1,4,5,7,8],"quorum_names":["f","c","h","i","e"],"quorum_leader_name":"e","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:56:13.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 10 librados: watch_flush enter 2024-09-11T02:56:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 10 librados: watch_flush exit 2024-09-11T02:56:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 20 client.24164.objecter shutdown clearing up homeless session... 2024-09-11T02:56:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 10 client.24164.objecter successfully canceled tick 2024-09-11T02:56:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe19c03df40 msgr2=0x7fe19c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe19c03df40 0x7fe19c040400 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc13a0a0 tx=0x7fe1a4009040 comp rx=0 tx=0).stop 2024-09-11T02:56:13.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 10 monclient: shutdown 2024-09-11T02:56:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc06f6c0 msgr2=0x7fe1bc136790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc06f6c0 0x7fe1bc136790 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00e870 tx=0x7fe1ac00ed30 comp rx=0 tx=0).stop 2024-09-11T02:56:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 shutdown_connections 2024-09-11T02:56:13.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe19c03df40 0x7fe19c040400 unknown :-1 s=CLOSED pgs=3494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1bc13ec80 0x7fe1bc137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe1bc06faa0 0x7fe1bc136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 --2- 172.21.15.98:0/2279785345 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe1bc06f6c0 0x7fe1bc136790 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 >> 172.21.15.98:0/2279785345 conn(0x7fe1bc0acac0 msgr2=0x7fe1bc0a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 shutdown_connections 2024-09-11T02:56:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 -- 172.21.15.98:0/2279785345 wait complete. 2024-09-11T02:56:13.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:13.876+0000 7fe1c22bd640 1 librados: shutdown 2024-09-11T02:56:16.896 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient: get_monmap_and_config 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient: build_initial_monmap 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient: monmap: 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:17.026620+0000 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:17.026620+0000 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:17.025 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:17.026 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 1 Processor -- start 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 1 -- start start 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient: init 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient: _add_conns ranks=[10,0,7,8,6,2,9,1,5,4,3] 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient(hunting): picked mon.noname-f con 0x7fcd780a7aa0 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd780a1db0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient(hunting): picked mon.noname-i con 0x7fcd780a15f0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:17.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd780a9730 con 0x7fcd780a1db0 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fcd780a1a70 con 0x7fcd780a7aa0 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fcd780a8900 con 0x7fcd780a15f0 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7d245640 10 monclient(hunting): authenticate will time out at 1330.906976s 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcd780a1db0 0x7fcd78139aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:55422/0 (socket says 172.21.15.98:55422) 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 learned_addr learned my addr 172.21.15.98:0/2245840447 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd780a15f0 auth_method 0 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:17.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd780a7aa0 auth_method 0 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:17.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.024+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:17.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:17.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd780a15f0 auth_method 0 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd780a7aa0 auth_method 0 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:17.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:17.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:17.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:17.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.228+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd780a15f0 auth_method 0 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd780a7aa0 auth_method 0 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:17.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:17.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:17.632+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:18.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:18.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:18.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd780a15f0 auth_method 0 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd780a7aa0 auth_method 0 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:18.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:18.436+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): tick 2024-09-11T02:56:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:56:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 monclient(hunting): continuing hunt 2024-09-11T02:56:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:56:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 msgr2=0x7fcd780a83c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd780a15f0 0x7fcd780a83c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:20.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 msgr2=0x7fcd780a7e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.190:3300/0 conn(0x7fcd780a7aa0 0x7fcd780a7e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> v1:172.21.15.98:6789/0 conn(0x7fcd780a1db0 legacy=0x7fcd78139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient: _add_conns ranks=[3,6,8,5,4,1,9,2] 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-g con 0x7fcd64003820 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-d con 0x7fcd64006250 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-e con 0x7fcd64008db0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:20.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcd780a8900 con 0x7fcd64003820 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcd6400ba10 con 0x7fcd64006250 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcd6400bc10 con 0x7fcd64008db0 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd74d99640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64006250 auth_method 0 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64008db0 auth_method 0 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd64003820 auth_method 0 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.024+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:20.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.028+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:20.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64006250 auth_method 0 2024-09-11T02:56:20.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:20.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64008db0 auth_method 0 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd64003820 auth_method 0 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:20.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.232+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:20.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64006250 auth_method 0 2024-09-11T02:56:20.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64008db0 auth_method 0 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:20.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd64003820 auth_method 0 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:20.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:20.636+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:21.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:21.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64006250 auth_method 0 2024-09-11T02:56:21.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:21.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64008db0 auth_method 0 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:21.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd64003820 auth_method 0 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:21.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:21.440+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): tick 2024-09-11T02:56:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:56:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 monclient(hunting): continuing hunt 2024-09-11T02:56:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:56:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 msgr2=0x7fcd6400b240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd64008db0 0x7fcd6400b240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:23.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 msgr2=0x7fcd640086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd64006250 0x7fcd640086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 msgr2=0x7fcd64005c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcd64003820 0x7fcd64005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 reap_dead start 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient: _add_conns ranks=[2,9,4,5,1] 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-b con 0x7fcd640066b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-a con 0x7fcd64009770 addr v2:172.21.15.98:3300/0 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-c con 0x7fcd64009bd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:23.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fcd780a8900 con 0x7fcd64009770 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcd6400ba10 con 0x7fcd640066b0 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fcd6400bc10 con 0x7fcd64009bd0 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd74d99640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd640066b0 auth_method 0 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:23.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64009770 auth_method 0 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64009bd0 auth_method 0 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:23.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.028+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64009bd0 auth_method 0 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd640066b0 auth_method 0 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64009770 auth_method 0 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:23.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:23.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:23.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.232+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64009770 auth_method 0 2024-09-11T02:56:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:23.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64009bd0 auth_method 0 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd640066b0 auth_method 0 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:23.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:23.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:23.636+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd64009bd0 auth_method 0 2024-09-11T02:56:24.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64009770 auth_method 0 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request con 0x7fcd640066b0 auth_method 0 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:24.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7759e640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:24.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:24.440+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:26.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 10 monclient(hunting): tick 2024-09-11T02:56:26.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:56:26.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 1 monclient(hunting): continuing hunt 2024-09-11T02:56:26.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:56:26.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 msgr2=0x7fcd6400a080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:26.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fcd64009bd0 0x7fcd6400a080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 msgr2=0x7fcd64009230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcd640066b0 0x7fcd64009230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.024+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 msgr2=0x7fcd64005c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> v2:172.21.15.98:3300/0 conn(0x7fcd64009770 0x7fcd64005c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 10 monclient: _add_conns ranks=[1,5] 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcd6400a970 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd64011220 0x7fcd64013610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 10 monclient(hunting): picked mon.noname-h con 0x7fcd64011220 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd780a3ea0 con 0x7fcd6400a970 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 1 -- 172.21.15.98:0/2245840447 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcd780a8900 con 0x7fcd64011220 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd74d99640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd64011220 0x7fcd64013610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd64011220 auth_method 0 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient(hunting): handle_auth_done global_id 19832 payload 274 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 >> v1:172.21.15.190:6789/0 conn(0x7fcd6400a970 legacy=0x7fcd6400ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:26.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient: _finish_hunting 0 2024-09-11T02:56:26.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 1 monclient: found mon.noname-h 2024-09-11T02:56:26.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:26.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:56:26.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.028+0000 7fcd7659c640 1 -- 172.21.15.98:0/2245840447 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd64014b60 con 0x7fcd64011220 2024-09-11T02:56:26.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7659c640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd64011220 0x7fcd64013610 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c002130 tx=0x7fcd6c01eb20 comp rx=0 tx=0).ready entity=mon.5 client_cookie=d4622fe622e7f4d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullP163:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:5d165639:::164:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f43765fc:::165:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b4c720e9:::166:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e694b040:::167:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:afa38db2:::168:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:77ba9f53:::169:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:87495034:::170:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7c96bf0e:::171:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:dbe346cc:::172:head 2024-09-11T02:56:26.037 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:e943ec24:::173:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f97a9c0c:::174:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:6f26e74d:::175:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4f95e106:::176:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:0e6f2f8f:::177:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:05db05f1:::178:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:38a78d66:::179:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:d095610b:::180:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:a1a9d709:::181:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1e5d39db:::182:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:f7df4fb9:::183:head 2024-09-11T02:56:26.038 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:03a7f161:::184:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ba70721e:::185:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:28e5662d:::186:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:973d52de:::187:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:4303eb1c:::188:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:b990b48e:::189:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:29b8165b:::190:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:3547f197:::191:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:7e260936:::192:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:1abec7b1:::193:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:10fdda93:::194:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:15817eea:::195:head 2024-09-11T02:56:26.039 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:770bab57:::196:head 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:ed9e13e7:::197:head 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:71471a8f:::198:head 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: checking for 270:10fb1d02:::199:head 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (9048 ms) 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,0 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first is 1726023331 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,0 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,0 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,0 2024-09-11T02:56:26.040 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,1726023336,1726023337,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,1726023336,1726023337,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,1726023336,1726023337,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,1726023336,1726023337,1726023339,1726023340,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,1726023336,1726023337,1726023339,1726023340,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023331,1726023333,1726023334,1726023336,1726023337,1726023339,1726023340,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023334,1726023336,1726023337,1726023339,1726023340,1726023342,1726023343,0 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first now 1726023334, trimmed 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20959 ms) 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-11T02:56:26.041 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: foo0 2024-09-11T02:56:26.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-11T02:56:26.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14055 ms) 2024-09-11T02:56:26.042 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-11T02:56:26.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 1 -- 172.21.15.98:0/2245840447 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd6c0ad070 con 0x7fcd64011220 2024-09-11T02:56:26.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:56:26.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: dump: 2024-09-11T02:56:26.048 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.049 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.052 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.052 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.052 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.052 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.052 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.052 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: _finish_auth 0 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 1 -- 172.21.15.98:0/2245840447 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd6c0a79d0 con 0x7fcd64011220 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 5 monclient: authenticate success, global_id 19832 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 1 -- 172.21.15.98:0/2245840447 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd6c0a7c00 con 0x7fcd64011220 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:26.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd75d9b640 10 monclient: dump: 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.054 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 4 set_mon_vals no callback set 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 10 monclient: shutdown 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 1 -- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd64011220 msgr2=0x7fcd64013610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 1 --2- 172.21.15.98:0/2245840447 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd64011220 0x7fcd64013610 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c002130 tx=0x7fcd6c01eb20 comp rx=0 tx=0).stop 2024-09-11T02:56:26.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7759e640 1 -- 172.21.15.98:0/2245840447 reap_dead start 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 1 -- 172.21.15.98:0/2245840447 shutdown_connections 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.036+0000 7fcd7d245640 1 -- 172.21.15.98:0/2245840447 >> 172.21.15.98:0/2245840447 conn(0x7fcd780ac850 msgr2=0x7fcd780a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 -- 172.21.15.98:0/2245840447 shutdown_connections 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 -- 172.21.15.98:0/2245840447 wait complete. 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient: build_initial_monmap 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient: monmap: 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:26.056 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:26.057 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 librados: starting msgr at 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 librados: starting objecter 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 Processor -- start 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 -- start start 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 librados: setting wanted keys 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 librados: calling monclient init 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient: init 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient: _add_conns ranks=[1,5,0,3,6,7,4,2,8] 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd781c4910 0x7fcd781226e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): picked mon.noname-f con 0x7fcd781c4910 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:26.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd781377e0 0x7fcd78137c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): picked mon.noname-h con 0x7fcd781377e0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 0x7fcd781d1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): picked mon.noname-a con 0x7fcd781d0ea0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd78139e30 con 0x7fcd781d0ea0 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcd78139cb0 con 0x7fcd781c4910 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcd78139fb0 con 0x7fcd781377e0 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:26.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7d245640 10 monclient(hunting): authenticate will time out at 1339.922819s 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 0x7fcd781d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 0x7fcd781d1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45028/0 (socket says 172.21.15.98:45028) 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 1 -- 172.21.15.98:0/43895677 learned_addr learned my addr 172.21.15.98:0/43895677 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 10 monclient(hunting): get_auth_request con 0x7fcd781d0ea0 auth_method 0 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd781377e0 0x7fcd78137c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): get_auth_request con 0x7fcd781377e0 auth_method 0 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 1 -- 172.21.15.98:0/43895677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 msgr2=0x7fcd781d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 1 -- 172.21.15.98:0/43895677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 msgr2=0x7fcd781d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 0x7fcd781d1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7759e640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 0x7fcd781d1300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient(hunting): handle_auth_done global_id 19841 payload 824 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 1 -- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd781c4910 msgr2=0x7fcd781226e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:26.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd781c4910 0x7fcd781226e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 1 -- 172.21.15.98:0/43895677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 msgr2=0x7fcd781d1300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 0x7fcd781d1300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient: _finish_hunting 0 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 1 monclient: found mon.noname-h 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.040+0000 7fcd7659c640 1 -- 172.21.15.98:0/43895677 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd781381e0 con 0x7fcd781377e0 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7659c640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd781377e0 0x7fcd78137c20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0af000 tx=0x7fcd6c00b7e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 client.?.objecter ms_handle_connect 0x7fcd781377e0 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:26.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 1 -- 172.21.15.98:0/43895677 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd6c0bf030 con 0x7fcd781377e0 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: dump: 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.063 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:26.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 1 -- 172.21.15.98:0/43895677 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd6c005040 con 0x7fcd781377e0 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 5 monclient: authenticate success, global_id 19841 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 monclient: _renew_subs 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd78138fa0 con 0x7fcd781377e0 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 1 -- 172.21.15.98:0/43895677 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd6c0ad040 con 0x7fcd781377e0 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 monclient: dump: 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.065 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.066 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7559a640 4 set_mon_vals no callback set 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 client.19841.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 monclient: _renew_subs 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd781252a0 con 0x7fcd781377e0 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 1 librados: init done 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:26.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:56:26.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd7813a340 con 0x7fcd781377e0 2024-09-11T02:56:26.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 1 -- 172.21.15.98:0/43895677 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fcd6c004b70 con 0x7fcd781377e0 2024-09-11T02:56:26.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd64040220 0x7fcd640426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 1 -- 172.21.15.98:0/43895677 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(289..289 src has 1..289) ==== 6648+0+0 (secure 0 0 0) 0x7fcd6c0c0070 con 0x7fcd781377e0 2024-09-11T02:56:26.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 10 client.19841.objecter ms_dispatch 0x7fcd78000ba0 osd_map(289..289 src has 1..289) 2024-09-11T02:56:26.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 3 client.19841.objecter handle_osd_map got epochs [289,289] > 0 2024-09-11T02:56:26.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 3 client.19841.objecter handle_osd_map decoding full epoch 289 2024-09-11T02:56:26.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd5f7fe640 20 client.19841.objecter dump_active .. 0 homeless 2024-09-11T02:56:26.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd64040220 0x7fcd640426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.044+0000 7fcd76d9d640 10 monclient: get_auth_request con 0x7fcd64040220 auth_method 0 2024-09-11T02:56:26.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.048+0000 7fcd76d9d640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd64040220 0x7fcd640426e0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7fcd68006d50 tx=0x7fcd68006be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:26.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.048+0000 7fcd5f7fe640 10 client.19841.objecter ms_handle_connect 0x7fcd64040220 2024-09-11T02:56:26.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.052+0000 7fcd5f7fe640 1 -- 172.21.15.98:0/43895677 <== mon.5 v2:172.21.15.190:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fcd6c0ef030 con 0x7fcd781377e0 2024-09-11T02:56:26.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.052+0000 7fcd5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:26.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.052+0000 7fcd5f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:26.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.292+0000 7fcd7d245640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:56:26.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.292+0000 7fcd7d245640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:26.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.292+0000 7fcd7d245640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:56:26.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.292+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcd781c4cf0 con 0x7fcd781377e0 2024-09-11T02:56:26.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd5f7fe640 1 -- 172.21.15.98:0/43895677 <== mon.5 v2:172.21.15.190:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fcd6c00d5b0 con 0x7fcd781377e0 2024-09-11T02:56:26.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:26.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd5f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:26.296 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":56,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:56:26.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 10 librados: watch_flush enter 2024-09-11T02:56:26.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 10 librados: watch_flush exit 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 20 client.19841.objecter shutdown clearing up homeless session... 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 10 client.19841.objecter successfully canceled tick 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd64040220 msgr2=0x7fcd640426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd64040220 0x7fcd640426e0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7fcd68006d50 tx=0x7fcd68006be0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 10 monclient: shutdown 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd781377e0 msgr2=0x7fcd78137c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd781377e0 0x7fcd78137c20 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0af000 tx=0x7fcd6c00b7e0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 shutdown_connections 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd64040220 0x7fcd640426e0 unknown :-1 s=CLOSED pgs=3495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd781d0ea0 0x7fcd781d1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd781377e0 0x7fcd78137c20 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 --2- 172.21.15.98:0/43895677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd781c4910 0x7fcd781226e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 >> 172.21.15.98:0/43895677 conn(0x7fcd780ac850 msgr2=0x7fcd780a5a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:26.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 shutdown_connections 2024-09-11T02:56:26.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 -- 172.21.15.98:0/43895677 wait complete. 2024-09-11T02:56:26.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.296+0000 7fcd7d245640 1 librados: shutdown 2024-09-11T02:56:26.315 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:56:26.315 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:56:26.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient: get_monmap_and_config 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient: build_initial_monmap 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient: monmap: 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:26.435634+0000 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:26.435634+0000 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.434 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:26.435 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 Processor -- start 2024-09-11T02:56:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 -- start start 2024-09-11T02:56:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient: init 2024-09-11T02:56:26.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient: _add_conns ranks=[4,2,7,8,10,0,3,1,9,5,6] 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c139b80 0x7f505c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): picked mon.noname-c con 0x7f505c139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 0x7f505c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): picked mon.noname-b con 0x7f505c06fce0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f505c0724b0 0x7f505c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): picked mon.noname-i con 0x7f505c0724b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f505c071850 con 0x7f505c06fce0 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f505c071aa0 con 0x7f505c139b80 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f505c070160 con 0x7f505c0724b0 2024-09-11T02:56:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.432+0000 7f5064724640 10 monclient(hunting): authenticate will time out at 1340.315877s 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c139b80 0x7f505c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 0x7f505c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c139b80 0x7f505c139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:49136/0 (socket says 172.21.15.98:49136) 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 -- 172.21.15.98:0/2194257974 learned_addr learned my addr 172.21.15.98:0/2194257974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): get_auth_request con 0x7f505c139b80 auth_method 0 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 10 monclient(hunting): get_auth_request con 0x7f505c06fce0 auth_method 0 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 1 -- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 msgr2=0x7f505c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 1 -- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 msgr2=0x7f505c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 0x7f505c0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5061cbf640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 0x7f505c0a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5062cc1640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f505c0724b0 0x7f505c0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient(hunting): handle_auth_done global_id 23665 payload 274 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 -- 172.21.15.98:0/2194257974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f505c0724b0 msgr2=0x7f505c0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f505c0724b0 0x7f505c0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 -- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 msgr2=0x7f505c0a7a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 0x7f505c0a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient: _finish_hunting 0 2024-09-11T02:56:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 monclient: found mon.noname-c 2024-09-11T02:56:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 -- 172.21.15.98:0/2194257974 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505c0a94a0 con 0x7f505c139b80 2024-09-11T02:56:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5062cc1640 10 monclient: get_auth_request con 0x7f505c0724b0 auth_method 0 2024-09-11T02:56:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5062cc1640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f505c0724b0 0x7f505c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50624c0640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c139b80 0x7f505c139f60 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f50540098d0 tx=0x7f505401c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1e9a006926ba7c0a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 1 -- 172.21.15.98:0/2194257974 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50540b2070 con 0x7f505c139b80 2024-09-11T02:56:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: dump: 2024-09-11T02:56:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: _finish_auth 0 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: _check_auth_tickets 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 1 -- 172.21.15.98:0/2194257974 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50540ab7f0 con 0x7f505c139b80 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 1 -- 172.21.15.98:0/2194257974 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50540abe50 con 0x7f505c139b80 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f50614be640 10 monclient: dump: 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.451 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 5 monclient: authenticate success, global_id 23665 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 4 set_mon_vals no callback set 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 10 monclient: shutdown 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 -- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c139b80 msgr2=0x7f505c139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c139b80 0x7f505c139f60 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f50540098d0 tx=0x7f505401c440 comp rx=0 tx=0).stop 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 -- 172.21.15.98:0/2194257974 shutdown_connections 2024-09-11T02:56:26.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f505c0724b0 0x7f505c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f505c06fce0 0x7f505c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 --2- 172.21.15.98:0/2194257974 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c139b80 0x7f505c139f60 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 -- 172.21.15.98:0/2194257974 >> 172.21.15.98:0/2194257974 conn(0x7f505c0ac6b0 msgr2=0x7f505c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 -- 172.21.15.98:0/2194257974 shutdown_connections 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.436+0000 7f5064724640 1 -- 172.21.15.98:0/2194257974 wait complete. 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: build_initial_monmap 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: monmap: 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:26.453 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 librados: starting msgr at 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 librados: starting objecter 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 Processor -- start 2024-09-11T02:56:26.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 -- start start 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 librados: setting wanted keys 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 librados: calling monclient init 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: init 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: _add_conns ranks=[6,4,1,8,3,5,2,0,7] 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 0x7f505c0a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): picked mon.noname-d con 0x7f505c06fce0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c0724b0 0x7f505c0a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): picked mon.noname-c con 0x7f505c0724b0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c139b80 0x7f505c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): picked mon.noname-f con 0x7f505c139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:26.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f505c143440 con 0x7f505c139b80 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f505c0a88a0 con 0x7f505c0724b0 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f505c1435c0 con 0x7f505c06fce0 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): authenticate will time out at 1340.321488s 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c0724b0 0x7f505c0a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c0724b0 0x7f505c0a4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:49142/0 (socket says 172.21.15.98:49142) 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 -- 172.21.15.98:0/4058011620 learned_addr learned my addr 172.21.15.98:0/4058011620 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): get_auth_request con 0x7f505c0724b0 auth_method 0 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 0x7f505c0a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient(hunting): get_auth_request con 0x7f505c06fce0 auth_method 0 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:26.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 msgr2=0x7f505c0a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T02:56:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 msgr2=0x7f505c0a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 0x7f505c0a4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:26.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 0x7f505c0a4030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5062cc1640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c139b80 0x7f505c0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient(hunting): handle_auth_done global_id 23674 payload 824 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 msgr2=0x7f505c0a4030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f505c06fce0 0x7f505c0a4030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c139b80 msgr2=0x7f505c0a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c139b80 0x7f505c0a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient: _finish_hunting 0 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 monclient: found mon.noname-c 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 -- 172.21.15.98:0/4058011620 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505c13c090 con 0x7f505c0724b0 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5062cc1640 10 monclient: get_auth_request con 0x7f505c139b80 auth_method 0 2024-09-11T02:56:26.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5062cc1640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f505c139b80 0x7f505c0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5061cbf640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c0724b0 0x7f505c0a4570 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f5050007e70 tx=0x7f505000e4b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 client.?.objecter ms_handle_connect 0x7f505c0724b0 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5050014890 con 0x7f505c0724b0 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: dump: 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: _finish_auth 0 2024-09-11T02:56:26.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5050019070 con 0x7f505c0724b0 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5050013380 con 0x7f505c0724b0 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 monclient: dump: 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:26.464 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5060cbd640 4 set_mon_vals no callback set 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 5 monclient: authenticate success, global_id 23674 2024-09-11T02:56:26.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: _renew_subs 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505c13ae50 con 0x7f505c0724b0 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 client.23674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: _renew_subs 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f505c0a7be0 con 0x7f505c0724b0 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 librados: init done 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:26.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f505c06ca10 con 0x7f505c1d16c0 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient: get_auth_request con 0x7f505c1d16c0 auth_method 0 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5050014420 con 0x7f505c0724b0 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f504803df40 0x7f5048040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(289..289 src has 1..289) ==== 6648+0+0 (secure 0 0 0) 0x7f5050050180 con 0x7f505c0724b0 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 10 client.23674.objecter ms_dispatch 0x7f505c000ba0 osd_map(289..289 src has 1..289) 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 3 client.23674.objecter handle_osd_map got epochs [289,289] > 0 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.440+0000 7f505affd640 3 client.23674.objecter handle_osd_map decoding full epoch 289 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.444+0000 7f505affd640 20 client.23674.objecter dump_active .. 0 homeless 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.444+0000 7f5062cc1640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f504803df40 0x7f5048040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.444+0000 7f5062cc1640 10 monclient: get_auth_request con 0x7f504803df40 auth_method 0 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.444+0000 7f505affd640 10 client.23674.objecter ms_handle_connect 0x7f504803df40 2024-09-11T02:56:26.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.444+0000 7f5062cc1640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f504803df40 0x7f5048040400 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f505c13a930 tx=0x7f504c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:26.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:26.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 10 monclient: get_auth_request con 0x7f505c1d16c0 auth_method 0 2024-09-11T02:56:26.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:26.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:26.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:26.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T02:56:26.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:26.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:26.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:26.644+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:27.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.048+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:27.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.048+0000 7f50624c0640 10 monclient: get_auth_request con 0x7f505c1d16c0 auth_method 0 2024-09-11T02:56:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.052+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.052+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:27.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.052+0000 7f50624c0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.052+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T02:56:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.052+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.052+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:27.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.052+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 10 monclient: get_auth_request con 0x7f505c1d16c0 auth_method 0 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:27.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:27.856+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.440+0000 7f5059ffb640 10 monclient: tick 2024-09-11T02:56:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.440+0000 7f5059ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.440+0000 7f5059ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:29.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.440+0000 7f5059ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient: get_auth_request con 0x7f505c1d16c0 auth_method 0 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 10 monclient(hunting): handle_auth_done global_id 44109 payload 824 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f505c140580 con 0x7f505c1d16c0 2024-09-11T02:56:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 10 client.23674.objecter ms_handle_connect 0x7f505c1d16c0 2024-09-11T02:56:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 10 client.23674.objecter resend_mon_ops 2024-09-11T02:56:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f50540098a0 tx=0x7f50540191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50540b2040 con 0x7f505c1d16c0 2024-09-11T02:56:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 10 monclient: dump: 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5054004410 con 0x7f505c1d16c0 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f50540098a0 tx=0x7f50540191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 msgr2=0x7f505c1d1aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.460+0000 7f505affd640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c1d16c0 0x7f505c1d1aa0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f50540098a0 tx=0x7f50540191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f5064724640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f5064724640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c133760 0x7f505c0a9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f5064724640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f505c06e870 con 0x7f505c133760 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c133760 0x7f505c0a9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient: get_auth_request con 0x7f505c133760 auth_method 0 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 10 monclient(hunting): handle_auth_done global_id 44118 payload 824 2024-09-11T02:56:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 1 -- 172.21.15.98:0/4058011620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f505c0a7890 con 0x7f505c133760 2024-09-11T02:56:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f50624c0640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c133760 0x7f505c0a9f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f50540097c0 tx=0x7f50540a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f505affd640 10 client.23674.objecter ms_handle_connect 0x7f505c133760 2024-09-11T02:56:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.468+0000 7f505affd640 10 client.23674.objecter resend_mon_ops 2024-09-11T02:56:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50540b2040 con 0x7f505c133760 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 10 monclient: dump: 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f50540ad730 con 0x7f505c133760 2024-09-11T02:56:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c133760 msgr2=0x7f505c0a9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c133760 0x7f505c0a9f00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f50540097c0 tx=0x7f50540a9000 comp rx=0 tx=0).stop 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c133760 msgr2=0x7f505c0a9f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f505affd640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f505c133760 0x7f505c0a9f00 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f50540097c0 tx=0x7f50540a9000 comp rx=0 tx=0).stop 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:29.475 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 15907, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.476 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.477 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.478 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.479 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:29.480 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:29.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:29.482 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.483 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:29.484 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.485 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.489 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:29.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.497 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:29.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.502 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:29.503 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:29.503 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:29.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.503 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:29.503 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 10 librados: watch_flush enter 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 10 librados: watch_flush exit 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 20 client.23674.objecter shutdown clearing up homeless session... 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 10 client.23674.objecter successfully canceled tick 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f504803df40 msgr2=0x7f5048040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f504803df40 0x7f5048040400 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f505c13a930 tx=0x7f504c008040 comp rx=0 tx=0).stop 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 10 monclient: shutdown 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:29.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c0724b0 msgr2=0x7f505c0a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 --2- 172.21.15.98:0/4058011620 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f505c0724b0 0x7f505c0a4570 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f5050007e70 tx=0x7f505000e4b0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5062cc1640 1 -- 172.21.15.98:0/4058011620 reap_dead start 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5062cc1640 1 -- 172.21.15.98:0/4058011620 reap_dead start 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 shutdown_connections 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 >> 172.21.15.98:0/4058011620 conn(0x7f505c0ac6b0 msgr2=0x7f505c134ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 shutdown_connections 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 -- 172.21.15.98:0/4058011620 wait complete. 2024-09-11T02:56:29.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.472+0000 7f5064724640 1 librados: shutdown 2024-09-11T02:56:29.506 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:29.507 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:56:29.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:56:29.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient: get_monmap_and_config 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient: build_initial_monmap 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient: monmap: 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:29.629431+0000 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:29.629431+0000 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:29.628 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:29.629 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 Processor -- start 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 -- start start 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient: init 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient: _add_conns ranks=[5,2,6,3,4,9,0,7,10,8,1] 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f309c0a46a0 0x7f309c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): picked mon.noname-h con 0x7f309c0a46a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a4fc0 0x7f309c139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): picked mon.noname-b con 0x7f309c0a4fc0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f309c139e70 0x7f309c13a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): picked mon.noname-d con 0x7f309c139e70 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f309c0a0750 con 0x7f309c0a4fc0 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f309c0a17b0 con 0x7f309c0a46a0 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f309c0775d0 con 0x7f309c139e70 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a2876640 10 monclient(hunting): authenticate will time out at 1343.509881s 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a4fc0 0x7f309c139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a4fc0 0x7f309c139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:35568/0 (socket says 172.21.15.98:35568) 2024-09-11T02:56:29.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 -- 172.21.15.98:0/3451798531 learned_addr learned my addr 172.21.15.98:0/3451798531 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): get_auth_request con 0x7f309c0a4fc0 auth_method 0 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f309c139e70 0x7f309c13a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 10 monclient(hunting): get_auth_request con 0x7f309c139e70 auth_method 0 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309bfff640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f309c0a46a0 0x7f309c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309bfff640 10 monclient(hunting): get_auth_request con 0x7f309c0a46a0 auth_method 0 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f30a0e13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient(hunting): handle_auth_done global_id 44102 payload 274 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 -- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f309c139e70 msgr2=0x7f309c13a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f309c139e70 0x7f309c13a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 -- 172.21.15.98:0/3451798531 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f309c0a46a0 msgr2=0x7f309c0a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f309c0a46a0 0x7f309c0a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 monclient: found mon.noname-b 2024-09-11T02:56:29.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.628+0000 7f309b7fe640 1 -- 172.21.15.98:0/3451798531 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309c0a0200 con 0x7f309c0a4fc0 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a4fc0 0x7f309c139930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30900098d0 tx=0x7f309001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa3f6667a7bf9b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 1 -- 172.21.15.98:0/3451798531 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f30900b0070 con 0x7f309c0a4fc0 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: dump: 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.636 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: _finish_auth 0 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:29.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 1 -- 172.21.15.98:0/3451798531 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30900aa810 con 0x7f309c0a4fc0 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 1 -- 172.21.15.98:0/3451798531 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f30900a9300 con 0x7f309c0a4fc0 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 5 monclient: authenticate success, global_id 44102 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 4 set_mon_vals no callback set 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309affd640 10 monclient: dump: 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.638 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.639 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient: shutdown 2024-09-11T02:56:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a4fc0 msgr2=0x7f309c139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a4fc0 0x7f309c139930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30900098d0 tx=0x7f309001c440 comp rx=0 tx=0).stop 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- 172.21.15.98:0/3451798531 shutdown_connections 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f309c139e70 0x7f309c13a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a4fc0 0x7f309c139930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 --2- 172.21.15.98:0/3451798531 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f309c0a46a0 0x7f309c0a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- 172.21.15.98:0/3451798531 >> 172.21.15.98:0/3451798531 conn(0x7f309c06ed00 msgr2=0x7f309c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- 172.21.15.98:0/3451798531 shutdown_connections 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- 172.21.15.98:0/3451798531 wait complete. 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient: build_initial_monmap 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient: monmap: 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 librados: starting msgr at 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 librados: starting objecter 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 Processor -- start 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- start start 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 librados: setting wanted keys 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 librados: calling monclient init 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient: init 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient: _add_conns ranks=[2,0,8,6,4,3,5,1,7] 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a46a0 0x7f309c136c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): picked mon.noname-b con 0x7f309c0a46a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:29.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f309c0a4fc0 0x7f309c137140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): picked mon.noname-a con 0x7f309c0a4fc0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f309c139e70 0x7f309c1d1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): picked mon.noname-e con 0x7f309c139e70 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f309c0779d0 con 0x7f309c0a4fc0 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f309c077850 con 0x7f309c0a46a0 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f309c143410 con 0x7f309c139e70 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a2876640 10 monclient(hunting): authenticate will time out at 1343.515412s 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a46a0 0x7f309c136c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f309c0a4fc0 0x7f309c137140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f309c139e70 0x7f309c1d1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a46a0 0x7f309c136c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:35582/0 (socket says 172.21.15.98:35582) 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 -- 172.21.15.98:0/4084044521 learned_addr learned my addr 172.21.15.98:0/4084044521 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): get_auth_request con 0x7f309c0a46a0 auth_method 0 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 10 monclient(hunting): get_auth_request con 0x7f309c0a4fc0 auth_method 0 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 10 monclient(hunting): get_auth_request con 0x7f309c139e70 auth_method 0 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f30a0e13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient(hunting): handle_auth_done global_id 44111 payload 824 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f309c139e70 msgr2=0x7f309c1d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f309c139e70 0x7f309c1d1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f309c0a4fc0 msgr2=0x7f309c137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f309c0a4fc0 0x7f309c137140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 monclient: found mon.noname-b 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309bfff640 1 -- 172.21.15.98:0/4084044521 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309c06f170 con 0x7f309c0a46a0 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.632+0000 7f309b7fe640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f309c0a4fc0 0x7f309c137140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309bfff640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a46a0 0x7f309c136c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3094007f20 tx=0x7f309400e570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 client.?.objecter ms_handle_connect 0x7f309c0a46a0 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3094014890 con 0x7f309c0a46a0 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: dump: 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: _finish_auth 0 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3094019070 con 0x7f309c0a46a0 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3094013380 con 0x7f309c0a46a0 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 5 monclient: authenticate success, global_id 44111 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 10 monclient: _renew_subs 2024-09-11T02:56:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309c0a87d0 con 0x7f309c0a46a0 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309a7fc640 4 set_mon_vals no callback set 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: dump: 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 10 client.44111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 10 monclient: _renew_subs 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f309c137680 con 0x7f309c0a46a0 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 1 librados: init done 2024-09-11T02:56:29.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02d450 0x7f309c02d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f309c06ca10 con 0x7f309c02d450 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3094014420 con 0x7f309c0a46a0 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02d450 0x7f309c02d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient: get_auth_request con 0x7f309c02d450 auth_method 0 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f308803df40 0x7f3088040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 10 monclient(hunting): handle_auth_done global_id 44120 payload 824 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 1 -- 172.21.15.98:0/4084044521 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f309c13e530 con 0x7f309c02d450 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(293..293 src has 1..293) ==== 6194+0+0 (secure 0 0 0) 0x7f309404fd60 con 0x7f309c0a46a0 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 client.44111.objecter ms_dispatch 0x7f309c000ba0 osd_map(293..293 src has 1..293) 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 3 client.44111.objecter handle_osd_map got epochs [293,293] > 0 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 3 client.44111.objecter handle_osd_map decoding full epoch 293 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 20 client.44111.objecter dump_active .. 0 homeless 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 client.44111.objecter ms_handle_connect 0x7f309c02d450 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 client.44111.objecter resend_mon_ops 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f309b7fe640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02d450 0x7f309c02d830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30900098a0 tx=0x7f30900191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a0e13640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f308803df40 0x7f3088040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f30900ba070 con 0x7f309c02d450 2024-09-11T02:56:29.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: dump: 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a0e13640 10 monclient: get_auth_request con 0x7f308803df40 auth_method 0 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f30900aa750 con 0x7f309c02d450 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02d450 msgr2=0x7f309c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02d450 0x7f309c02d830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30900098a0 tx=0x7f30900191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02d450 msgr2=0x7f309c02d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02d450 0x7f309c02d830 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30900098a0 tx=0x7f30900191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f30a0e13640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f308803df40 0x7f3088040400 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f309c070450 tx=0x7f308c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.636+0000 7f3098ff9640 10 client.44111.objecter ms_handle_connect 0x7f308803df40 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02dd70 0x7f309c02e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f309c06e870 con 0x7f309c02dd70 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02dd70 0x7f309c02e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient: get_auth_request con 0x7f309c02dd70 auth_method 0 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 10 monclient(hunting): handle_auth_done global_id 44129 payload 824 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 1 -- 172.21.15.98:0/4084044521 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f309c06f3a0 con 0x7f309c02dd70 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 10 client.44111.objecter ms_handle_connect 0x7f309c02dd70 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 10 client.44111.objecter resend_mon_ops 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f309b7fe640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02dd70 0x7f309c02e150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f309001c270 tx=0x7f30900a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f30900b0040 con 0x7f309c02dd70 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 10 monclient: dump: 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f30900bfd30 con 0x7f309c02dd70 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:29.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02dd70 msgr2=0x7f309c02e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02dd70 0x7f309c02e150 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f309001c270 tx=0x7f30900a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02dd70 msgr2=0x7f309c02e150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f3098ff9640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c02dd70 0x7f309c02e150 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f309001c270 tx=0x7f30900a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:29.665 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16079, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:29.666 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.667 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.668 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:29.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.670 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.671 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.672 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:29.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.676 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:29.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:29.681 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.683 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.684 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:29.685 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:29.687 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:29.688 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 10 librados: watch_flush enter 2024-09-11T02:56:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 10 librados: watch_flush exit 2024-09-11T02:56:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 20 client.44111.objecter shutdown clearing up homeless session... 2024-09-11T02:56:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 10 client.44111.objecter successfully canceled tick 2024-09-11T02:56:29.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f308803df40 msgr2=0x7f3088040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f308803df40 0x7f3088040400 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f309c070450 tx=0x7f308c009190 comp rx=0 tx=0).stop 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 10 monclient: shutdown 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a46a0 msgr2=0x7f309c136c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 --2- 172.21.15.98:0/4084044521 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f309c0a46a0 0x7f309c136c00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3094007f20 tx=0x7f309400e570 comp rx=0 tx=0).stop 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a0e13640 1 -- 172.21.15.98:0/4084044521 reap_dead start 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a0e13640 1 -- 172.21.15.98:0/4084044521 reap_dead start 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 shutdown_connections 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 >> 172.21.15.98:0/4084044521 conn(0x7f309c06ed00 msgr2=0x7f309c1402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 shutdown_connections 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 -- 172.21.15.98:0/4084044521 wait complete. 2024-09-11T02:56:29.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.644+0000 7f30a2876640 1 librados: shutdown 2024-09-11T02:56:29.692 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:29.692 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:56:29.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:56:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient: get_monmap_and_config 2024-09-11T02:56:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient: build_initial_monmap 2024-09-11T02:56:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient: monmap: 2024-09-11T02:56:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:29.809969+0000 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:29.809969+0000 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:29.810 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 1 Processor -- start 2024-09-11T02:56:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 1 -- start start 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient: init 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient: _add_conns ranks=[0,1,6,3,4,2,8,7,5,10,9] 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f960c0a46a0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f960c0a5190 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c140300 0x7f960c1426f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient(hunting): picked mon.noname-d con 0x7f960c140300 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960c0a17b0 con 0x7f960c0a46a0 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960c0775d0 con 0x7f960c0a5190 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f960c0a0750 con 0x7f960c140300 2024-09-11T02:56:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f961182f640 10 monclient(hunting): authenticate will time out at 1343.690545s 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f960c0a46a0 0x7f960c0a4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54370/0 (socket says 172.21.15.98:54370) 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c140300 0x7f960c1426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960affd640 1 -- 172.21.15.98:0/1053601780 learned_addr learned my addr 172.21.15.98:0/1053601780 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): get_auth_request con 0x7f960c140300 auth_method 0 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f9609ffb640 1 -- 172.21.15.98:0/1053601780 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2574737825 0 0) 0x7f960c0a17b0 con 0x7f960c0a46a0 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f9609ffb640 10 monclient(hunting): _negotiate 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f9609ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f9609ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f9609ffb640 10 monclient(hunting): my global_id is 44136 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f9609ffb640 1 -- 172.21.15.98:0/1053601780 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95f4003530 con 0x7f960c0a46a0 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient(hunting): handle_auth_done global_id 44124 payload 274 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 1 -- 172.21.15.98:0/1053601780 >> v1:172.21.15.190:6789/0 conn(0x7f960c0a5190 legacy=0x7f960c13ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 1 -- 172.21.15.98:0/1053601780 >> v1:172.21.15.98:6789/0 conn(0x7f960c0a46a0 legacy=0x7f960c0a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 1 monclient: found mon.noname-d 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:56:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.808+0000 7f960b7fe640 1 -- 172.21.15.98:0/1053601780 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c0a0200 con 0x7f960c140300 2024-09-11T02:56:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f960b7fe640 1 --2- 172.21.15.98:0/1053601780 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c140300 0x7f960c1426f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9604009dc0 tx=0x7f960401c850 comp rx=0 tx=0).ready entity=mon.6 client_cookie=acf8bbb747f7a683 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 1 -- 172.21.15.98:0/1053601780 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f96040a6880 con 0x7f960c140300 2024-09-11T02:56:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: dump: 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 5 monclient: authenticate success, global_id 44124 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 1 -- 172.21.15.98:0/1053601780 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96040a6a40 con 0x7f960c140300 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 1 -- 172.21.15.98:0/1053601780 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f96040a6bf0 con 0x7f960c140300 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 4 set_mon_vals no callback set 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f9609ffb640 10 monclient: dump: 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.827 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 10 monclient: shutdown 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 -- 172.21.15.98:0/1053601780 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c140300 msgr2=0x7f960c1426f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 --2- 172.21.15.98:0/1053601780 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c140300 0x7f960c1426f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9604009dc0 tx=0x7f960401c850 comp rx=0 tx=0).stop 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 -- 172.21.15.98:0/1053601780 shutdown_connections 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 --2- 172.21.15.98:0/1053601780 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c140300 0x7f960c1426f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 -- 172.21.15.98:0/1053601780 >> 172.21.15.98:0/1053601780 conn(0x7f960c06ed00 msgr2=0x7f960c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 -- 172.21.15.98:0/1053601780 shutdown_connections 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 -- 172.21.15.98:0/1053601780 wait complete. 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 10 monclient: build_initial_monmap 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 10 monclient: monmap: 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:29.828 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:29.829 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 librados: starting msgr at 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.812+0000 7f961182f640 1 librados: starting objecter 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 Processor -- start 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 -- start start 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 librados: setting wanted keys 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 librados: calling monclient init 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient: init 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient: _add_conns ranks=[6,8,3,0,5,7,4,1,2] 2024-09-11T02:56:29.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c0a46a0 0x7f960c12fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): picked mon.noname-d con 0x7f960c0a46a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f960c0a5190 0x7f960c1d1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): picked mon.noname-e con 0x7f960c0a5190 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f960c140300 0x7f960c1d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): picked mon.noname-g con 0x7f960c140300 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f960c1455d0 con 0x7f960c140300 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f960c145450 con 0x7f960c0a46a0 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f960c145750 con 0x7f960c0a5190 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:29.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): authenticate will time out at 1343.696686s 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f960c0a5190 0x7f960c1d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c0a46a0 0x7f960c12fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f960c0a5190 0x7f960c1d1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44578/0 (socket says 172.21.15.98:44578) 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c0a46a0 0x7f960c12fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:51970/0 (socket says 172.21.15.98:51970) 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 -- 172.21.15.98:0/1474366055 learned_addr learned my addr 172.21.15.98:0/1474366055 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): get_auth_request con 0x7f960c0a5190 auth_method 0 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): get_auth_request con 0x7f960c0a46a0 auth_method 0 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f960c140300 0x7f960c1d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 10 monclient(hunting): get_auth_request con 0x7f960c140300 auth_method 0 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient(hunting): handle_auth_done global_id 24182 payload 824 2024-09-11T02:56:29.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c0a46a0 msgr2=0x7f960c12fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c0a46a0 0x7f960c12fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f960c140300 msgr2=0x7f960c1d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f960c140300 0x7f960c1d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 monclient: found mon.noname-e 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 -- 172.21.15.98:0/1474366055 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c072de0 con 0x7f960c0a5190 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f960c140300 0x7f960c1d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f960c0a46a0 0x7f960c12fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960a7fc640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f960c0a5190 0x7f960c1d1650 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f960000c4e0 tx=0x7f960000c9a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 client.?.objecter ms_handle_connect 0x7f960c0a5190 2024-09-11T02:56:29.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9600012630 con 0x7f960c0a5190 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: dump: 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.835 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.836 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: _finish_auth 0 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9600012810 con 0x7f960c0a5190 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 5 monclient: authenticate success, global_id 24182 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient: _renew_subs 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960c071e80 con 0x7f960c0a5190 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9600012e30 con 0x7f960c0a5190 2024-09-11T02:56:29.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: dump: 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.838 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 client.24182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient: _renew_subs 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f960c1306c0 con 0x7f960c0a5190 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 librados: init done 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f96097fa640 4 set_mon_vals no callback set 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c1d20d0 0x7f960c0a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f960c06ca10 con 0x7f960c1d20d0 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c1d20d0 0x7f960c0a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient: get_auth_request con 0x7f960c1d20d0 auth_method 0 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f96000129c0 con 0x7f960c0a5190 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 10 monclient(hunting): handle_auth_done global_id 23692 payload 824 2024-09-11T02:56:29.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 1 -- 172.21.15.98:0/1474366055 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f960c0a4090 con 0x7f960c1d20d0 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f95f803df40 0x7f95f8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(293..293 src has 1..293) ==== 6194+0+0 (secure 0 0 0) 0x7f960004f160 con 0x7f960c0a5190 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 client.24182.objecter ms_dispatch 0x7f960c000ba0 osd_map(293..293 src has 1..293) 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 3 client.24182.objecter handle_osd_map got epochs [293,293] > 0 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 3 client.24182.objecter handle_osd_map decoding full epoch 293 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960affd640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c1d20d0 0x7f960c0a7450 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005890 tx=0x7f95fc007770 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 20 client.24182.objecter dump_active .. 0 homeless 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 client.24182.objecter ms_handle_connect 0x7f960c1d20d0 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 client.24182.objecter resend_mon_ops 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95fc00f770 con 0x7f960c1d20d0 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:29.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: dump: 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.842 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f95f803df40 0x7f95f8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f960b7fe640 10 monclient: get_auth_request con 0x7f95f803df40 auth_method 0 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f95fc018070 con 0x7f960c1d20d0 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c1d20d0 msgr2=0x7f960c0a7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c1d20d0 0x7f960c0a7450 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005890 tx=0x7f95fc007770 comp rx=0 tx=0).stop 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c1d20d0 msgr2=0x7f960c0a7450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:29.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.816+0000 7f95f3fff640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c1d20d0 0x7f960c0a7450 secure :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005890 tx=0x7f95fc007770 comp rx=0 tx=0).stop 2024-09-11T02:56:29.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.820+0000 7f95f3fff640 10 client.24182.objecter ms_handle_connect 0x7f95f803df40 2024-09-11T02:56:29.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.820+0000 7f960b7fe640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f95f803df40 0x7f95f8040400 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7f96040af000 tx=0x7f96040a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:29.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c0a7990 0x7f960c0a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:29.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:29.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f960c06e870 con 0x7f960c0a7990 2024-09-11T02:56:29.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c0a7990 0x7f960c0a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient: get_auth_request con 0x7f960c0a7990 auth_method 0 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 10 monclient(hunting): handle_auth_done global_id 23701 payload 824 2024-09-11T02:56:29.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 1 -- 172.21.15.98:0/1474366055 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f960c1d24b0 con 0x7f960c0a7990 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960affd640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c0a7990 0x7f960c0a7da0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005890 tx=0x7f95fc005800 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 10 client.24182.objecter ms_handle_connect 0x7f960c0a7990 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 10 client.24182.objecter resend_mon_ops 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95fc006050 con 0x7f960c0a7990 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 10 monclient: dump: 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:29.847 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f95fc00fac0 con 0x7f960c0a7990 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:29.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c0a7990 msgr2=0x7f960c0a7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c0a7990 0x7f960c0a7da0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005890 tx=0x7f95fc005800 comp rx=0 tx=0).stop 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c0a7990 msgr2=0x7f960c0a7da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f95f3fff640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f960c0a7990 0x7f960c0a7da0 secure :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f95fc005890 tx=0x7f95fc005800 comp rx=0 tx=0).stop 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:29.849 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 405608, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.850 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.851 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.852 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:29.854 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.855 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:29.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:29.856 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:29.856 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.857 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.861 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.863 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.864 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.867 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:29.868 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.870 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.872 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:29.873 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:29.874 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 10 librados: watch_flush enter 2024-09-11T02:56:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 10 librados: watch_flush exit 2024-09-11T02:56:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 20 client.24182.objecter shutdown clearing up homeless session... 2024-09-11T02:56:29.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 10 client.24182.objecter successfully canceled tick 2024-09-11T02:56:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f95f803df40 msgr2=0x7f95f8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f95f803df40 0x7f95f8040400 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7f96040af000 tx=0x7f96040a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 10 monclient: shutdown 2024-09-11T02:56:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f960c0a5190 msgr2=0x7f960c1d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 --2- 172.21.15.98:0/1474366055 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f960c0a5190 0x7f960c1d1650 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f960000c4e0 tx=0x7f960000c9a0 comp rx=0 tx=0).stop 2024-09-11T02:56:29.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960b7fe640 1 -- 172.21.15.98:0/1474366055 reap_dead start 2024-09-11T02:56:29.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f960b7fe640 1 -- 172.21.15.98:0/1474366055 reap_dead start 2024-09-11T02:56:29.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 shutdown_connections 2024-09-11T02:56:29.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 >> 172.21.15.98:0/1474366055 conn(0x7f960c06ed00 msgr2=0x7f960c0a2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:29.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 shutdown_connections 2024-09-11T02:56:29.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 -- 172.21.15.98:0/1474366055 wait complete. 2024-09-11T02:56:29.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.824+0000 7f961182f640 1 librados: shutdown 2024-09-11T02:56:29.878 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:29.878 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:56:29.879 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:56:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.996+0000 7f223c870640 10 monclient: get_monmap_and_config 2024-09-11T02:56:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.996+0000 7f223c870640 10 monclient: build_initial_monmap 2024-09-11T02:56:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:29.996+0000 7f223c870640 10 monclient: monmap: 2024-09-11T02:56:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:30.000958+0000 2024-09-11T02:56:29.999 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:30.000958+0000 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:30.000 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:30.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 1 Processor -- start 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 1 -- start start 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient: init 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient: _add_conns ranks=[1,4,3,9,7,5,8,10,2,0,6] 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f22380a3de0 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f22380a48d0 0x7f22380a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient(hunting): picked mon.noname-c con 0x7f22380a48d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:30.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f223813c9f0 0x7f223813ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient(hunting): picked mon.noname-g con 0x7f223813c9f0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2238068fa0 con 0x7f22380a3de0 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f22380a0170 con 0x7f223813c9f0 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2238068b40 con 0x7f22380a48d0 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f223c870640 10 monclient(hunting): authenticate will time out at 1343.882426s 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f22380a48d0 0x7f22380a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f22380a48d0 0x7f22380a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38138/0 (socket says 172.21.15.98:38138) 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 -- 172.21.15.98:0/3455691519 learned_addr learned my addr 172.21.15.98:0/3455691519 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): get_auth_request con 0x7f22380a48d0 auth_method 0 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2236d9d640 1 --2- 172.21.15.98:0/3455691519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f223813c9f0 0x7f223813ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2236d9d640 10 monclient(hunting): get_auth_request con 0x7f223813c9f0 auth_method 0 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2236d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2236d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2236d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient(hunting): handle_auth_done global_id 23710 payload 274 2024-09-11T02:56:30.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 -- 172.21.15.98:0/3455691519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f223813c9f0 msgr2=0x7f223813ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 --2- 172.21.15.98:0/3455691519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f223813c9f0 0x7f223813ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 -- 172.21.15.98:0/3455691519 >> v1:172.21.15.190:6789/0 conn(0x7f22380a3de0 legacy=0x7f22380a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 monclient: found mon.noname-c 2024-09-11T02:56:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.000+0000 7f2235d9b640 1 -- 172.21.15.98:0/3455691519 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f223809eaf0 con 0x7f22380a48d0 2024-09-11T02:56:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f2235d9b640 1 --2- 172.21.15.98:0/3455691519 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f22380a48d0 0x7f22380a0570 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f222c0098a0 tx=0x7f222c01c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=21e95fc039759395 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 1 -- 172.21.15.98:0/3455691519 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c0b1070 con 0x7f22380a48d0 2024-09-11T02:56:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:30.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: dump: 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.013 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 5 monclient: authenticate success, global_id 23710 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 1 -- 172.21.15.98:0/3455691519 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222c0aa7b0 con 0x7f22380a48d0 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 4 set_mon_vals no callback set 2024-09-11T02:56:30.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 1 -- 172.21.15.98:0/3455691519 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c0aae10 con 0x7f22380a48d0 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223559a640 10 monclient: dump: 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.015 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 10 monclient: shutdown 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 -- 172.21.15.98:0/3455691519 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f22380a48d0 msgr2=0x7f22380a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 --2- 172.21.15.98:0/3455691519 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f22380a48d0 0x7f22380a0570 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f222c0098a0 tx=0x7f222c01c440 comp rx=0 tx=0).stop 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 -- 172.21.15.98:0/3455691519 shutdown_connections 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 --2- 172.21.15.98:0/3455691519 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f223813c9f0 0x7f223813ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 --2- 172.21.15.98:0/3455691519 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f22380a48d0 0x7f22380a0570 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 -- 172.21.15.98:0/3455691519 >> 172.21.15.98:0/3455691519 conn(0x7f22380a4ed0 msgr2=0x7f22380a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 -- 172.21.15.98:0/3455691519 shutdown_connections 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 -- 172.21.15.98:0/3455691519 wait complete. 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 10 monclient: monmap: 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:30.017 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 librados: starting msgr at 2024-09-11T02:56:30.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 librados: starting objecter 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 Processor -- start 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 -- start start 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 librados: setting wanted keys 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 1 librados: calling monclient init 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.004+0000 7f223c870640 10 monclient: init 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient: _add_conns ranks=[1,2,8,4,7,5,6,0,3] 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22380a3de0 0x7f223813b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): picked mon.noname-f con 0x7f22380a3de0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22380a48d0 0x7f2238137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): picked mon.noname-b con 0x7f22380a48d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f223813c9f0 0x7f22381d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): picked mon.noname-e con 0x7f223813c9f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2238141ed0 con 0x7f22380a3de0 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2238141d50 con 0x7f22380a48d0 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2238142050 con 0x7f223813c9f0 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): authenticate will time out at 1343.888373s 2024-09-11T02:56:30.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22380a48d0 0x7f2238137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f223813c9f0 0x7f22381d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22380a48d0 0x7f2238137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:35602/0 (socket says 172.21.15.98:35602) 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 -- 172.21.15.98:0/3432145948 learned_addr learned my addr 172.21.15.98:0/3432145948 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): get_auth_request con 0x7f22380a48d0 auth_method 0 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient(hunting): get_auth_request con 0x7f223813c9f0 auth_method 0 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22380a3de0 0x7f223813b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient(hunting): handle_auth_done global_id 44138 payload 824 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f223813c9f0 msgr2=0x7f22381d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f223813c9f0 0x7f22381d1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22380a3de0 msgr2=0x7f223813b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22380a3de0 0x7f223813b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 monclient: found mon.noname-b 2024-09-11T02:56:30.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 -- 172.21.15.98:0/3432145948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2238131a40 con 0x7f22380a48d0 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient: get_auth_request con 0x7f22380a3de0 auth_method 0 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f22380a3de0 0x7f223813b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2235d9b640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22380a48d0 0x7f2238137850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f222c0af040 tx=0x7f222c0191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 client.?.objecter ms_handle_connect 0x7f22380a48d0 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c0ba070 con 0x7f22380a48d0 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: dump: 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.023 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.024 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 5 monclient: authenticate success, global_id 44138 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient: _renew_subs 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223809d3f0 con 0x7f22380a48d0 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 client.44138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient: _renew_subs 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f223813c270 con 0x7f22380a48d0 2024-09-11T02:56:30.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 librados: init done 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222c0aa960 con 0x7f22380a48d0 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c0b1040 con 0x7f22380a48d0 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2234d99640 4 set_mon_vals no callback set 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: dump: 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.026 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2238137d90 0x7f22381d15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f223806ca10 con 0x7f2238137d90 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2238137d90 0x7f22381d15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient: get_auth_request con 0x7f2238137d90 auth_method 0 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 10 monclient(hunting): handle_auth_done global_id 44142 payload 824 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 1 -- 172.21.15.98:0/3432145948 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22380a0570 con 0x7f2238137d90 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f222c0aab10 con 0x7f22380a48d0 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f223659c640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2238137d90 0x7f22381d15a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f222800db80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f221803df40 0x7f2218040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 client.44138.objecter ms_handle_connect 0x7f2238137d90 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 client.44138.objecter resend_mon_ops 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(293..293 src has 1..293) ==== 6194+0+0 (secure 0 0 0) 0x7f222c0e8f20 con 0x7f22380a48d0 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 client.44138.objecter ms_dispatch 0x7f2238000ba0 osd_map(293..293 src has 1..293) 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 3 client.44138.objecter handle_osd_map got epochs [293,293] > 0 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 3 client.44138.objecter handle_osd_map decoding full epoch 293 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 20 client.44138.objecter dump_active .. 0 homeless 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222800b680 con 0x7f2238137d90 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: dump: 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.029 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.030 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f221803df40 0x7f2218040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 10 monclient: get_auth_request con 0x7f221803df40 auth_method 0 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2228005070 con 0x7f2238137d90 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2238137d90 msgr2=0x7f22381d15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2238137d90 0x7f22381d15a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f222800db80 comp rx=0 tx=0).stop 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2238137d90 msgr2=0x7f22381d15a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2238137d90 0x7f22381d15a0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f222800db80 comp rx=0 tx=0).stop 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2226ffd640 10 client.44138.objecter ms_handle_connect 0x7f221803df40 2024-09-11T02:56:30.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.008+0000 7f2236d9d640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f221803df40 0x7f2218040400 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f22380a62a0 tx=0x7f2230009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22381d1ae0 0x7f2238138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f22381d1ef0 con 0x7f22381d1ae0 2024-09-11T02:56:30.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22381d1ae0 0x7f2238138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient: get_auth_request con 0x7f22381d1ae0 auth_method 0 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 10 monclient(hunting): handle_auth_done global_id 44151 payload 824 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 1 -- 172.21.15.98:0/3432145948 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f223809da40 con 0x7f22381d1ae0 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 10 client.44138.objecter ms_handle_connect 0x7f22381d1ae0 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 10 client.44138.objecter resend_mon_ops 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223659c640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22381d1ae0 0x7f2238138740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f2228001130 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222801be10 con 0x7f22381d1ae0 2024-09-11T02:56:30.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 10 monclient: dump: 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.035 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f222801be10 con 0x7f22381d1ae0 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22381d1ae0 msgr2=0x7f2238138740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22381d1ae0 0x7f2238138740 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f2228001130 comp rx=0 tx=0).stop 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22381d1ae0 msgr2=0x7f2238138740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2226ffd640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f22381d1ae0 0x7f2238138740 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2228000f70 tx=0x7f2228001130 comp rx=0 tx=0).stop 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16443, 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.039 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:30.040 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.041 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.042 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:30.045 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:30.046 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.047 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.048 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.049 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.050 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.051 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.053 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.054 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:30.055 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.056 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.058 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.060 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.060 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.060 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.060 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:30.060 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 10 librados: watch_flush enter 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 10 librados: watch_flush exit 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 20 client.44138.objecter shutdown clearing up homeless session... 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 10 client.44138.objecter successfully canceled tick 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f221803df40 msgr2=0x7f2218040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f221803df40 0x7f2218040400 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f22380a62a0 tx=0x7f2230009780 comp rx=0 tx=0).stop 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 10 monclient: shutdown 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22380a48d0 msgr2=0x7f2238137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 --2- 172.21.15.98:0/3432145948 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22380a48d0 0x7f2238137850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f222c0af040 tx=0x7f222c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2236d9d640 1 -- 172.21.15.98:0/3432145948 reap_dead start 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f2236d9d640 1 -- 172.21.15.98:0/3432145948 reap_dead start 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 shutdown_connections 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 >> 172.21.15.98:0/3432145948 conn(0x7f22380a4ed0 msgr2=0x7f223813cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 shutdown_connections 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 -- 172.21.15.98:0/3432145948 wait complete. 2024-09-11T02:56:30.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.016+0000 7f223c870640 1 librados: shutdown 2024-09-11T02:56:30.063 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:30.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:56:30.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:56:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient: get_monmap_and_config 2024-09-11T02:56:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient: monmap: 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:30.182162+0000 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:30.182162+0000 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.181 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:30.182 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:30.182 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 Processor -- start 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 -- start start 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient: init 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient: _add_conns ranks=[6,2,5,4,0,8,9,7,3,10,1] 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25b8139b10 0x7f25b8072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): picked mon.noname-d con 0x7f25b8139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25b8139ef0 0x7f25b80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): picked mon.noname-b con 0x7f25b8139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f25b813a2d0 0x7f25b80a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): picked mon.noname-h con 0x7f25b813a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f25b80702b0 con 0x7f25b8139ef0 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f25b8071bf0 con 0x7f25b813a2d0 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f25b8071350 con 0x7f25b8139b10 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25bd4cc640 10 monclient(hunting): authenticate will time out at 1344.062410s 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25b8139ef0 0x7f25b80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25b8139b10 0x7f25b8072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25b8139ef0 0x7f25b80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:35618/0 (socket says 172.21.15.98:35618) 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25b8139b10 0x7f25b8072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:51996/0 (socket says 172.21.15.98:51996) 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 1 -- 172.21.15.98:0/2957825024 learned_addr learned my addr 172.21.15.98:0/2957825024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 10 monclient(hunting): get_auth_request con 0x7f25b8139ef0 auth_method 0 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): get_auth_request con 0x7f25b8139b10 auth_method 0 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b77fe640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f25b813a2d0 0x7f25b80a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b77fe640 10 monclient(hunting): get_auth_request con 0x7f25b813a2d0 auth_method 0 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_done global_id 44160 payload 274 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 -- 172.21.15.98:0/2957825024 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f25b813a2d0 msgr2=0x7f25b80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f25b813a2d0 0x7f25b80a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 -- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25b8139ef0 msgr2=0x7f25b80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25b8139ef0 0x7f25b80a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 monclient: found mon.noname-d 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b6ffd640 1 -- 172.21.15.98:0/2957825024 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b80a9620 con 0x7f25b8139b10 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.180+0000 7f25b67fc640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25b8139ef0 0x7f25b80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25b8139b10 0x7f25b8072600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0097f0 tx=0x7f25ac01c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c65abb90c4388538 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 1 -- 172.21.15.98:0/2957825024 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25ac0b0070 con 0x7f25b8139b10 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: dump: 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 1 -- 172.21.15.98:0/2957825024 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25ac0aa810 con 0x7f25b8139b10 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 5 monclient: authenticate success, global_id 44160 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 1 -- 172.21.15.98:0/2957825024 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25ac0a4e60 con 0x7f25b8139b10 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 4 set_mon_vals no callback set 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b5ffb640 10 monclient: dump: 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient: shutdown 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25b8139b10 msgr2=0x7f25b8072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25b8139b10 0x7f25b8072600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0097f0 tx=0x7f25ac01c3a0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- 172.21.15.98:0/2957825024 shutdown_connections 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f25b813a2d0 0x7f25b80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f25b8139ef0 0x7f25b80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/2957825024 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f25b8139b10 0x7f25b8072600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- 172.21.15.98:0/2957825024 >> 172.21.15.98:0/2957825024 conn(0x7f25b80ac790 msgr2=0x7f25b80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- 172.21.15.98:0/2957825024 shutdown_connections 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- 172.21.15.98:0/2957825024 wait complete. 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient: monmap: 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 librados: starting msgr at 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 librados: starting objecter 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 Processor -- start 2024-09-11T02:56:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- start start 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 librados: setting wanted keys 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 librados: calling monclient init 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient: init 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient: _add_conns ranks=[0,4,3,6,2,7,8,5,1] 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8139b10 0x7f25b813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): picked mon.noname-a con 0x7f25b8139b10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25b8139ef0 0x7f25b813ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): picked mon.noname-c con 0x7f25b8139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25b813a2d0 0x7f25b813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): picked mon.noname-g con 0x7f25b813a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8143460 con 0x7f25b8139b10 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f25b80a88a0 con 0x7f25b813a2d0 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f25b81435e0 con 0x7f25b8139ef0 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25bd4cc640 10 monclient(hunting): authenticate will time out at 1344.067671s 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25b8139ef0 0x7f25b813ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8139b10 0x7f25b813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25b8139ef0 0x7f25b813ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38140/0 (socket says 172.21.15.98:38140) 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 1 -- 172.21.15.98:0/1599534397 learned_addr learned my addr 172.21.15.98:0/1599534397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 10 monclient(hunting): get_auth_request con 0x7f25b8139ef0 auth_method 0 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 10 monclient(hunting): get_auth_request con 0x7f25b8139b10 auth_method 0 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.184+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b77fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25b813a2d0 0x7f25b813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 10 monclient(hunting): handle_auth_done global_id 23719 payload 824 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25b813a2d0 msgr2=0x7f25b813b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f25b813a2d0 0x7f25b813b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8139b10 msgr2=0x7f25b813a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b8139b10 0x7f25b813a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 1 monclient: found mon.noname-c 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 1 -- 172.21.15.98:0/1599534397 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b813cff0 con 0x7f25b8139ef0 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b67fc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25b8139ef0 0x7f25b813ad10 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f25a800e9d0 tx=0x7f25a800ee90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 client.?.objecter ms_handle_connect 0x7f25b8139ef0 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25a8017070 con 0x7f25b8139ef0 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: dump: 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a8013470 con 0x7f25b8139ef0 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25a8013a90 con 0x7f25b8139ef0 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 5 monclient: authenticate success, global_id 23719 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 10 monclient: _renew_subs 2024-09-11T02:56:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b81406f0 con 0x7f25b8139ef0 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b57fa640 4 set_mon_vals no callback set 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: dump: 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 10 client.23719.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 10 monclient: _renew_subs 2024-09-11T02:56:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b80aa510 con 0x7f25b8139ef0 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 1 librados: init done 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8133760 0x7f25b8133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f25b806ca10 con 0x7f25b8133760 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f25a8013c30 con 0x7f25b8139ef0 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8133760 0x7f25b8133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient: get_auth_request con 0x7f25b8133760 auth_method 0 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25a403df40 0x7f25a4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(293..293 src has 1..293) ==== 6194+0+0 (secure 0 0 0) 0x7f25a800f040 con 0x7f25b8139ef0 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 client.23719.objecter ms_dispatch 0x7f25b8000ba0 osd_map(293..293 src has 1..293) 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 3 client.23719.objecter handle_osd_map got epochs [293,293] > 0 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 3 client.23719.objecter handle_osd_map decoding full epoch 293 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_done global_id 24200 payload 824 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 1 -- 172.21.15.98:0/1599534397 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25b8135940 con 0x7f25b8133760 2024-09-11T02:56:30.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 20 client.23719.objecter dump_active .. 0 homeless 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 client.23719.objecter ms_handle_connect 0x7f25b8133760 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 client.23719.objecter resend_mon_ops 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b6ffd640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8133760 0x7f25b8133b40 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0097c0 tx=0x7f25ac019140 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b77fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25a403df40 0x7f25a4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25ac0b0040 con 0x7f25b8133760 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: dump: 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.215 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.216 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b77fe640 10 monclient: get_auth_request con 0x7f25a403df40 auth_method 0 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f25ac0a4e60 con 0x7f25b8133760 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8133760 msgr2=0x7f25b8133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8133760 0x7f25b8133b40 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0097c0 tx=0x7f25ac019140 comp rx=0 tx=0).stop 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8133760 msgr2=0x7f25b8133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8133760 0x7f25b8133b40 secure :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f25ac0097c0 tx=0x7f25ac019140 comp rx=0 tx=0).stop 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f25b77fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25a403df40 0x7f25a4040400 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f25b80a9a70 tx=0x7f25b000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.188+0000 7f259f7fe640 10 client.23719.objecter ms_handle_connect 0x7f25a403df40 2024-09-11T02:56:30.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:30.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8134080 0x7f25b8137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f25b806e870 con 0x7f25b8134080 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8134080 0x7f25b8137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient: get_auth_request con 0x7f25b8134080 auth_method 0 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 10 monclient(hunting): handle_auth_done global_id 24209 payload 824 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 1 -- 172.21.15.98:0/1599534397 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f25b806e680 con 0x7f25b8134080 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 10 client.23719.objecter ms_handle_connect 0x7f25b8134080 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 10 client.23719.objecter resend_mon_ops 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b6ffd640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8134080 0x7f25b8137b60 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f25ac000c10 tx=0x7f25ac0a8000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25ac0b0040 con 0x7f25b8134080 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 10 monclient: dump: 2024-09-11T02:56:30.221 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.222 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f25ac0bf890 con 0x7f25b8134080 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8134080 msgr2=0x7f25b8137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8134080 0x7f25b8137b60 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f25ac000c10 tx=0x7f25ac0a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8134080 msgr2=0x7f25b8137b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f259f7fe640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f25b8134080 0x7f25b8137b60 secure :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f25ac000c10 tx=0x7f25ac0a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 405988, 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:56:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.240 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.241 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.243 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:30.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.245 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:30.250 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:30.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.252 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.252 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:30.252 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 10 librados: watch_flush enter 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 10 librados: watch_flush exit 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 20 client.23719.objecter shutdown clearing up homeless session... 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 10 client.23719.objecter successfully canceled tick 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25a403df40 msgr2=0x7f25a4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25a403df40 0x7f25a4040400 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f25b80a9a70 tx=0x7f25b000b040 comp rx=0 tx=0).stop 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 10 monclient: shutdown 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25b8139ef0 msgr2=0x7f25b813ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25b77fe640 1 -- 172.21.15.98:0/1599534397 reap_dead start 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25b8139ef0 0x7f25b813ad10 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f25a800e9d0 tx=0x7f25a800ee90 comp rx=0 tx=0).stop 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 shutdown_connections 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 --2- 172.21.15.98:0/1599534397 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f25b8139ef0 0x7f25b813ad10 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 >> 172.21.15.98:0/1599534397 conn(0x7f25b80ac790 msgr2=0x7f25b80aa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 shutdown_connections 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 -- 172.21.15.98:0/1599534397 wait complete. 2024-09-11T02:56:30.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.196+0000 7f25bd4cc640 1 librados: shutdown 2024-09-11T02:56:30.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:30.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:56:30.256 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:56:30.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient: get_monmap_and_config 2024-09-11T02:56:30.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient: monmap: 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:30.381395+0000 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:30.381395+0000 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.380 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.381 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.381 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.381 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.381 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:30.381 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:30.381 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 Processor -- start 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 -- start start 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient: init 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient: _add_conns ranks=[9,6,5,3,0,10,4,8,2,7,1] 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faeb006f6c0 0x7faeb00726d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): picked mon.noname-a con 0x7faeb006f6c0 addr v2:172.21.15.98:3300/0 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb00a8380 0x7faeb006faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): picked mon.noname-d con 0x7faeb00a8380 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7faeb006ffe0 0x7faeb01397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): picked mon.noname-h con 0x7faeb006ffe0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7faeb00718a0 con 0x7faeb006f6c0 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7faeb0071af0 con 0x7faeb006ffe0 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7faeb00a8140 con 0x7faeb00a8380 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb7838640 10 monclient(hunting): authenticate will time out at 1344.261631s 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faeb006f6c0 0x7faeb00726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7faeb006f6c0 0x7faeb00726d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57528/0 (socket says 172.21.15.98:57528) 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 -- 172.21.15.98:0/1767982210 learned_addr learned my addr 172.21.15.98:0/1767982210 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): get_auth_request con 0x7faeb006f6c0 auth_method 0 2024-09-11T02:56:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb00a8380 0x7faeb006faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 10 monclient(hunting): get_auth_request con 0x7faeb00a8380 auth_method 0 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb5dd5640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7faeb006ffe0 0x7faeb01397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb5dd5640 10 monclient(hunting): get_auth_request con 0x7faeb006ffe0 auth_method 0 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb5dd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb5dd5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb5dd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient(hunting): handle_auth_done global_id 44154 payload 274 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 -- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb00a8380 msgr2=0x7faeb006faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb00a8380 0x7faeb006faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 -- 172.21.15.98:0/1767982210 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7faeb006ffe0 msgr2=0x7faeb01397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7faeb006ffe0 0x7faeb01397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 monclient: found mon.noname-a 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb55d4640 1 -- 172.21.15.98:0/1767982210 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb00a7d30 con 0x7faeb006f6c0 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.380+0000 7faeb4dd3640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb00a8380 0x7faeb006faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb00726d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faea0009870 tx=0x7faea001c160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afc85fc858f20af5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 1 -- 172.21.15.98:0/1767982210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faea00b0070 con 0x7faeb006f6c0 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: dump: 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 5 monclient: authenticate success, global_id 44154 2024-09-11T02:56:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 1 -- 172.21.15.98:0/1767982210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea00aa810 con 0x7faeb006f6c0 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 4 set_mon_vals no callback set 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 1 -- 172.21.15.98:0/1767982210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faea00a9300 con 0x7faeb006f6c0 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea7fff640 10 monclient: dump: 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: shutdown 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 msgr2=0x7faeb00726d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb00726d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faea0009870 tx=0x7faea001c160 comp rx=0 tx=0).stop 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- 172.21.15.98:0/1767982210 shutdown_connections 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb00a8380 0x7faeb006faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7faeb006ffe0 0x7faeb01397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 --2- 172.21.15.98:0/1767982210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb00726d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- 172.21.15.98:0/1767982210 >> 172.21.15.98:0/1767982210 conn(0x7faeb00ac8e0 msgr2=0x7faeb00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- 172.21.15.98:0/1767982210 shutdown_connections 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- 172.21.15.98:0/1767982210 wait complete. 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: monmap: 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.400 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 librados: starting msgr at 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 librados: starting objecter 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 Processor -- start 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- start start 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 librados: setting wanted keys 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 librados: calling monclient init 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: init 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: _add_conns ranks=[0,6,7,1,3,4,5,8,2] 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb01d18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): picked mon.noname-a con 0x7faeb006f6c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:30.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb006ffe0 0x7faeb0138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): picked mon.noname-d con 0x7faeb006ffe0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7faeb00a8380 0x7faeb013d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): picked mon.noname-i con 0x7faeb00a8380 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7faeb01437d0 con 0x7faeb006f6c0 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7faeb0143650 con 0x7faeb006ffe0 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7faeb0143950 con 0x7faeb00a8380 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient(hunting): authenticate will time out at 1344.266623s 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb01d18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb006ffe0 0x7faeb0138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb01d18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57530/0 (socket says 172.21.15.98:57530) 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 -- 172.21.15.98:0/189101109 learned_addr learned my addr 172.21.15.98:0/189101109 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): get_auth_request con 0x7faeb006f6c0 auth_method 0 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 10 monclient(hunting): get_auth_request con 0x7faeb006ffe0 auth_method 0 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient(hunting): handle_auth_done global_id 44163 payload 824 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7faeb00a8380 msgr2=0x7faeb013d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7faeb00a8380 0x7faeb013d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb006ffe0 msgr2=0x7faeb0138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb006ffe0 0x7faeb0138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 monclient: found mon.noname-a 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 -- 172.21.15.98:0/189101109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb0124220 con 0x7faeb006f6c0 2024-09-11T02:56:30.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb4dd3640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faeb006ffe0 0x7faeb0138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb55d4640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb01d18e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faea001bf90 tx=0x7faea0018f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 client.?.objecter ms_handle_connect 0x7faeb006f6c0 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faea00b0040 con 0x7faeb006f6c0 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: dump: 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.405 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea00044b0 con 0x7faeb006f6c0 2024-09-11T02:56:30.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 5 monclient: authenticate success, global_id 44163 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: _renew_subs 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb0122ba0 con 0x7faeb006f6c0 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faea0003a90 con 0x7faeb006f6c0 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea77fe640 4 set_mon_vals no callback set 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 client.44163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faea5ffb640 10 monclient: dump: 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.407 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: _renew_subs 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb006fb30 con 0x7faeb006f6c0 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.384+0000 7faeb7838640 1 librados: init done 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7faea00b3070 con 0x7faeb006f6c0 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb7838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb7838640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb0138dc0 0x7faeb013df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faeb006ca10 con 0x7faeb0138dc0 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae9403de70 0x7fae94040330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(293..293 src has 1..293) ==== 6194+0+0 (secure 0 0 0) 0x7faea00e9160 con 0x7faeb006f6c0 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 client.44163.objecter ms_dispatch 0x7faeb0000ba0 osd_map(293..293 src has 1..293) 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 3 client.44163.objecter handle_osd_map got epochs [293,293] > 0 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 3 client.44163.objecter handle_osd_map decoding full epoch 293 2024-09-11T02:56:30.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 20 client.44163.objecter dump_active .. 0 homeless 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb0138dc0 0x7faeb013df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb5dd5640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae9403de70 0x7fae94040330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient: get_auth_request con 0x7faeb0138dc0 auth_method 0 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb5dd5640 10 monclient: get_auth_request con 0x7fae9403de70 auth_method 0 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_done global_id 25093 payload 824 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 1 -- 172.21.15.98:0/189101109 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faeb0072740 con 0x7faeb0138dc0 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 client.44163.objecter ms_handle_connect 0x7fae9403de70 2024-09-11T02:56:30.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb5dd5640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae9403de70 0x7fae94040330 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7faea80070b0 tx=0x7faea8007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 client.44163.objecter ms_handle_connect 0x7faeb0138dc0 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 client.44163.objecter resend_mon_ops 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faeb4dd3640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb0138dc0 0x7faeb013df20 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7faeac011be0 tx=0x7faeac00f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeac019070 con 0x7faeb0138dc0 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 monclient: dump: 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faeac004030 con 0x7faeb0138dc0 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:30.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb0138dc0 msgr2=0x7faeb013df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb0138dc0 0x7faeb013df20 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7faeac011be0 tx=0x7faeac00f610 comp rx=0 tx=0).stop 2024-09-11T02:56:30.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb0138dc0 msgr2=0x7faeb013df20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.388+0000 7faea5ffb640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb0138dc0 0x7faeb013df20 secure :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7faeac011be0 tx=0x7faeac00f610 comp rx=0 tx=0).stop 2024-09-11T02:56:30.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb7838640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:30.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb7838640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb013e460 0x7faeb00a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb7838640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faeb006e870 con 0x7faeb013e460 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb013e460 0x7faeb00a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient: get_auth_request con 0x7faeb013e460 auth_method 0 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 10 monclient(hunting): handle_auth_done global_id 25102 payload 824 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 1 -- 172.21.15.98:0/189101109 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faeb013e870 con 0x7faeb013e460 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faeb4dd3640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb013e460 0x7faeb00a3e80 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7faeac01b000 tx=0x7faeac0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faea5ffb640 10 client.44163.objecter ms_handle_connect 0x7faeb013e460 2024-09-11T02:56:30.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.396+0000 7faea5ffb640 10 client.44163.objecter resend_mon_ops 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeac019040 con 0x7faeb013e460 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 10 monclient: dump: 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.416 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7faeac01e450 con 0x7faeb013e460 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb013e460 msgr2=0x7faeb00a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb013e460 0x7faeb00a3e80 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7faeac01b000 tx=0x7faeac0046d0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb013e460 msgr2=0x7faeb00a3e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faea5ffb640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faeb013e460 0x7faeb00a3e80 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 406173, 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.425 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:30.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:30.434 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:30.436 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:30.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:30.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.440 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.442 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:30.442 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 10 librados: watch_flush enter 2024-09-11T02:56:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 10 librados: watch_flush exit 2024-09-11T02:56:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 20 client.44163.objecter shutdown clearing up homeless session... 2024-09-11T02:56:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 10 client.44163.objecter successfully canceled tick 2024-09-11T02:56:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae9403de70 msgr2=0x7fae94040330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae9403de70 0x7fae94040330 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7faea80070b0 tx=0x7faea8007040 comp rx=0 tx=0).stop 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 10 monclient: shutdown 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 msgr2=0x7faeb01d18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 --2- 172.21.15.98:0/189101109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb006f6c0 0x7faeb01d18e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faea001bf90 tx=0x7faea0018f00 comp rx=0 tx=0).stop 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb5dd5640 1 -- 172.21.15.98:0/189101109 reap_dead start 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb5dd5640 1 -- 172.21.15.98:0/189101109 reap_dead start 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 shutdown_connections 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 >> 172.21.15.98:0/189101109 conn(0x7faeb00ac8e0 msgr2=0x7faeb00a9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 shutdown_connections 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 -- 172.21.15.98:0/189101109 wait complete. 2024-09-11T02:56:30.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.404+0000 7faeb7838640 1 librados: shutdown 2024-09-11T02:56:30.446 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:30.446 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:56:30.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:56:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient: get_monmap_and_config 2024-09-11T02:56:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient: monmap: 2024-09-11T02:56:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:30.561998+0000 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:30.561998+0000 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:30.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 1 Processor -- start 2024-09-11T02:56:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 1 -- start start 2024-09-11T02:56:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient: init 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient: _add_conns ranks=[4,7,0,1,2,8,10,6,5,3,9] 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7a10 0x7f4a740724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient(hunting): picked mon.noname-c con 0x7f4a740a7a10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a740a7df0 0x7f4a740a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient(hunting): picked mon.noname-i con 0x7f4a740a7df0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a741397c0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a74071210 con 0x7f4a741397c0 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4a74073d30 con 0x7f4a740a7a10 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4a74071ab0 con 0x7f4a740a7df0 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a78d42640 10 monclient(hunting): authenticate will time out at 1344.442385s 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7a10 0x7f4a740724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7a10 0x7f4a740724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38142/0 (socket says 172.21.15.98:38142) 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 -- 172.21.15.98:0/309460880 learned_addr learned my addr 172.21.15.98:0/309460880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): get_auth_request con 0x7f4a740a7a10 auth_method 0 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a71d9b640 1 --2- 172.21.15.98:0/309460880 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a740a7df0 0x7f4a740a81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a71d9b640 10 monclient(hunting): get_auth_request con 0x7f4a740a7df0 auth_method 0 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7159a640 1 -- 172.21.15.98:0/309460880 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810866775 0 0) 0x7f4a74071210 con 0x7f4a741397c0 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7159a640 10 monclient(hunting): _negotiate 2024-09-11T02:56:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7159a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7159a640 10 monclient(hunting): my global_id is 44172 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7159a640 1 -- 172.21.15.98:0/309460880 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a5c003530 con 0x7f4a741397c0 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient(hunting): handle_auth_done global_id 23728 payload 274 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 -- 172.21.15.98:0/309460880 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a740a7df0 msgr2=0x7f4a740a81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 --2- 172.21.15.98:0/309460880 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a740a7df0 0x7f4a740a81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 -- 172.21.15.98:0/309460880 >> v1:172.21.15.98:6789/0 conn(0x7f4a741397c0 legacy=0x7f4a74139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 monclient: found mon.noname-c 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.560+0000 7f4a7259c640 1 -- 172.21.15.98:0/309460880 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a740a98e0 con 0x7f4a740a7a10 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 1 --2- 172.21.15.98:0/309460880 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7a10 0x7f4a740724c0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f4a680098a0 tx=0x7f4a6801c4d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=579d6b01e144550c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 1 -- 172.21.15.98:0/309460880 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a680b0070 con 0x7f4a740a7a10 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: dump: 2024-09-11T02:56:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 1 -- 172.21.15.98:0/309460880 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a680aa810 con 0x7f4a740a7a10 2024-09-11T02:56:30.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 1 -- 172.21.15.98:0/309460880 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a680a9300 con 0x7f4a740a7a10 2024-09-11T02:56:30.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:30.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7159a640 10 monclient: dump: 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.585 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 5 monclient: authenticate success, global_id 23728 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 4 set_mon_vals no callback set 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient: shutdown 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- 172.21.15.98:0/309460880 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7a10 msgr2=0x7f4a740724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 --2- 172.21.15.98:0/309460880 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7a10 0x7f4a740724c0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f4a680098a0 tx=0x7f4a6801c4d0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- 172.21.15.98:0/309460880 shutdown_connections 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 --2- 172.21.15.98:0/309460880 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a740a7df0 0x7f4a740a81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 --2- 172.21.15.98:0/309460880 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7a10 0x7f4a740724c0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- 172.21.15.98:0/309460880 >> 172.21.15.98:0/309460880 conn(0x7f4a740a8b00 msgr2=0x7f4a740a8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- 172.21.15.98:0/309460880 shutdown_connections 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- 172.21.15.98:0/309460880 wait complete. 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient: monmap: 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.587 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 librados: starting msgr at 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 librados: starting objecter 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 Processor -- start 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- start start 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 librados: setting wanted keys 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 librados: calling monclient init 2024-09-11T02:56:30.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient: init 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient: _add_conns ranks=[6,4,7,1,3,8,5,2,0] 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4a740a7a10 0x7f4a74137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): picked mon.noname-d con 0x7f4a740a7a10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7df0 0x7f4a741384f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): picked mon.noname-c con 0x7f4a740a7df0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a741397c0 0x7f4a7413d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): picked mon.noname-i con 0x7f4a741397c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4a74144640 con 0x7f4a740a7df0 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4a741444c0 con 0x7f4a740a7a10 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4a741447c0 con 0x7f4a741397c0 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a78d42640 10 monclient(hunting): authenticate will time out at 1344.447645s 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7df0 0x7f4a741384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4a740a7a10 0x7f4a74137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7df0 0x7f4a741384f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38150/0 (socket says 172.21.15.98:38150) 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4a740a7a10 0x7f4a74137fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:52024/0 (socket says 172.21.15.98:52024) 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 1 -- 172.21.15.98:0/774563012 learned_addr learned my addr 172.21.15.98:0/774563012 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 10 monclient(hunting): get_auth_request con 0x7f4a740a7df0 auth_method 0 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 10 monclient(hunting): get_auth_request con 0x7f4a740a7a10 auth_method 0 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a71d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.564+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a72d9d640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a741397c0 0x7f4a7413d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 10 monclient(hunting): handle_auth_done global_id 23737 payload 824 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a741397c0 msgr2=0x7f4a7413d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a741397c0 0x7f4a7413d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4a740a7a10 msgr2=0x7f4a74137fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4a740a7a10 0x7f4a74137fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 1 monclient: found mon.noname-c 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 1 -- 172.21.15.98:0/774563012 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a74122cb0 con 0x7f4a740a7df0 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a72d9d640 10 monclient: get_auth_request con 0x7f4a741397c0 auth_method 0 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a72d9d640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4a741397c0 0x7f4a7413d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4a740a7a10 0x7f4a74137fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:30.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a71d9b640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7df0 0x7f4a741384f0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f4a64007f20 tx=0x7f4a6400e570 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 client.?.objecter ms_handle_connect 0x7f4a740a7df0 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a64014890 con 0x7f4a740a7df0 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: dump: 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.594 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.595 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a64019070 con 0x7f4a740a7df0 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a64013380 con 0x7f4a740a7df0 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 5 monclient: authenticate success, global_id 23737 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a70d99640 4 set_mon_vals no callback set 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: dump: 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 10 monclient: _renew_subs 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a74139e90 con 0x7f4a740a7df0 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 10 client.23737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 10 monclient: _renew_subs 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a740abd40 con 0x7f4a740a7df0 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 1 librados: init done 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413dcd0 0x7f4a7413e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7406ca10 con 0x7f4a7413dcd0 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4a64014420 con 0x7f4a740a7df0 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413dcd0 0x7f4a7413e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient: get_auth_request con 0x7f4a7413dcd0 auth_method 0 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4a5403df40 0x7f4a54040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(294..294 src has 1..294) ==== 6523+0+0 (secure 0 0 0) 0x7f4a6404fea0 con 0x7f4a740a7df0 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 client.23737.objecter ms_dispatch 0x7f4a74000ba0 osd_map(294..294 src has 1..294) 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 3 client.23737.objecter handle_osd_map got epochs [294,294] > 0 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 3 client.23737.objecter handle_osd_map decoding full epoch 294 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 20 client.23737.objecter dump_active .. 0 homeless 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a72d9d640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4a5403df40 0x7f4a54040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a72d9d640 10 monclient: get_auth_request con 0x7f4a5403df40 auth_method 0 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 10 monclient(hunting): handle_auth_done global_id 44121 payload 824 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 1 -- 172.21.15.98:0/774563012 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a740724c0 con 0x7f4a7413dcd0 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a7259c640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413dcd0 0x7f4a7413e0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a740a66c0 tx=0x7f4a68019270 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 client.23737.objecter ms_handle_connect 0x7f4a7413dcd0 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 client.23737.objecter resend_mon_ops 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a680ba030 con 0x7f4a7413dcd0 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: dump: 2024-09-11T02:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.601 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 client.23737.objecter ms_handle_connect 0x7f4a5403df40 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a72d9d640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4a5403df40 0x7f4a54040400 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c007fd0 tx=0x7f4a6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4a680aa750 con 0x7f4a7413dcd0 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413dcd0 msgr2=0x7f4a7413e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413dcd0 0x7f4a7413e0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a740a66c0 tx=0x7f4a68019270 comp rx=0 tx=0).stop 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413dcd0 msgr2=0x7f4a7413e0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.568+0000 7f4a62ffd640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413dcd0 0x7f4a7413e0b0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a740a66c0 tx=0x7f4a68019270 comp rx=0 tx=0).stop 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a78d42640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a78d42640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413e5f0 0x7f4a740a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a78d42640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7406e870 con 0x7f4a7413e5f0 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413e5f0 0x7f4a740a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient: get_auth_request con 0x7f4a7413e5f0 auth_method 0 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 10 monclient(hunting): handle_auth_done global_id 44130 payload 824 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 1 -- 172.21.15.98:0/774563012 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a7406f9e0 con 0x7f4a7413e5f0 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a7259c640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413e5f0 0x7f4a740a3e80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a680098a0 tx=0x7f4a680a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 10 client.23737.objecter ms_handle_connect 0x7f4a7413e5f0 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 10 client.23737.objecter resend_mon_ops 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a680ba040 con 0x7f4a7413e5f0 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 10 monclient: dump: 2024-09-11T02:56:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f4a68003620 con 0x7f4a7413e5f0 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413e5f0 msgr2=0x7f4a740a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413e5f0 0x7f4a740a3e80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4a680098a0 tx=0x7f4a680a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413e5f0 msgr2=0x7f4a740a3e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.576+0000 7f4a62ffd640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4a7413e5f0 0x7f4a740a3e80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.610 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 17003, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:30.611 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.613 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.614 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:30.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.616 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.618 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:30.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.624 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:30.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.626 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:30.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:30.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.633 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.634 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.635 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.635 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.635 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:30.635 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:30.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 10 librados: watch_flush enter 2024-09-11T02:56:30.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 10 librados: watch_flush exit 2024-09-11T02:56:30.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 20 client.23737.objecter shutdown clearing up homeless session... 2024-09-11T02:56:30.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 10 client.23737.objecter successfully canceled tick 2024-09-11T02:56:30.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4a5403df40 msgr2=0x7f4a54040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4a5403df40 0x7f4a54040400 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c007fd0 tx=0x7f4a6c009040 comp rx=0 tx=0).stop 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 10 monclient: shutdown 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7df0 msgr2=0x7f4a741384f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 --2- 172.21.15.98:0/774563012 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4a740a7df0 0x7f4a741384f0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f4a64007f20 tx=0x7f4a6400e570 comp rx=0 tx=0).stop 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a72d9d640 1 -- 172.21.15.98:0/774563012 reap_dead start 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a72d9d640 1 -- 172.21.15.98:0/774563012 reap_dead start 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 shutdown_connections 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 >> 172.21.15.98:0/774563012 conn(0x7f4a740a8b00 msgr2=0x7f4a740a9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 shutdown_connections 2024-09-11T02:56:30.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 -- 172.21.15.98:0/774563012 wait complete. 2024-09-11T02:56:30.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.580+0000 7f4a78d42640 1 librados: shutdown 2024-09-11T02:56:30.639 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:30.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:56:30.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:56:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient: get_monmap_and_config 2024-09-11T02:56:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient: monmap: 2024-09-11T02:56:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:30.758189+0000 2024-09-11T02:56:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:30.758189+0000 2024-09-11T02:56:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:30.760 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 1 Processor -- start 2024-09-11T02:56:30.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 1 -- start start 2024-09-11T02:56:30.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient: init 2024-09-11T02:56:30.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient: _add_conns ranks=[7,1,0,10,8,6,4,9,3,5,2] 2024-09-11T02:56:30.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 0x7fbec8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient(hunting): picked mon.noname-i con 0x7fbec8139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbec806fe50 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbec8072640 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec80a8830 con 0x7fbec8072640 2024-09-11T02:56:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec8071370 con 0x7fbec806fe50 2024-09-11T02:56:30.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbec80719c0 con 0x7fbec8139b80 2024-09-11T02:56:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbece933640 10 monclient(hunting): authenticate will time out at 1344.638542s 2024-09-11T02:56:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbecced0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbec8072640 0x7fbec80a8120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54382/0 (socket says 172.21.15.98:54382) 2024-09-11T02:56:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbecced0640 1 -- 172.21.15.98:0/3595372005 learned_addr learned my addr 172.21.15.98:0/3595372005 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:30.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 0x7fbec8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): get_auth_request con 0x7fbec8139b80 auth_method 0 2024-09-11T02:56:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560701888 0 0) 0x7fbec80a8830 con 0x7fbec8072640 2024-09-11T02:56:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): _negotiate 2024-09-11T02:56:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): my global_id is 44181 2024-09-11T02:56:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb0003530 con 0x7fbec8072640 2024-09-11T02:56:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2008301415 0 0) 0x7fbeb0003530 con 0x7fbec8072640 2024-09-11T02:56:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 msgr2=0x7fbec8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 0x7fbec8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 >> v1:172.21.15.190:6789/0 conn(0x7fbec806fe50 legacy=0x7fbec80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:56:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:56:30.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec80a9600 con 0x7fbec8072640 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fbec0001cf0 con 0x7fbec8072640 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: dump: 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.796 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:56:30.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 >> v1:172.21.15.98:6789/0 conn(0x7fbec8072640 legacy=0x7fbec80a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient: _add_conns ranks=[8,5,7,0,2,1,3,6,4] 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 0x7fbeb0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): picked mon.e con 0x7fbeb0004270 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbeb0004e10 0x7fbeb0005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): picked mon.h con 0x7fbeb0004e10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 0x7fbeb000bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): picked mon.i con 0x7fbec8139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbec80719c0 con 0x7fbeb0004e10 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbeb000c410 con 0x7fbec8139b80 2024-09-11T02:56:30.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbeb000c5c0 con 0x7fbeb0004270 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 0x7fbeb0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): get_auth_request con 0x7fbeb0004270 auth_method 0 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 1 -- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 msgr2=0x7fbeb0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 1 -- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 msgr2=0x7fbeb0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 0x7fbeb0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:30.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec7fff640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 0x7fbeb0004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec77fe640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbeb0004e10 0x7fbeb0005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec77fe640 10 monclient(hunting): get_auth_request con 0x7fbeb0004e10 auth_method 0 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbec77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbecced0640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 0x7fbeb000bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbecced0640 10 monclient(hunting): get_auth_request con 0x7fbec8139b80 auth_method 0 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbecced0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbecced0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.756+0000 7fbecced0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbecced0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbecced0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbecced0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient(hunting): handle_auth_done global_id 44181 payload 293 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 1 -- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 msgr2=0x7fbeb0004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbeb0004270 0x7fbeb0004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 1 -- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 msgr2=0x7fbeb000bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbec8139b80 0x7fbeb000bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 1 monclient: found mon.h 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:56:30.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 1 -- 172.21.15.98:0/3595372005 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb0003530 con 0x7fbeb0004e10 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 5 monclient: authenticate success, global_id 44181 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec77fe640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbeb0004e10 0x7fbeb0005290 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbebc001790 tx=0x7fbebc01c370 comp rx=0 tx=0).ready entity=mon.5 client_cookie=3c22ecf4415a5d29 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbebc0013e0 con 0x7fbeb0004e10 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 10 monclient: dump: 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.802 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.803 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbebc0050d0 con 0x7fbeb0004e10 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 4 set_mon_vals no callback set 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 1 -- 172.21.15.98:0/3595372005 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbebc005280 con 0x7fbeb0004e10 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbec6ffd640 10 monclient: dump: 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.804 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 10 monclient: shutdown 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 1 -- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbeb0004e10 msgr2=0x7fbeb0005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 1 --2- 172.21.15.98:0/3595372005 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbeb0004e10 0x7fbeb0005290 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbebc001790 tx=0x7fbebc01c370 comp rx=0 tx=0).stop 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbecced0640 1 -- 172.21.15.98:0/3595372005 reap_dead start 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 1 -- 172.21.15.98:0/3595372005 shutdown_connections 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 1 -- 172.21.15.98:0/3595372005 >> 172.21.15.98:0/3595372005 conn(0x7fbec80ac730 msgr2=0x7fbec80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 1 -- 172.21.15.98:0/3595372005 shutdown_connections 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.760+0000 7fbece933640 1 -- 172.21.15.98:0/3595372005 wait complete. 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: monmap: 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.806 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.807 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 librados: starting msgr at 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 librados: starting objecter 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 Processor -- start 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 -- start start 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 librados: setting wanted keys 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 librados: calling monclient init 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: init 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: _add_conns ranks=[3,4,8,2,5,7,0,6,1] 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbeb0004e10 0x7fbec81d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): picked mon.noname-g con 0x7fbeb0004e10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbec8122ab0 0x7fbec8138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): picked mon.noname-c con 0x7fbec8122ab0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbec8138e50 0x7fbec80a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): picked mon.noname-e con 0x7fbec8138e50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbec8143450 con 0x7fbeb0004e10 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbec80517e0 con 0x7fbec8122ab0 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbec81435d0 con 0x7fbec8138e50 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): authenticate will time out at 1344.644893s 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbec8122ab0 0x7fbec8138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbec8138e50 0x7fbec80a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbec8122ab0 0x7fbec8138910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38154/0 (socket says 172.21.15.98:38154) 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 1 -- 172.21.15.98:0/486633394 learned_addr learned my addr 172.21.15.98:0/486633394 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient(hunting): get_auth_request con 0x7fbec8122ab0 auth_method 0 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): get_auth_request con 0x7fbec8138e50 auth_method 0 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbeb0004e10 0x7fbec81d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): get_auth_request con 0x7fbeb0004e10 auth_method 0 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient(hunting): handle_auth_done global_id 24218 payload 824 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbec8122ab0 msgr2=0x7fbec8138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbec8122ab0 0x7fbec8138910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbeb0004e10 msgr2=0x7fbec81d15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbeb0004e10 0x7fbec81d15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 monclient: found mon.noname-e 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 -- 172.21.15.98:0/486633394 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec81242b0 con 0x7fbec8138e50 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbec8122ab0 0x7fbec8138910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbecced0640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbec8138e50 0x7fbec80a3d00 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c590 tx=0x7fbec000f5a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 client.?.objecter ms_handle_connect 0x7fbec8138e50 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbec000fe10 con 0x7fbec8138e50 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: dump: 2024-09-11T02:56:30.812 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.813 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec0018070 con 0x7fbec8138e50 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 5 monclient: authenticate success, global_id 24218 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: _renew_subs 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec80aa120 con 0x7fbec8138e50 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbec0013cd0 con 0x7fbec8138e50 2024-09-11T02:56:30.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 monclient: dump: 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.815 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec67fc640 4 set_mon_vals no callback set 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 client.24218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: _renew_subs 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec80abd20 con 0x7fbec8138e50 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 librados: init done 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec8123e70 0x7fbec80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbec806ca10 con 0x7fbec8123e70 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbec00133f0 con 0x7fbec8138e50 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec8123e70 0x7fbec80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient: get_auth_request con 0x7fbec8123e70 auth_method 0 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbeb403df40 0x7fbeb4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(294..294 src has 1..294) ==== 6523+0+0 (secure 0 0 0) 0x7fbec0010040 con 0x7fbec8138e50 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 client.24218.objecter ms_dispatch 0x7fbec8000ba0 osd_map(294..294 src has 1..294) 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 3 client.24218.objecter handle_osd_map got epochs [294,294] > 0 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 3 client.24218.objecter handle_osd_map decoding full epoch 294 2024-09-11T02:56:30.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 20 client.24218.objecter dump_active .. 0 homeless 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbeb403df40 0x7fbeb4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec77fe640 10 monclient: get_auth_request con 0x7fbeb403df40 auth_method 0 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 10 monclient(hunting): handle_auth_done global_id 19886 payload 824 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 1 -- 172.21.15.98:0/486633394 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbec8123b30 con 0x7fbec8123e70 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 client.24218.objecter ms_handle_connect 0x7fbec8123e70 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec4ff9640 10 client.24218.objecter resend_mon_ops 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.764+0000 7fbec7fff640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec8123e70 0x7fbec80a4240 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80135b0 tx=0x7fbeb8013a70 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbeb801c070 con 0x7fbec8123e70 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 10 monclient: dump: 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.819 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 10 client.24218.objecter ms_handle_connect 0x7fbeb403df40 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec77fe640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbeb403df40 0x7fbeb4040400 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fbebc000c10 tx=0x7fbebc0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbeb8011840 con 0x7fbec8123e70 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec8123e70 msgr2=0x7fbec80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec8123e70 0x7fbec80a4240 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80135b0 tx=0x7fbeb8013a70 comp rx=0 tx=0).stop 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec8123e70 msgr2=0x7fbec80a4240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.768+0000 7fbec4ff9640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec8123e70 0x7fbec80a4240 secure :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80135b0 tx=0x7fbeb8013a70 comp rx=0 tx=0).stop 2024-09-11T02:56:30.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbece933640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbece933640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec80a4780 0x7fbec813fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbece933640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbec806e870 con 0x7fbec80a4780 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec80a4780 0x7fbec813fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient: get_auth_request con 0x7fbec80a4780 auth_method 0 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 10 monclient(hunting): handle_auth_done global_id 19895 payload 824 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.772+0000 7fbec7fff640 1 -- 172.21.15.98:0/486633394 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbec80a4b90 con 0x7fbec80a4780 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 10 client.24218.objecter ms_handle_connect 0x7fbec80a4780 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 10 client.24218.objecter resend_mon_ops 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec7fff640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec80a4780 0x7fbec813fa10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80135b0 tx=0x7fbeb8004740 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbeb8004490 con 0x7fbec80a4780 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 10 monclient: dump: 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.822 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fbeb8000e50 con 0x7fbec80a4780 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:30.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec80a4780 msgr2=0x7fbec813fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec80a4780 0x7fbec813fa10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80135b0 tx=0x7fbeb8004740 comp rx=0 tx=0).stop 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec80a4780 msgr2=0x7fbec813fa10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbec4ff9640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbec80a4780 0x7fbec813fa10 secure :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fbeb80135b0 tx=0x7fbeb8004740 comp rx=0 tx=0).stop 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 406521, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:30.848 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:30.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 10 librados: watch_flush enter 2024-09-11T02:56:30.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 10 librados: watch_flush exit 2024-09-11T02:56:30.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 20 client.24218.objecter shutdown clearing up homeless session... 2024-09-11T02:56:30.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 10 client.24218.objecter successfully canceled tick 2024-09-11T02:56:30.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbeb403df40 msgr2=0x7fbeb4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbeb403df40 0x7fbeb4040400 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fbebc000c10 tx=0x7fbebc0011a0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 10 monclient: shutdown 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbec8138e50 msgr2=0x7fbec80a3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 --2- 172.21.15.98:0/486633394 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbec8138e50 0x7fbec80a3d00 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000c590 tx=0x7fbec000f5a0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbecced0640 1 -- 172.21.15.98:0/486633394 reap_dead start 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbecced0640 1 -- 172.21.15.98:0/486633394 reap_dead start 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 shutdown_connections 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 >> 172.21.15.98:0/486633394 conn(0x7fbec80ac730 msgr2=0x7fbec80a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 shutdown_connections 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 -- 172.21.15.98:0/486633394 wait complete. 2024-09-11T02:56:30.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.776+0000 7fbece933640 1 librados: shutdown 2024-09-11T02:56:30.852 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:30.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:56:30.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:56:30.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient: get_monmap_and_config 2024-09-11T02:56:30.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient: monmap: 2024-09-11T02:56:30.969 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.969 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.969 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:30.970960+0000 2024-09-11T02:56:30.969 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:30.970960+0000 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:30.970 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 1 Processor -- start 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 1 -- start start 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient: init 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient: _add_conns ranks=[4,1,0,6,7,8,10,5,2,9,3] 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f52880a46a0 0x7f52880a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient(hunting): picked mon.noname-c con 0x7f52880a46a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f52880a4fc0 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:30.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f528813a040 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52880a17b0 con 0x7f528813a040 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52880775d0 con 0x7f52880a4fc0 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f52880a0750 con 0x7f52880a46a0 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.968+0000 7f528df4a640 10 monclient(hunting): authenticate will time out at 1344.851948s 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f52880a46a0 0x7f52880a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5287fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f528813a040 0x7f5288141c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54386/0 (socket says 172.21.15.98:54386) 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5287fff640 1 -- 172.21.15.98:0/1456291907 learned_addr learned my addr 172.21.15.98:0/1456291907 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): get_auth_request con 0x7f52880a46a0 auth_method 0 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 1 -- 172.21.15.98:0/1456291907 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2498279921 0 0) 0x7f52880a17b0 con 0x7f528813a040 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient(hunting): _negotiate 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient(hunting): my global_id is 44190 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 1 -- 172.21.15.98:0/1456291907 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5274003530 con 0x7f528813a040 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient(hunting): handle_auth_done global_id 23755 payload 274 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 1 -- 172.21.15.98:0/1456291907 >> v1:172.21.15.190:6789/0 conn(0x7f52880a4fc0 legacy=0x7f5288139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 1 -- 172.21.15.98:0/1456291907 >> v1:172.21.15.98:6789/0 conn(0x7f528813a040 legacy=0x7f5288141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 1 monclient: found mon.noname-c 2024-09-11T02:56:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:30.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 1 -- 172.21.15.98:0/1456291907 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52880a0200 con 0x7f52880a46a0 2024-09-11T02:56:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f52877fe640 1 --2- 172.21.15.98:0/1456291907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f52880a46a0 0x7f52880a4a80 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f52780097f0 tx=0x7f527801c3a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=9399daa0655bb6d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 1 -- 172.21.15.98:0/1456291907 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52780b0070 con 0x7f52880a46a0 2024-09-11T02:56:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: dump: 2024-09-11T02:56:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 1 -- 172.21.15.98:0/1456291907 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52780aa810 con 0x7f52880a46a0 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 5 monclient: authenticate success, global_id 23755 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 1 -- 172.21.15.98:0/1456291907 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52780a9300 con 0x7f52880a46a0 2024-09-11T02:56:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 4 set_mon_vals no callback set 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f5286ffd640 10 monclient: dump: 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 10 monclient: shutdown 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 1 -- 172.21.15.98:0/1456291907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f52880a46a0 msgr2=0x7f52880a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 1 --2- 172.21.15.98:0/1456291907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f52880a46a0 0x7f52880a4a80 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f52780097f0 tx=0x7f527801c3a0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 1 -- 172.21.15.98:0/1456291907 shutdown_connections 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 1 --2- 172.21.15.98:0/1456291907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f52880a46a0 0x7f52880a4a80 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 1 -- 172.21.15.98:0/1456291907 >> 172.21.15.98:0/1456291907 conn(0x7f528806ed00 msgr2=0x7f528806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 1 -- 172.21.15.98:0/1456291907 shutdown_connections 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.972+0000 7f528df4a640 1 -- 172.21.15.98:0/1456291907 wait complete. 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: build_initial_monmap 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: monmap: 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 librados: starting msgr at 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 librados: starting objecter 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 Processor -- start 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 -- start start 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 librados: setting wanted keys 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 librados: calling monclient init 2024-09-11T02:56:30.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: init 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: _add_conns ranks=[1,6,2,5,4,8,0,3,7] 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52880a46a0 0x7f5288138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): picked mon.noname-f con 0x7f52880a46a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f52880a4fc0 0x7f5288138d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): picked mon.noname-d con 0x7f52880a4fc0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f528813a040 0x7f52881392c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): picked mon.noname-b con 0x7f528813a040 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52880779d0 con 0x7f52880a46a0 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5288077850 con 0x7f528813a040 2024-09-11T02:56:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5288145b40 con 0x7f52880a4fc0 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): authenticate will time out at 1344.857361s 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f52880a4fc0 0x7f5288138d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f528813a040 0x7f52881392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f528813a040 0x7f52881392c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:35620/0 (socket says 172.21.15.98:35620) 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 1 -- 172.21.15.98:0/139849371 learned_addr learned my addr 172.21.15.98:0/139849371 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient(hunting): get_auth_request con 0x7f528813a040 auth_method 0 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): get_auth_request con 0x7f52880a4fc0 auth_method 0 2024-09-11T02:56:30.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52880a46a0 0x7f5288138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient(hunting): handle_auth_done global_id 44196 payload 824 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f528813a040 msgr2=0x7f52881392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f528813a040 0x7f52881392c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52880a46a0 msgr2=0x7f5288138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52880a46a0 0x7f5288138840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 monclient: found mon.noname-d 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 -- 172.21.15.98:0/139849371 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528813f940 con 0x7f52880a4fc0 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient: get_auth_request con 0x7f52880a46a0 auth_method 0 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52880a46a0 0x7f5288138840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f528813a040 0x7f52881392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f527ffff640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f52880a4fc0 0x7f5288138d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5288077360 tx=0x7f527000c870 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 client.?.objecter ms_handle_connect 0x7f52880a4fc0 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:30.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5270011a00 con 0x7f52880a4fc0 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: dump: 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: _finish_auth 0 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5270017070 con 0x7f52880a4fc0 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5270011c60 con 0x7f52880a4fc0 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:30.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 monclient: dump: 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:30.996 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5285ffb640 4 set_mon_vals no callback set 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 5 monclient: authenticate success, global_id 44196 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: _renew_subs 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52881412e0 con 0x7f52880a4fc0 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 client.44196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: _renew_subs 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:56:30.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f5288141000 con 0x7f52880a4fc0 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 librados: init done 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813d800 0x7f528813dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f528806ca10 con 0x7f528813d800 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5270011590 con 0x7f52880a4fc0 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813d800 0x7f528813dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:30.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f526803df70 0x7f5268040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient: get_auth_request con 0x7f528813d800 auth_method 0 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(294..294 src has 1..294) ==== 6523+0+0 (secure 0 0 0) 0x7f527004ef90 con 0x7f52880a4fc0 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 10 client.44196.objecter ms_dispatch 0x7f5288000ba0 osd_map(294..294 src has 1..294) 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 3 client.44196.objecter handle_osd_map got epochs [294,294] > 0 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 3 client.44196.objecter handle_osd_map decoding full epoch 294 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5284ff9640 20 client.44196.objecter dump_active .. 0 homeless 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:31.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f526803df70 0x7f5268040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.976+0000 7f5287fff640 10 monclient: get_auth_request con 0x7f526803df70 auth_method 0 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f52877fe640 10 monclient(hunting): handle_auth_done global_id 19510 payload 824 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f52877fe640 1 -- 172.21.15.98:0/139849371 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52880a7d00 con 0x7f528813d800 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f52877fe640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813d800 0x7f528813dbe0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f52880a21e0 tx=0x7f5278019140 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 client.44196.objecter ms_handle_connect 0x7f528813d800 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 client.44196.objecter resend_mon_ops 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52780b9030 con 0x7f528813d800 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 monclient: dump: 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.002 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5287fff640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f526803df70 0x7f5268040430 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7f5280005c10 tx=0x7f5280009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:31.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 client.44196.objecter ms_handle_connect 0x7f526803df70 2024-09-11T02:56:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5278003620 con 0x7f528813d800 2024-09-11T02:56:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813d800 msgr2=0x7f528813dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813d800 0x7f528813dbe0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f52880a21e0 tx=0x7f5278019140 comp rx=0 tx=0).stop 2024-09-11T02:56:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813d800 msgr2=0x7f528813dbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:31.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.980+0000 7f5284ff9640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813d800 0x7f528813dbe0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.984+0000 7f528df4a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.984+0000 7f528df4a640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813e120 0x7f528809d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.984+0000 7f528df4a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.984+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f528806e870 con 0x7f528813e120 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813e120 0x7f528809d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient: get_auth_request con 0x7f528813e120 auth_method 0 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:31.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 10 monclient(hunting): handle_auth_done global_id 19519 payload 824 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 1 -- 172.21.15.98:0/139849371 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5288069cc0 con 0x7f528813e120 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f52877fe640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813e120 0x7f528809d210 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f527801aa50 tx=0x7f52780a8000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 10 client.44196.objecter ms_handle_connect 0x7f528813e120 2024-09-11T02:56:31.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 10 client.44196.objecter resend_mon_ops 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52780b9040 con 0x7f528813e120 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 10 monclient: dump: 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:31.008 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f5278003aa0 con 0x7f528813e120 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:31.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813e120 msgr2=0x7f528809d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813e120 0x7f528809d210 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f527801aa50 tx=0x7f52780a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813e120 msgr2=0x7f528809d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5284ff9640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f528813e120 0x7f528809d210 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:31.010 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 406757, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:31.011 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:31.012 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:31.013 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:31.014 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:31.015 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:31.016 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.017 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:31.019 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:31.020 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:31.021 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:31.023 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.025 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:31.027 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:31.028 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.028 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.028 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.028 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:31.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:31.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:31.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:31.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:31.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:31.032 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.032 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.033 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.034 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:31.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:31.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:31.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:31.036 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:31.037 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:31.037 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.037 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.037 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.038 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:31.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:31.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:31.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:31.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:31.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:31.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:31.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:31.041 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:31.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.041 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:31.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:31.042 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:31.042 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:31.042 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:31.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.043 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:31.043 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:31.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.043 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:31.043 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:31.044 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:31.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.044 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:31.044 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:31.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:31.045 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:31.045 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:31.045 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T02:56:31.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:31.045 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:31.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:31.046 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:31.046 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:31.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 10 librados: watch_flush enter 2024-09-11T02:56:31.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 10 librados: watch_flush exit 2024-09-11T02:56:31.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 20 client.44196.objecter shutdown clearing up homeless session... 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 10 client.44196.objecter successfully canceled tick 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f526803df70 msgr2=0x7f5268040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f526803df70 0x7f5268040430 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7f5280005c10 tx=0x7f5280009040 comp rx=0 tx=0).stop 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 10 monclient: shutdown 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f52880a4fc0 msgr2=0x7f5288138d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 --2- 172.21.15.98:0/139849371 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f52880a4fc0 0x7f5288138d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5288077360 tx=0x7f527000c870 comp rx=0 tx=0).stop 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5287fff640 1 -- 172.21.15.98:0/139849371 reap_dead start 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f5287fff640 1 -- 172.21.15.98:0/139849371 reap_dead start 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 shutdown_connections 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 >> 172.21.15.98:0/139849371 conn(0x7f528806ed00 msgr2=0x7f528809f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 shutdown_connections 2024-09-11T02:56:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 -- 172.21.15.98:0/139849371 wait complete. 2024-09-11T02:56:31.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:30.988+0000 7f528df4a640 1 librados: shutdown 2024-09-11T02:56:31.051 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:31.051 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T02:56:31.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T02:56:31.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient: get_monmap_and_config 2024-09-11T02:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient: build_initial_monmap 2024-09-11T02:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient: monmap: 2024-09-11T02:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:31.170623+0000 2024-09-11T02:56:31.170 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:31.170623+0000 2024-09-11T02:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:31.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:31.172 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:31.172 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:31.172 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:31.172 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:31.172 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:31.172 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 Processor -- start 2024-09-11T02:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 -- start start 2024-09-11T02:56:31.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient: init 2024-09-11T02:56:31.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient: _add_conns ranks=[6,9,8,0,7,2,5,4,10,1,3] 2024-09-11T02:56:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f260006f6c0 0x7f2600072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): picked mon.noname-d con 0x7f260006f6c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f260006faa0 0x7f260006fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): picked mon.noname-a con 0x7f260006faa0 addr v2:172.21.15.98:3300/0 2024-09-11T02:56:31.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f26000703c0 0x7f26001397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): picked mon.noname-e con 0x7f26000703c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2600071ff0 con 0x7f260006faa0 2024-09-11T02:56:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f26000a8750 con 0x7f260006f6c0 2024-09-11T02:56:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:31.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f26000716c0 con 0x7f26000703c0 2024-09-11T02:56:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f26066da640 10 monclient(hunting): authenticate will time out at 1345.050753s 2024-09-11T02:56:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f260006faa0 0x7f260006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f260006f6c0 0x7f2600072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f26000703c0 0x7f26001397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f260006f6c0 0x7f2600072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:52046/0 (socket says 172.21.15.98:52046) 2024-09-11T02:56:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 1 -- 172.21.15.98:0/373671952 learned_addr learned my addr 172.21.15.98:0/373671952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 10 monclient(hunting): get_auth_request con 0x7f260006f6c0 auth_method 0 2024-09-11T02:56:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f260006faa0 0x7f260006fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57538/0 (socket says 172.21.15.98:57538) 2024-09-11T02:56:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:31.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f26000703c0 0x7f26001397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44640/0 (socket says 172.21.15.98:44640) 2024-09-11T02:56:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 10 monclient(hunting): get_auth_request con 0x7f260006faa0 auth_method 0 2024-09-11T02:56:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:31.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): get_auth_request con 0x7f26000703c0 auth_method 0 2024-09-11T02:56:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:31.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:31.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:31.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient(hunting): handle_auth_done global_id 24227 payload 274 2024-09-11T02:56:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 -- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f260006f6c0 msgr2=0x7f2600072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 --2- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f260006f6c0 0x7f2600072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 -- 172.21.15.98:0/373671952 >> v2:172.21.15.98:3300/0 conn(0x7f260006faa0 msgr2=0x7f260006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 --2- 172.21.15.98:0/373671952 >> v2:172.21.15.98:3300/0 conn(0x7f260006faa0 0x7f260006fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient: _finish_hunting 0 2024-09-11T02:56:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 monclient: found mon.noname-e 2024-09-11T02:56:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f2604c77640 1 -- 172.21.15.98:0/373671952 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26000a64a0 con 0x7f26000703c0 2024-09-11T02:56:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25fffff640 1 --2- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f260006f6c0 0x7f2600072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:31.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.168+0000 7f25ff7fe640 1 --2- 172.21.15.98:0/373671952 >> v2:172.21.15.98:3300/0 conn(0x7f260006faa0 0x7f260006fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f2604c77640 1 --2- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f26000703c0 0x7f26001397c0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009dc0 tx=0x7f25f801c550 comp rx=0 tx=0).ready entity=mon.8 client_cookie=eced6ddafbdc9778 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 1 -- 172.21.15.98:0/373671952 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25f80a8890 con 0x7f26000703c0 2024-09-11T02:56:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:31.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: dump: 2024-09-11T02:56:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:31.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:31.204 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:31.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:31.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 1 -- 172.21.15.98:0/373671952 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f80a8a50 con 0x7f26000703c0 2024-09-11T02:56:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 5 monclient: authenticate success, global_id 24227 2024-09-11T02:56:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:31.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 1 -- 172.21.15.98:0/373671952 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25f80a8c40 con 0x7f26000703c0 2024-09-11T02:56:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 4 set_mon_vals no callback set 2024-09-11T02:56:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f25feffd640 10 monclient: dump: 2024-09-11T02:56:31.207 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:31.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:31.210 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient: shutdown 2024-09-11T02:56:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f26000703c0 msgr2=0x7f26001397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 --2- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f26000703c0 0x7f26001397c0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009dc0 tx=0x7f25f801c550 comp rx=0 tx=0).stop 2024-09-11T02:56:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- 172.21.15.98:0/373671952 shutdown_connections 2024-09-11T02:56:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 --2- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f26000703c0 0x7f26001397c0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 --2- 172.21.15.98:0/373671952 >> v2:172.21.15.98:3300/0 conn(0x7f260006faa0 0x7f260006fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 --2- 172.21.15.98:0/373671952 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f260006f6c0 0x7f2600072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- 172.21.15.98:0/373671952 >> 172.21.15.98:0/373671952 conn(0x7f26000acac0 msgr2=0x7f26000a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- 172.21.15.98:0/373671952 shutdown_connections 2024-09-11T02:56:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- 172.21.15.98:0/373671952 wait complete. 2024-09-11T02:56:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient: build_initial_monmap 2024-09-11T02:56:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient: monmap: 2024-09-11T02:56:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:31.214 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:31.215 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 librados: starting msgr at 2024-09-11T02:56:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 librados: starting objecter 2024-09-11T02:56:31.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 Processor -- start 2024-09-11T02:56:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- start start 2024-09-11T02:56:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 librados: setting wanted keys 2024-09-11T02:56:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 librados: calling monclient init 2024-09-11T02:56:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient: init 2024-09-11T02:56:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:31.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient: _add_conns ranks=[4,3,2,5,7,1,8,0,6] 2024-09-11T02:56:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f260006f6c0 0x7f2600135c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): picked mon.noname-c con 0x7f260006f6c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f260006faa0 0x7f2600136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): picked mon.noname-g con 0x7f260006faa0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26000703c0 0x7f26001366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): picked mon.noname-b con 0x7f26000703c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2600143840 con 0x7f26000703c0 2024-09-11T02:56:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f26001436c0 con 0x7f260006faa0 2024-09-11T02:56:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:31.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f26001439c0 con 0x7f260006f6c0 2024-09-11T02:56:31.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:31.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.172+0000 7f26066da640 10 monclient(hunting): authenticate will time out at 1345.055944s 2024-09-11T02:56:31.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f260006f6c0 0x7f2600135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f260006f6c0 0x7f2600135c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38166/0 (socket says 172.21.15.98:38166) 2024-09-11T02:56:31.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 1 -- 172.21.15.98:0/231353741 learned_addr learned my addr 172.21.15.98:0/231353741 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:31.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient(hunting): get_auth_request con 0x7f260006f6c0 auth_method 0 2024-09-11T02:56:31.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:31.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:31.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:31.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26000703c0 0x7f26001366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): get_auth_request con 0x7f26000703c0 auth_method 0 2024-09-11T02:56:31.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:31.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient(hunting): handle_auth_done global_id 44165 payload 824 2024-09-11T02:56:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25ff7fe640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f260006faa0 0x7f2600136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 -- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f260006f6c0 msgr2=0x7f2600135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f260006f6c0 0x7f2600135c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 -- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f260006faa0 msgr2=0x7f2600136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f260006faa0 0x7f2600136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient: _finish_hunting 0 2024-09-11T02:56:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 monclient: found mon.noname-b 2024-09-11T02:56:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:56:31.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 -- 172.21.15.98:0/231353741 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260013fd40 con 0x7f26000703c0 2024-09-11T02:56:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f260006f6c0 0x7f2600135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f2604c77640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26000703c0 0x7f26001366c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f25f80ac000 tx=0x7f25f80192f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 client.?.objecter ms_handle_connect 0x7f26000703c0 2024-09-11T02:56:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 1 -- 172.21.15.98:0/231353741 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25f80a8860 con 0x7f26000703c0 2024-09-11T02:56:31.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:31.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:56:31.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: dump: 2024-09-11T02:56:31.225 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:31.225 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:31.225 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.225 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.225 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:31.226 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:31.226 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:31.226 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:31.226 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:31.226 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:31.227 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:31.227 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:31.227 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:31.227 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:31.227 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:31.227 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: _finish_auth 0 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: _check_auth_tickets 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 1 -- 172.21.15.98:0/231353741 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25f8003bb0 con 0x7f26000703c0 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 5 monclient: authenticate success, global_id 44165 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 monclient: _renew_subs 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26000a5800 con 0x7f26000703c0 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:31.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 1 -- 172.21.15.98:0/231353741 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25f8003da0 con 0x7f26000703c0 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 client.44165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 monclient: dump: 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:31.229 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fe7fc640 4 set_mon_vals no callback set 2024-09-11T02:56:31.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 monclient: _renew_subs 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26000a5240 con 0x7f26000703c0 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 1 librados: init done 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 1 -- 172.21.15.98:0/231353741 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f25f8004040 con 0x7f26000703c0 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f260006fe80 con 0x7f26000703c0 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25ec03df40 0x7f25ec040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 1 -- 172.21.15.98:0/231353741 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(294..294 src has 1..294) ==== 6523+0+0 (secure 0 0 0) 0x7f25f80b7070 con 0x7f26000703c0 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 client.44165.objecter ms_dispatch 0x7f2600000ba0 osd_map(294..294 src has 1..294) 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 3 client.44165.objecter handle_osd_map got epochs [294,294] > 0 2024-09-11T02:56:31.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 3 client.44165.objecter handle_osd_map decoding full epoch 294 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 20 client.44165.objecter dump_active .. 0 homeless 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25ec03df40 0x7f25ec040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 10 monclient: get_auth_request con 0x7f25ec03df40 auth_method 0 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fffff640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25ec03df40 0x7f25ec040400 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7f25f0009630 tx=0x7f25f0009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.176+0000 7f25fcff9640 10 client.44165.objecter ms_handle_connect 0x7f25ec03df40 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.180+0000 7f25fcff9640 1 -- 172.21.15.98:0/231353741 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f25f80ac030 con 0x7f26000703c0 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.180+0000 7f25fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:31.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.180+0000 7f25fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:31.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.416+0000 7f26066da640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T02:56:31.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.416+0000 7f26066da640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T02:56:31.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.416+0000 7f26066da640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:31.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.416+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2600069070 con 0x7f26000703c0 2024-09-11T02:56:31.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.424+0000 7f25fcff9640 1 -- 172.21.15.98:0/231353741 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f25f8016520 con 0x7f26000703c0 2024-09-11T02:56:31.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.424+0000 7f25fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T02:56:31.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.424+0000 7f25fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:31.425 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=1,health=11,kv=9,logm=68} crc {auth=146041424,config=1228700967,health=3041346918,kv=4292987402,logm=2170791626}) 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=1,health=12,kv=10,logm=65} crc {auth=3241573959,config=3678701417,health=1185056311,kv=3991532845,logm=3306116247}) 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3027930083,config=692619927,health=3041346918,kv=3991532845,logm=1886235231}) 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.426 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=9,config=2,health=11,kv=10,logm=68} crc {auth=3571865777,config=692619927,health=1447803656,kv=3991532845,logm=3253426659}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=9,config=1,health=12,kv=10,logm=68} crc {auth=2071310238,config=1228700967,health=1185056311,kv=373919883,logm=3604811070}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=3027930083,config=692619927,health=1185056311,kv=3991532845,logm=1571798135}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=10,config=1,health=12,kv=9,logm=68} crc {auth=993711987,config=3678701417,health=1185056311,kv=3994774503,logm=989492858}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=2965665292,config=692619927,health=2415631119,kv=3991532845,logm=2619194175}) 2024-09-11T02:56:31.427 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.425+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=10,config=1,health=12,kv=10,logm=67} crc {auth=788753746,config=1228700967,health=1185056311,kv=3991532845,logm=1619932867}) 2024-09-11T02:56:31.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.424+0000 7f26066da640 10 librados: watch_flush enter 2024-09-11T02:56:31.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.424+0000 7f26066da640 10 librados: watch_flush exit 2024-09-11T02:56:31.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.424+0000 7f26066da640 20 client.44165.objecter shutdown clearing up homeless session... 2024-09-11T02:56:31.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.424+0000 7f26066da640 10 client.44165.objecter successfully canceled tick 2024-09-11T02:56:31.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25ec03df40 msgr2=0x7f25ec040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25ec03df40 0x7f25ec040400 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7f25f0009630 tx=0x7f25f0009240 comp rx=0 tx=0).stop 2024-09-11T02:56:31.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 10 monclient: shutdown 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26000703c0 msgr2=0x7f26001366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26000703c0 0x7f26001366c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f25f80ac000 tx=0x7f25f80192f0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 shutdown_connections 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f25ec03df40 0x7f25ec040400 unknown :-1 s=CLOSED pgs=3506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f26000703c0 0x7f26001366c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f260006faa0 0x7f2600136180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 --2- 172.21.15.98:0/231353741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f260006f6c0 0x7f2600135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 >> 172.21.15.98:0/231353741 conn(0x7f26000acac0 msgr2=0x7f2600072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 shutdown_connections 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 -- 172.21.15.98:0/231353741 wait complete. 2024-09-11T02:56:31.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:31.428+0000 7f26066da640 1 librados: shutdown 2024-09-11T02:56:31.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858007342}) 2024-09-11T02:56:31.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707302098}) 2024-09-11T02:56:31.432 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3744571353}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2485939952}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3464308582}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3064492691}) 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.433 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.434 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2743999332}) 2024-09-11T02:56:31.434 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.434 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797456535}) 2024-09-11T02:56:31.434 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.429+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4162899446}) 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4194231138}) 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1843924774}) 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1053516799}) 2024-09-11T02:56:31.436 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=248784557}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1126869044}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2716114705}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2784901755}) 2024-09-11T02:56:31.437 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666636514}) 2024-09-11T02:56:31.438 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.433+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2502995102}) 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096679970}) 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3546692039}) 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.439 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=47014261}) 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2308499200}) 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=152488247}) 2024-09-11T02:56:31.440 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3519922837}) 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346855072}) 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1162482639}) 2024-09-11T02:56:31.441 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=79706501}) 2024-09-11T02:56:31.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260044145}) 2024-09-11T02:56:31.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.444 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168598439}) 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484371710}) 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4031406860}) 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1498538931}) 2024-09-11T02:56:31.445 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=164908780}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.437+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1561055106}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906474138}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=293659785}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866841588}) 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.446 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1268802993}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2173731477}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=807157033}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2249076999}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.447 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1932675829}) 2024-09-11T02:56:31.448 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.448 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.448 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1773628599}) 2024-09-11T02:56:31.448 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.448 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=180674191}) 2024-09-11T02:56:31.448 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.441+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=74934102}) 2024-09-11T02:56:31.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T02:56:31.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.456 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617851586}) 2024-09-11T02:56:31.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817069712}) 2024-09-11T02:56:31.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.457 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=880426430}) 2024-09-11T02:56:31.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=942547705}) 2024-09-11T02:56:31.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4131341126}) 2024-09-11T02:56:31.458 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3918917681}) 2024-09-11T02:56:31.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1486700051}) 2024-09-11T02:56:31.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.459 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829555610}) 2024-09-11T02:56:31.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.445+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3640406044}) 2024-09-11T02:56:31.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1231241200}) 2024-09-11T02:56:31.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.460 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=771937258}) 2024-09-11T02:56:31.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=273850735}) 2024-09-11T02:56:31.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.461 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1071746912}) 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=449166056}) 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2117693502}) 2024-09-11T02:56:31.462 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3360391521}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657830743}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=299621706}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025614965}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=199240472}) 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.463 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2870457714}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2476928288}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.449+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1910225088}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1864330127}) 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.464 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1014891694}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751873787}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2674923545}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625381565}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537311771}) 2024-09-11T02:56:31.465 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2772129777}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=377896045}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1556392024}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2534898712}) 2024-09-11T02:56:31.466 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.467 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.467 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2469096276}) 2024-09-11T02:56:31.467 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.467 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322558019}) 2024-09-11T02:56:31.467 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.453+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3548097755}) 2024-09-11T02:56:31.473 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340593908}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=843389946}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1668545864}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=696703439}) 2024-09-11T02:56:31.474 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1893978953}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4091146540}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3932809981}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1748621614}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=410605907}) 2024-09-11T02:56:31.475 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169861159}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1678556682}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2872701178}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1626314122}) 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.476 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1086671733}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1582908815}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3469565829}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466275074}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.457+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3042291497}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.477 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727237878}) 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3260665257}) 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1125834237}) 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.478 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1071256024}) 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2287291089}) 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1562967326}) 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2549263220}) 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3712956252}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.461+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=821878590}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943477109}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3527420041}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2234648624}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1676173363}) 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2893073126}) 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=869631704}) 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241521916}) 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3813045093}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.465+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2676911036}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178778531}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1934385061}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=501660645}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2018000073}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3770962615}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4097654999}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3510751356}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863448960}) 2024-09-11T02:56:31.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1563979709}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329708253}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2878019450}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342336189}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3848528043}) 2024-09-11T02:56:31.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2032420035}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2548863957}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=973310865}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646421970}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.469+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=430128480}) 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4196938287}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185333635}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=164630263}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2097344799}) 2024-09-11T02:56:31.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2448830603}) 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3986925400}) 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=805476239}) 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544487116}) 2024-09-11T02:56:31.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.473+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1262676728}) 2024-09-11T02:56:31.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448419643}) 2024-09-11T02:56:31.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1986294200}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=36136442}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2759807836}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=893405133}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2261649562}) 2024-09-11T02:56:31.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1216476610}) 2024-09-11T02:56:31.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863944601}) 2024-09-11T02:56:31.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=292127003}) 2024-09-11T02:56:31.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389885896}) 2024-09-11T02:56:31.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2279794226}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4172991787}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1421190170}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3677562580}) 2024-09-11T02:56:31.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=611668292}) 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4129104116}) 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773106348}) 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.477+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4133959422}) 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294672177}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2865292366}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2369110456}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=183000611}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2494429017}) 2024-09-11T02:56:31.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2964338704}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3842936242}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614359069}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.481+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=37219483}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685053671}) 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654705174}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1254752583}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2919267915}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3571041100}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2545907425}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1053135307}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160856363}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=531777680}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872495206}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2182140632}) 2024-09-11T02:56:31.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2040399740}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1507064813}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1361483800}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3476864216}) 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=922231936}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621876660}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.485+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1494724560}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180692136}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=36880574}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1163562772}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2636883072}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3434785248}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3032861646}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2520996955}) 2024-09-11T02:56:31.522 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596245876}) 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.489+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=590536328}) 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932442374}) 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121804669}) 2024-09-11T02:56:31.523 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1164194761}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3361789237}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2631542449}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=200274942}) 2024-09-11T02:56:31.524 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3203955625}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620293927}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=818144775}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144796650}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1390421512}) 2024-09-11T02:56:31.525 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=95811823}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2959290476}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=682993276}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4100669268}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.526 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1151396655}) 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978278305}) 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.493+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3768128871}) 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848053777}) 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=377202829}) 2024-09-11T02:56:31.527 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3664218923}) 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=957424183}) 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1941610711}) 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.528 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=895266213}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3417965758}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427305396}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.497+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3015429899}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543041681}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.529 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4089701407}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2819835134}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3831030950}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2691305844}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4048720219}) 2024-09-11T02:56:31.530 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1868234669}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788718039}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.501+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=941007403}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444352338}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198323178}) 2024-09-11T02:56:31.531 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=484023208}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2848421366}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2293823449}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4006958080}) 2024-09-11T02:56:31.532 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2857597062}) 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983343134}) 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.505+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1982189199}) 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586709841}) 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.533 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3551329179}) 2024-09-11T02:56:31.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2066301228}) 2024-09-11T02:56:31.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.534 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1695263885}) 2024-09-11T02:56:31.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2484763346}) 2024-09-11T02:56:31.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2221939498}) 2024-09-11T02:56:31.535 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2897908488}) 2024-09-11T02:56:31.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024966763}) 2024-09-11T02:56:31.536 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3462159935}) 2024-09-11T02:56:31.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1122673244}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2055627748}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3162801046}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.509+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1850465591}) 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=102665340}) 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3552504169}) 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3674971790}) 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445581515}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1287861471}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2230637201}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428713876}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2643949061}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2727844031}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1598102014}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2814396808}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707506978}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3915697128}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.513+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1975428730}) 2024-09-11T02:56:31.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247913518}) 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166729158}) 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3904499263}) 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=42252593}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1275042943}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1944221098}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574654026}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451248514}) 2024-09-11T02:56:31.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.517+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=47115217}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192623555}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061384898}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1829014961}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2330598833}) 2024-09-11T02:56:31.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2417288009}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=386921894}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=424520593}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655646091}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.521+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3392407134}) 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1811931337}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415809409}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437446465}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2676965839}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3046832897}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1580104584}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871702495}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=221785940}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2336221745}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561836749}) 2024-09-11T02:56:31.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121379236}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1704331019}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=904776734}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1348006568}) 2024-09-11T02:56:31.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=836285779}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.525+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=316613415}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249497123}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=796146457}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243705643}) 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1247611371}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2779944738}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=664358611}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3070232614}) 2024-09-11T02:56:31.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1311283731}) 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1370192393}) 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157739555}) 2024-09-11T02:56:31.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.529+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=325735935}) 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753691345}) 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173862170}) 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2374593120}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2591160898}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2885997128}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2578831404}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4155789420}) 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074413995}) 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2705178837}) 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782127951}) 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3256952636}) 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2233901273}) 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=107496494}) 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.533+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2506279264}) 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1243620492}) 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3661232484}) 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627734526}) 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2890666602}) 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330908329}) 2024-09-11T02:56:31.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513423862}) 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1549528468}) 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1026770151}) 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3393573614}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2637364945}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3939616184}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204247744}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.537+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3989922997}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.611 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822286284}) 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3077761641}) 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=233992623}) 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1098708775}) 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3598237936}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4078327679}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3681007854}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228479439}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=868273276}) 2024-09-11T02:56:31.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016635321}) 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654307551}) 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3367611122}) 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4287721816}) 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4258042174}) 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1032381929}) 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=220418273}) 2024-09-11T02:56:31.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514574101}) 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.541+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1265447108}) 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090438949}) 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698195882}) 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602004321}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3392867262}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2572843467}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4092174884}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=68548211}) 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422558987}) 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2701224964}) 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3471842064}) 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639339537}) 2024-09-11T02:56:31.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2793464273}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=420313766}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.545+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2763371400}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=163722725}) 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4162041447}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161802554}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=861809391}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2672211277}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535515611}) 2024-09-11T02:56:31.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3043762038}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=354651336}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3375775052}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2771281643}) 2024-09-11T02:56:31.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3512858157}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918838754}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.549+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=566402507}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709582360}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336142104}) 2024-09-11T02:56:31.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2950207049}) 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3128210375}) 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=752125036}) 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1402059361}) 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=603816551}) 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131352833}) 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.553+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=697155006}) 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=20} crc {logm=2408774193,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=898089964,mon_config_key=4199536219,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=2417396548,osdmap=2150180208}) 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=20} crc {logm=2408774193,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=2417396548,osdmap=1571893538}) 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=6,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=22} crc {logm=2408774193,mdsmap=3491474967,mgr=911745861,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1682269933,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=518953061,osdmap=2622509003}) 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=6,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=23} crc {logm=2408774193,mdsmap=3491474967,mgr=2314025951,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1749564801,mon_config_key=871825949,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1224002371,osdmap=72271792}) 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=22} crc {logm=2408774193,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1682269933,mon_config_key=484053719,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=187256361,osdmap=1179248706}) 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=4,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=20} crc {logm=3062296718,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1682269933,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=2417396548,osdmap=661366297}) 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=9,monmap=2,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=24} crc {logm=2408774193,mdsmap=3491474967,mgr=867251544,mgr_metadata=3513241537,mgrstat=1682269933,mon_config_key=1559687142,monmap=416611756,nvmeofgw=1596038039,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1653064365,osdmap=806367807}) 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=21} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1418148450,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2724600101,osd_snap=1919413132,osdmap=4929398}) 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=24} crc {logm=2836099131,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3926526108,mon_config_key=2093508880,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1489146596,osd_pg_creating=2724600101,osd_snap=2978932006,osdmap=724497613}) 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2513576049}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2954504899}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1635817257}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3835851589}) 2024-09-11T02:56:31.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2227206180}) 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=739308138}) 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3758203605}) 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=78756003}) 2024-09-11T02:56:31.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.557+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1902429728}) 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2649660837}) 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4239304259}) 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1974573300}) 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4001256060}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=572715248}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1809012949}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=409336743}) 2024-09-11T02:56:31.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3471076234}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.561+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3090268984}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4157220483}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1873237696}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=133950606}) 2024-09-11T02:56:31.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3648673318}) 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3289021466}) 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=182151641}) 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1954861014}) 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3896059572}) 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.565+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3563751085}) 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2816522742}) 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2321687258}) 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=584186996}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=347999364}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=208840855}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1599809872}) 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2712241844}) 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=29830129}) 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.569+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4246935548}) 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.577+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3302267400}) 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=16} crc {osdmap=142441225}) 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3302267400}) 2024-09-11T02:56:31.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=15} crc {osdmap=2220492964}) 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3302267400}) 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=16} crc {osdmap=1274164797}) 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3302267400}) 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=16} crc {osdmap=1074120267}) 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3302267400}) 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=16} crc {osdmap=495496348}) 2024-09-11T02:56:31.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:56:31.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3302267400}) 2024-09-11T02:56:31.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T02:56:31.581+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=16} crc {osdmap=585121956}) 2024-09-11T02:56:32.451 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:56:32.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:56:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient: get_monmap_and_config 2024-09-11T02:56:32.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient: build_initial_monmap 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient: monmap: 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:32.586155+0000 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:32.586155+0000 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:32.585 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:32.586 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:32.586 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 Processor -- start 2024-09-11T02:56:32.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 -- start start 2024-09-11T02:56:32.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient: init 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient: _add_conns ranks=[10,9,6,5,7,2,3,0,1,4,8] 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f052006f6c0 0x7f0520072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): picked mon.noname-f con 0x7f052006f6c0 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f052006faa0 0x7f052006fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): picked mon.noname-a con 0x7f052006faa0 addr v2:172.21.15.98:3300/0 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05200703c0 0x7f05201397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): picked mon.noname-d con 0x7f05200703c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0520071ff0 con 0x7f052006faa0 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f05200a8750 con 0x7f052006f6c0 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f05200716c0 con 0x7f05200703c0 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f052617e640 10 monclient(hunting): authenticate will time out at 1346.466484s 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05200703c0 0x7f05201397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f052006faa0 0x7f052006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05200703c0 0x7f05201397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:52056/0 (socket says 172.21.15.98:52056) 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f052006faa0 0x7f052006fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57548/0 (socket says 172.21.15.98:57548) 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 1 -- 172.21.15.98:0/3559050994 learned_addr learned my addr 172.21.15.98:0/3559050994 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 10 monclient(hunting): get_auth_request con 0x7f05200703c0 auth_method 0 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): get_auth_request con 0x7f052006faa0 auth_method 0 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051f7fe640 1 --2- 172.21.15.98:0/3559050994 >> v2:172.21.15.190:3300/0 conn(0x7f052006f6c0 0x7f0520072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051f7fe640 10 monclient(hunting): get_auth_request con 0x7f052006f6c0 auth_method 0 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient(hunting): handle_auth_done global_id 44208 payload 274 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 -- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05200703c0 msgr2=0x7f05201397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 --2- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05200703c0 0x7f05201397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 -- 172.21.15.98:0/3559050994 >> v2:172.21.15.190:3300/0 conn(0x7f052006f6c0 msgr2=0x7f0520072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 --2- 172.21.15.98:0/3559050994 >> v2:172.21.15.190:3300/0 conn(0x7f052006f6c0 0x7f0520072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient: _finish_hunting 0 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 monclient: found mon.noname-a 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:32.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051effd640 1 -- 172.21.15.98:0/3559050994 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05200a64a0 con 0x7f052006faa0 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051f7fe640 1 --2- 172.21.15.98:0/3559050994 >> v2:172.21.15.190:3300/0 conn(0x7f052006f6c0 0x7f0520072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.584+0000 7f051ffff640 1 --2- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05200703c0 0x7f05201397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051effd640 1 --2- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052006faa0 0x7f052006fe80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f05200712e0 tx=0x7f051401c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd03697c21faa6af server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 1 -- 172.21.15.98:0/3559050994 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05140b0070 con 0x7f052006faa0 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: dump: 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.594 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:56:32.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 1 -- 172.21.15.98:0/3559050994 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05140aa810 con 0x7f052006faa0 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 1 -- 172.21.15.98:0/3559050994 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05140a9300 con 0x7f052006faa0 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f051e7fc640 10 monclient: dump: 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 5 monclient: authenticate success, global_id 44208 2024-09-11T02:56:32.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 4 set_mon_vals no callback set 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient: shutdown 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052006faa0 msgr2=0x7f052006fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 --2- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052006faa0 0x7f052006fe80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f05200712e0 tx=0x7f051401c080 comp rx=0 tx=0).stop 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- 172.21.15.98:0/3559050994 shutdown_connections 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 --2- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05200703c0 0x7f05201397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 --2- 172.21.15.98:0/3559050994 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052006faa0 0x7f052006fe80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 --2- 172.21.15.98:0/3559050994 >> v2:172.21.15.190:3300/0 conn(0x7f052006f6c0 0x7f0520072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- 172.21.15.98:0/3559050994 >> 172.21.15.98:0/3559050994 conn(0x7f05200acac0 msgr2=0x7f05200a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- 172.21.15.98:0/3559050994 shutdown_connections 2024-09-11T02:56:32.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- 172.21.15.98:0/3559050994 wait complete. 2024-09-11T02:56:32.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient: build_initial_monmap 2024-09-11T02:56:32.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient: monmap: 2024-09-11T02:56:32.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:32.605 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:32.606 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 librados: starting msgr at 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 librados: starting objecter 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 Processor -- start 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- start start 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 librados: setting wanted keys 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 librados: calling monclient init 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient: init 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient: _add_conns ranks=[1,3,4,6,7,0,8,2,5] 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052006f6c0 0x7f05201362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): picked mon.noname-f con 0x7f052006f6c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052006faa0 0x7f0520136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): picked mon.noname-g con 0x7f052006faa0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05200703c0 0x7f052013b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): picked mon.noname-c con 0x7f05200703c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0520143840 con 0x7f052006f6c0 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f05201436c0 con 0x7f052006faa0 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f05201439c0 con 0x7f05200703c0 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.588+0000 7f052617e640 10 monclient(hunting): authenticate will time out at 1346.471902s 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05200703c0 0x7f052013b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05200703c0 0x7f052013b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38178/0 (socket says 172.21.15.98:38178) 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 -- 172.21.15.98:0/2067295327 learned_addr learned my addr 172.21.15.98:0/2067295327 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): get_auth_request con 0x7f05200703c0 auth_method 0 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051effd640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052006faa0 0x7f0520136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051effd640 10 monclient(hunting): get_auth_request con 0x7f052006faa0 auth_method 0 2024-09-11T02:56:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052006f6c0 0x7f05201362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 10 monclient(hunting): get_auth_request con 0x7f052006f6c0 auth_method 0 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient(hunting): handle_auth_done global_id 23773 payload 824 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 -- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052006faa0 msgr2=0x7f0520136830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052006faa0 0x7f0520136830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 -- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052006f6c0 msgr2=0x7f05201362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052006f6c0 0x7f05201362f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 monclient: found mon.noname-c 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 -- 172.21.15.98:0/2067295327 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052013f730 con 0x7f05200703c0 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051effd640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052006faa0 0x7f0520136830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051ffff640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05200703c0 0x7f052013b8c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f051800efc0 tx=0x7f051800c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 client.?.objecter ms_handle_connect 0x7f05200703c0 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 1 -- 172.21.15.98:0/2067295327 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f051800cbe0 con 0x7f05200703c0 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: dump: 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.612 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: _finish_auth 0 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 1 -- 172.21.15.98:0/2067295327 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f051800cd80 con 0x7f05200703c0 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 5 monclient: authenticate success, global_id 23773 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 monclient: _renew_subs 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05201402e0 con 0x7f05200703c0 2024-09-11T02:56:32.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 1 -- 172.21.15.98:0/2067295327 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0518014d60 con 0x7f05200703c0 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 client.23773.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051dffb640 4 set_mon_vals no callback set 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 monclient: dump: 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 monclient: _renew_subs 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f052013f170 con 0x7f05200703c0 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 1 librados: init done 2024-09-11T02:56:32.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 1 -- 172.21.15.98:0/2067295327 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0518014480 con 0x7f05200703c0 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f052013ffe0 con 0x7f05200703c0 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df40 0x7f050c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 1 -- 172.21.15.98:0/2067295327 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(296..296 src has 1..296) ==== 7049+0+0 (secure 0 0 0) 0x7f0518019020 con 0x7f05200703c0 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 client.23773.objecter ms_dispatch 0x7f0520000ba0 osd_map(296..296 src has 1..296) 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 3 client.23773.objecter handle_osd_map got epochs [296,296] > 0 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 3 client.23773.objecter handle_osd_map decoding full epoch 296 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 20 client.23773.objecter dump_active .. 0 homeless 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df40 0x7f050c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 10 monclient: get_auth_request con 0x7f050c03df40 auth_method 0 2024-09-11T02:56:32.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f051f7fe640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df40 0x7f050c040400 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f051000c900 tx=0x7f0510008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.592+0000 7f0507fff640 10 client.23773.objecter ms_handle_connect 0x7f050c03df40 2024-09-11T02:56:32.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.596+0000 7f0507fff640 1 -- 172.21.15.98:0/2067295327 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f052013ffe0 con 0x7f05200703c0 2024-09-11T02:56:32.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.596+0000 7f0507fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:32.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.596+0000 7f0507fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.816+0000 7f052617e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:56:32.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.816+0000 7f052617e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.816+0000 7f052617e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.816+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f052006fe80 con 0x7f05200703c0 2024-09-11T02:56:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.816+0000 7f0507fff640 1 -- 172.21.15.98:0/2067295327 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f051803ec90 con 0x7f05200703c0 2024-09-11T02:56:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.816+0000 7f0507fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:32.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.816+0000 7f0507fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:32.819 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":56,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:56:32.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 10 librados: watch_flush enter 2024-09-11T02:56:32.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 10 librados: watch_flush exit 2024-09-11T02:56:32.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 20 client.23773.objecter shutdown clearing up homeless session... 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 10 client.23773.objecter successfully canceled tick 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df40 msgr2=0x7f050c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df40 0x7f050c040400 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f051000c900 tx=0x7f0510008040 comp rx=0 tx=0).stop 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 10 monclient: shutdown 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05200703c0 msgr2=0x7f052013b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05200703c0 0x7f052013b8c0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f051800efc0 tx=0x7f051800c490 comp rx=0 tx=0).stop 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 shutdown_connections 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df40 0x7f050c040400 unknown :-1 s=CLOSED pgs=3507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05200703c0 0x7f052013b8c0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052006faa0 0x7f0520136830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 --2- 172.21.15.98:0/2067295327 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f052006f6c0 0x7f05201362f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 >> 172.21.15.98:0/2067295327 conn(0x7f05200acac0 msgr2=0x7f05200a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 shutdown_connections 2024-09-11T02:56:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 -- 172.21.15.98:0/2067295327 wait complete. 2024-09-11T02:56:32.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.820+0000 7f052617e640 1 librados: shutdown 2024-09-11T02:56:32.835 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:56:32.835 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T02:56:32.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:56:32.835 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:56:32.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient: get_monmap_and_config 2024-09-11T02:56:32.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient: build_initial_monmap 2024-09-11T02:56:32.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient: monmap: 2024-09-11T02:56:32.952 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:32.952 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:32.953838+0000 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:32.953838+0000 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:32.953 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:32.954 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:32.954 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 1 Processor -- start 2024-09-11T02:56:32.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 1 -- start start 2024-09-11T02:56:32.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient: init 2024-09-11T02:56:32.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:32.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient: _add_conns ranks=[8,1,4,9,10,0,5,7,6,3,2] 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc8980a5930 0x7fc898071ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient(hunting): picked mon.noname-e con 0x7fc8980a5930 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc8980a5d10 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc89813ebc0 0x7fc898140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient(hunting): picked mon.noname-c con 0x7fc89813ebc0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc89806fd10 con 0x7fc8980a5d10 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc89806f310 con 0x7fc89813ebc0 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc89806f4e0 con 0x7fc8980a5930 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89f5e7640 10 monclient(hunting): authenticate will time out at 1346.834189s 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc8980a5930 0x7fc898071ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc89813ebc0 0x7fc898140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc8980a5930 0x7fc898071ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44656/0 (socket says 172.21.15.98:44656) 2024-09-11T02:56:32.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc89813ebc0 0x7fc898140fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38194/0 (socket says 172.21.15.98:38194) 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 -- 172.21.15.98:0/3716134118 learned_addr learned my addr 172.21.15.98:0/3716134118 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): get_auth_request con 0x7fc8980a5930 auth_method 0 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 10 monclient(hunting): get_auth_request con 0x7fc89813ebc0 auth_method 0 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient(hunting): handle_auth_done global_id 24236 payload 274 2024-09-11T02:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 -- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc89813ebc0 msgr2=0x7fc898140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 --2- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc89813ebc0 0x7fc898140fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 -- 172.21.15.98:0/3716134118 >> v1:172.21.15.190:6789/0 conn(0x7fc8980a5d10 legacy=0x7fc8980a60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient: _finish_hunting 0 2024-09-11T02:56:32.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 monclient: found mon.noname-e 2024-09-11T02:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89d383640 1 -- 172.21.15.98:0/3716134118 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc898070860 con 0x7fc8980a5930 2024-09-11T02:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.952+0000 7fc89db84640 1 --2- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc89813ebc0 0x7fc898140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:32.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89d383640 1 --2- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc8980a5930 0x7fc898071ca0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fc8940098a0 tx=0x7fc89401c410 comp rx=0 tx=0).ready entity=mon.8 client_cookie=308f9e108766d761 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 1 -- 172.21.15.98:0/3716134118 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8940b1070 con 0x7fc8980a5930 2024-09-11T02:56:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: dump: 2024-09-11T02:56:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.972 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.973 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: _finish_auth 0 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 1 -- 172.21.15.98:0/3716134118 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8940ab820 con 0x7fc8980a5930 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 1 -- 172.21.15.98:0/3716134118 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8940aa310 con 0x7fc8980a5930 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 5 monclient: authenticate success, global_id 24236 2024-09-11T02:56:32.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 4 set_mon_vals no callback set 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc88ffff640 10 monclient: dump: 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.975 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient: shutdown 2024-09-11T02:56:32.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc8980a5930 msgr2=0x7fc898071ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 --2- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc8980a5930 0x7fc898071ca0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fc8940098a0 tx=0x7fc89401c410 comp rx=0 tx=0).stop 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- 172.21.15.98:0/3716134118 shutdown_connections 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 --2- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc89813ebc0 0x7fc898140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 --2- 172.21.15.98:0/3716134118 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc8980a5930 0x7fc898071ca0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- 172.21.15.98:0/3716134118 >> 172.21.15.98:0/3716134118 conn(0x7fc8980aa520 msgr2=0x7fc8980a6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- 172.21.15.98:0/3716134118 shutdown_connections 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- 172.21.15.98:0/3716134118 wait complete. 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient: build_initial_monmap 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient: monmap: 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:32.977 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:32.978 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:32.979 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 librados: starting msgr at 2024-09-11T02:56:32.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 librados: starting objecter 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 Processor -- start 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- start start 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 librados: setting wanted keys 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 librados: calling monclient init 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient: init 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient: _add_conns ranks=[2,5,8,0,7,3,4,6,1] 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8980a5930 0x7fc8981d1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): picked mon.noname-b con 0x7fc8980a5930 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:32.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8980a5d10 0x7fc89813cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): picked mon.noname-h con 0x7fc8980a5d10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc89813ebc0 0x7fc898123d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): picked mon.noname-e con 0x7fc89813ebc0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc898143e90 con 0x7fc8980a5930 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc898143d10 con 0x7fc8980a5d10 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc898144010 con 0x7fc89813ebc0 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.956+0000 7fc89f5e7640 10 monclient(hunting): authenticate will time out at 1346.839946s 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8980a5930 0x7fc8981d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc89813ebc0 0x7fc898123d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc89813ebc0 0x7fc898123d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44670/0 (socket says 172.21.15.98:44670) 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8980a5930 0x7fc8981d1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:35642/0 (socket says 172.21.15.98:35642) 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 -- 172.21.15.98:0/2758506444 learned_addr learned my addr 172.21.15.98:0/2758506444 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): get_auth_request con 0x7fc89813ebc0 auth_method 0 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): get_auth_request con 0x7fc8980a5930 auth_method 0 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8980a5d10 0x7fc89813cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 10 monclient(hunting): get_auth_request con 0x7fc8980a5d10 auth_method 0 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient(hunting): handle_auth_done global_id 24245 payload 824 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8980a5d10 msgr2=0x7fc89813cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc8980a5d10 0x7fc89813cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8980a5930 msgr2=0x7fc8981d1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc8980a5930 0x7fc8981d1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient: _finish_hunting 0 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 monclient: found mon.noname-e 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:32.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 -- 172.21.15.98:0/2758506444 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc89813fce0 con 0x7fc89813ebc0 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89db84640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc89813ebc0 0x7fc898123d90 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fc89000efc0 tx=0x7fc89000c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 client.?.objecter ms_handle_connect 0x7fc89813ebc0 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc890018030 con 0x7fc89813ebc0 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: dump: 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.985 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.986 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc89000cd30 con 0x7fc89813ebc0 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 5 monclient: authenticate success, global_id 24245 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc890014d60 con 0x7fc89813ebc0 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 10 monclient: _renew_subs 2024-09-11T02:56:32.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8980a86d0 con 0x7fc89813ebc0 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88f7fe640 4 set_mon_vals no callback set 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: dump: 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.988 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.989 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 10 client.24245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 10 monclient: _renew_subs 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8981274a0 con 0x7fc89813ebc0 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 1 librados: init done 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc89813d510 0x7fc8981242d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc89806ca10 con 0x7fc89813d510 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc89813d510 0x7fc8981242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient: get_auth_request con 0x7fc89813d510 auth_method 0 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc890014480 con 0x7fc89813ebc0 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 10 monclient(hunting): handle_auth_done global_id 44217 payload 824 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 1 -- 172.21.15.98:0/2758506444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc898126630 con 0x7fc89813d510 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc87c03df40 0x7fc87c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(296..296 src has 1..296) ==== 7049+0+0 (secure 0 0 0) 0x7fc890019070 con 0x7fc89813ebc0 2024-09-11T02:56:32.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 client.24245.objecter ms_dispatch 0x7fc898000ba0 osd_map(296..296 src has 1..296) 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 3 client.24245.objecter handle_osd_map got epochs [296,296] > 0 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 3 client.24245.objecter handle_osd_map decoding full epoch 296 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89d383640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc89813d510 0x7fc8981242d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8980a73f0 tx=0x7fc8940191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 20 client.24245.objecter dump_active .. 0 homeless 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 client.24245.objecter ms_handle_connect 0x7fc89813d510 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 client.24245.objecter resend_mon_ops 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8940bb030 con 0x7fc89813d510 2024-09-11T02:56:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: dump: 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc87c03df40 0x7fc87c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 10 monclient: get_auth_request con 0x7fc87c03df40 auth_method 0 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc894003620 con 0x7fc89813d510 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc89813d510 msgr2=0x7fc8981242d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc89813d510 0x7fc8981242d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8980a73f0 tx=0x7fc8940191b0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc89813d510 msgr2=0x7fc8981242d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc89813d510 0x7fc8981242d0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc8980a73f0 tx=0x7fc8940191b0 comp rx=0 tx=0).stop 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc88dffb640 10 client.24245.objecter ms_handle_connect 0x7fc87c03df40 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.960+0000 7fc89cb82640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc87c03df40 0x7fc87c040400 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7fc8981254b0 tx=0x7fc888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:32.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898124810 0x7fc898120570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc89806e870 con 0x7fc898124810 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898124810 0x7fc898120570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient: get_auth_request con 0x7fc898124810 auth_method 0 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 10 monclient(hunting): handle_auth_done global_id 44226 payload 824 2024-09-11T02:56:32.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 1 -- 172.21.15.98:0/2758506444 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc898124c20 con 0x7fc898124810 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89d383640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898124810 0x7fc898120570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc89401aac0 tx=0x7fc8940a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 10 client.24245.objecter ms_handle_connect 0x7fc898124810 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 10 client.24245.objecter resend_mon_ops 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8940bb040 con 0x7fc898124810 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 10 monclient: dump: 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:32.997 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:32.998 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7fc8940039e0 con 0x7fc898124810 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898124810 msgr2=0x7fc898120570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898124810 0x7fc898120570 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc89401aac0 tx=0x7fc8940a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898124810 msgr2=0x7fc898120570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc88dffb640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc898124810 0x7fc898120570 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc89401aac0 tx=0x7fc8940a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:32.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19407, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:33.000 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.001 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.002 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:33.003 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.004 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.005 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:33.006 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.007 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:33.008 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.009 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.010 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:33.012 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.014 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.016 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.017 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:33.018 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:33.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.023 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.027 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:33.028 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 10 librados: watch_flush enter 2024-09-11T02:56:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 10 librados: watch_flush exit 2024-09-11T02:56:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 20 client.24245.objecter shutdown clearing up homeless session... 2024-09-11T02:56:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 10 client.24245.objecter successfully canceled tick 2024-09-11T02:56:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc87c03df40 msgr2=0x7fc87c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc87c03df40 0x7fc87c040400 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7fc8981254b0 tx=0x7fc888009040 comp rx=0 tx=0).stop 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 10 monclient: shutdown 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc89813ebc0 msgr2=0x7fc898123d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 --2- 172.21.15.98:0/2758506444 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc89813ebc0 0x7fc898123d90 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fc89000efc0 tx=0x7fc89000c490 comp rx=0 tx=0).stop 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89db84640 1 -- 172.21.15.98:0/2758506444 reap_dead start 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89db84640 1 -- 172.21.15.98:0/2758506444 reap_dead start 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 shutdown_connections 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 >> 172.21.15.98:0/2758506444 conn(0x7fc8980aa520 msgr2=0x7fc89813f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 shutdown_connections 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 -- 172.21.15.98:0/2758506444 wait complete. 2024-09-11T02:56:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:32.968+0000 7fc89f5e7640 1 librados: shutdown 2024-09-11T02:56:33.032 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:33.032 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:56:33.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:56:33.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient: get_monmap_and_config 2024-09-11T02:56:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient: monmap: 2024-09-11T02:56:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:33.158024+0000 2024-09-11T02:56:33.157 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:33.158024+0000 2024-09-11T02:56:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:33.160 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:33.160 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 1 Processor -- start 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 1 -- start start 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient: init 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient: _add_conns ranks=[4,1,10,5,2,0,7,8,9,3,6] 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2270139b10 0x7f2270072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient(hunting): picked mon.noname-c con 0x7f2270139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2270139ef0 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f227013a2d0 0x7f22700a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient(hunting): picked mon.noname-f con 0x7f227013a2d0 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:33.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22700702b0 con 0x7f2270139ef0 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2270071bf0 con 0x7f227013a2d0 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2270071350 con 0x7f2270139b10 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f2274e94640 10 monclient(hunting): authenticate will time out at 1347.038321s 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2270139b10 0x7f2270072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2270139b10 0x7f2270072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38200/0 (socket says 172.21.15.98:38200) 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 -- 172.21.15.98:0/3880832467 learned_addr learned my addr 172.21.15.98:0/3880832467 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): get_auth_request con 0x7f2270139b10 auth_method 0 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226ed9d640 1 --2- 172.21.15.98:0/3880832467 >> v2:172.21.15.190:3300/0 conn(0x7f227013a2d0 0x7f22700a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226ed9d640 10 monclient(hunting): get_auth_request con 0x7f227013a2d0 auth_method 0 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient(hunting): handle_auth_done global_id 23791 payload 274 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 -- 172.21.15.98:0/3880832467 >> v2:172.21.15.190:3300/0 conn(0x7f227013a2d0 msgr2=0x7f22700a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:33.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 --2- 172.21.15.98:0/3880832467 >> v2:172.21.15.190:3300/0 conn(0x7f227013a2d0 0x7f22700a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 -- 172.21.15.98:0/3880832467 >> v1:172.21.15.190:6789/0 conn(0x7f2270139ef0 legacy=0x7f22700a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 monclient: found mon.noname-c 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.156+0000 7f226e59c640 1 -- 172.21.15.98:0/3880832467 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22700a9620 con 0x7f2270139b10 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226e59c640 1 --2- 172.21.15.98:0/3880832467 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2270139b10 0x7f2270072600 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f2268009e50 tx=0x7f226801c980 comp rx=0 tx=0).ready entity=mon.4 client_cookie=c2c2804253de3157 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 1 -- 172.21.15.98:0/3880832467 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22680090d0 con 0x7f2270139b10 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: dump: 2024-09-11T02:56:33.178 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.179 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 1 -- 172.21.15.98:0/3880832467 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22680a5ab0 con 0x7f2270139b10 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 5 monclient: authenticate success, global_id 23791 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 1 -- 172.21.15.98:0/3880832467 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22680a5ca0 con 0x7f2270139b10 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:33.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 4 set_mon_vals no callback set 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226d59a640 10 monclient: dump: 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.181 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.182 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.183 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient: shutdown 2024-09-11T02:56:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- 172.21.15.98:0/3880832467 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2270139b10 msgr2=0x7f2270072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 --2- 172.21.15.98:0/3880832467 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2270139b10 0x7f2270072600 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f2268009e50 tx=0x7f226801c980 comp rx=0 tx=0).stop 2024-09-11T02:56:33.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- 172.21.15.98:0/3880832467 shutdown_connections 2024-09-11T02:56:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 --2- 172.21.15.98:0/3880832467 >> v2:172.21.15.190:3300/0 conn(0x7f227013a2d0 0x7f22700a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 --2- 172.21.15.98:0/3880832467 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2270139b10 0x7f2270072600 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- 172.21.15.98:0/3880832467 >> 172.21.15.98:0/3880832467 conn(0x7f22700ac790 msgr2=0x7f22700a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- 172.21.15.98:0/3880832467 shutdown_connections 2024-09-11T02:56:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- 172.21.15.98:0/3880832467 wait complete. 2024-09-11T02:56:33.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient: monmap: 2024-09-11T02:56:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:33.185 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.186 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 librados: starting msgr at 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 librados: starting objecter 2024-09-11T02:56:33.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 Processor -- start 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- start start 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 librados: setting wanted keys 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 librados: calling monclient init 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient: init 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient: _add_conns ranks=[5,8,7,4,1,0,3,6,2] 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2270139b10 0x7f22701d1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): picked mon.noname-h con 0x7f2270139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2270139ef0 0x7f22701d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): picked mon.noname-e con 0x7f2270139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f227013a2d0 0x7f22701d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): picked mon.noname-i con 0x7f227013a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f2270144470 con 0x7f2270139b10 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f22700a8870 con 0x7f227013a2d0 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f22701445f0 con 0x7f2270139ef0 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f2274e94640 10 monclient(hunting): authenticate will time out at 1347.043420s 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2270139ef0 0x7f22701d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2270139ef0 0x7f22701d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44682/0 (socket says 172.21.15.98:44682) 2024-09-11T02:56:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 -- 172.21.15.98:0/2793494215 learned_addr learned my addr 172.21.15.98:0/2793494215 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): get_auth_request con 0x7f2270139ef0 auth_method 0 2024-09-11T02:56:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226e59c640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2270139b10 0x7f22701d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226ed9d640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f227013a2d0 0x7f22701d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226e59c640 10 monclient(hunting): get_auth_request con 0x7f2270139b10 auth_method 0 2024-09-11T02:56:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226ed9d640 10 monclient(hunting): get_auth_request con 0x7f227013a2d0 auth_method 0 2024-09-11T02:56:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient(hunting): handle_auth_done global_id 24254 payload 824 2024-09-11T02:56:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f227013a2d0 msgr2=0x7f22701d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f227013a2d0 0x7f22701d1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2270139b10 msgr2=0x7f22701d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2270139b10 0x7f22701d1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 monclient: found mon.noname-e 2024-09-11T02:56:33.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.160+0000 7f226dd9b640 1 -- 172.21.15.98:0/2793494215 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22700aa090 con 0x7f2270139ef0 2024-09-11T02:56:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226dd9b640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2270139ef0 0x7f22701d1a40 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f226400c500 tx=0x7f226400c9c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 client.?.objecter ms_handle_connect 0x7f2270139ef0 2024-09-11T02:56:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2264017070 con 0x7f2270139ef0 2024-09-11T02:56:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: dump: 2024-09-11T02:56:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.198 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2264003600 con 0x7f2270139ef0 2024-09-11T02:56:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2264003ca0 con 0x7f2270139ef0 2024-09-11T02:56:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 5 monclient: authenticate success, global_id 24254 2024-09-11T02:56:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 10 monclient: _renew_subs 2024-09-11T02:56:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227013ce00 con 0x7f2270139ef0 2024-09-11T02:56:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226cd99640 4 set_mon_vals no callback set 2024-09-11T02:56:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: dump: 2024-09-11T02:56:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.201 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 10 client.24254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 10 monclient: _renew_subs 2024-09-11T02:56:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:33.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f227013b890 con 0x7f2270139ef0 2024-09-11T02:56:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 1 librados: init done 2024-09-11T02:56:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a4910 0x7f22700a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f227013ab30 con 0x7f22700a4910 2024-09-11T02:56:33.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a4910 0x7f22700a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient: get_auth_request con 0x7f22700a4910 auth_method 0 2024-09-11T02:56:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2264003830 con 0x7f2270139ef0 2024-09-11T02:56:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 10 monclient(hunting): handle_auth_done global_id 44183 payload 824 2024-09-11T02:56:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f225003df40 0x7f2250040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 1 -- 172.21.15.98:0/2793494215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f227006e680 con 0x7f22700a4910 2024-09-11T02:56:33.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(296..296 src has 1..296) ==== 7049+0+0 (secure 0 0 0) 0x7f226404ebe0 con 0x7f2270139ef0 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 client.24254.objecter ms_dispatch 0x7f2270000ba0 osd_map(296..296 src has 1..296) 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 3 client.24254.objecter handle_osd_map got epochs [296,296] > 0 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 3 client.24254.objecter handle_osd_map decoding full epoch 296 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226e59c640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a4910 0x7f22700a4cf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f22700a9590 tx=0x7f2268019720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 20 client.24254.objecter dump_active .. 0 homeless 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 client.24254.objecter ms_handle_connect 0x7f22700a4910 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 client.24254.objecter resend_mon_ops 2024-09-11T02:56:33.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22680bd070 con 0x7f22700a4910 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226ed9d640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f225003df40 0x7f2250040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: dump: 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226ed9d640 10 monclient: get_auth_request con 0x7f225003df40 auth_method 0 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2268004040 con 0x7f22700a4910 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:33.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a4910 msgr2=0x7f22700a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a4910 0x7f22700a4cf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f22700a9590 tx=0x7f2268019720 comp rx=0 tx=0).stop 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a4910 msgr2=0x7f22700a4cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a4910 0x7f22700a4cf0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f22700a9590 tx=0x7f2268019720 comp rx=0 tx=0).stop 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f225effd640 10 client.24254.objecter ms_handle_connect 0x7f225003df40 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.164+0000 7f226ed9d640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f225003df40 0x7f2250040400 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f226000d8d0 tx=0x7f2260008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a5230 0x7f22700a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2270073930 con 0x7f22700a5230 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a5230 0x7f22700a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient: get_auth_request con 0x7f22700a5230 auth_method 0 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 10 monclient(hunting): handle_auth_done global_id 44192 payload 824 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 1 -- 172.21.15.98:0/2793494215 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22700a5e80 con 0x7f22700a5230 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226e59c640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a5230 0x7f22700a0e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2268000f90 tx=0x7f22680a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 10 client.24254.objecter ms_handle_connect 0x7f22700a5230 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 10 client.24254.objecter resend_mon_ops 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22680c2e30 con 0x7f22700a5230 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 10 monclient: dump: 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.212 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.213 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f22680c6070 con 0x7f22700a5230 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a5230 msgr2=0x7f22700a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a5230 0x7f22700a0e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2268000f90 tx=0x7f22680a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a5230 msgr2=0x7f22700a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f225effd640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f22700a5230 0x7f22700a0e60 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2268000f90 tx=0x7f22680a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19607, 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:33.215 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:33.216 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:33.217 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.218 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:33.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.220 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:33.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.227 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.230 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.232 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.234 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:33.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:33.238 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.239 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:33.240 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 10 librados: watch_flush enter 2024-09-11T02:56:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 10 librados: watch_flush exit 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 20 client.24254.objecter shutdown clearing up homeless session... 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 10 client.24254.objecter successfully canceled tick 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f225003df40 msgr2=0x7f2250040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f225003df40 0x7f2250040400 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f226000d8d0 tx=0x7f2260008040 comp rx=0 tx=0).stop 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 10 monclient: shutdown 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2270139ef0 msgr2=0x7f22701d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 --2- 172.21.15.98:0/2793494215 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2270139ef0 0x7f22701d1a40 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f226400c500 tx=0x7f226400c9c0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226ed9d640 1 -- 172.21.15.98:0/2793494215 reap_dead start 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f226ed9d640 1 -- 172.21.15.98:0/2793494215 reap_dead start 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 shutdown_connections 2024-09-11T02:56:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 >> 172.21.15.98:0/2793494215 conn(0x7f22700ac790 msgr2=0x7f22700a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 shutdown_connections 2024-09-11T02:56:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 -- 172.21.15.98:0/2793494215 wait complete. 2024-09-11T02:56:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.172+0000 7f2274e94640 1 librados: shutdown 2024-09-11T02:56:33.243 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:33.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:56:33.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:56:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient: get_monmap_and_config 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient: monmap: 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:33.370369+0000 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:33.370369+0000 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.369 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:33.370 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 1 Processor -- start 2024-09-11T02:56:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 1 -- start start 2024-09-11T02:56:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient: init 2024-09-11T02:56:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient: _add_conns ranks=[9,0,10,6,4,8,3,7,2,5,1] 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49c4139a70 0x7f49c4139e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f49c4139a70 addr v2:172.21.15.98:3300/0 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49c406f9a0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f49c4071af0 0x7f49c40a5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient(hunting): picked mon.noname-f con 0x7f49c4071af0 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c406f090 con 0x7f49c406f9a0 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f49c40703b0 con 0x7f49c4139a70 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f49c4070690 con 0x7f49c4071af0 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c9e9a640 10 monclient(hunting): authenticate will time out at 1347.250635s 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f49c4139a70 0x7f49c4139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49c406f9a0 0x7f49c40a58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54388/0 (socket says 172.21.15.98:54388) 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c2ffd640 1 -- 172.21.15.98:0/2015618817 learned_addr learned my addr 172.21.15.98:0/2015618817 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): get_auth_request con 0x7f49c4139a70 auth_method 0 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c3fff640 1 --2- 172.21.15.98:0/2015618817 >> v2:172.21.15.190:3300/0 conn(0x7f49c4071af0 0x7f49c40a5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient(hunting): handle_auth_done global_id 44235 payload 274 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 1 -- 172.21.15.98:0/2015618817 >> v2:172.21.15.190:3300/0 conn(0x7f49c4071af0 msgr2=0x7f49c40a5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:33.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 1 --2- 172.21.15.98:0/2015618817 >> v2:172.21.15.190:3300/0 conn(0x7f49c4071af0 0x7f49c40a5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 1 -- 172.21.15.98:0/2015618817 >> v1:172.21.15.98:6789/0 conn(0x7f49c406f9a0 legacy=0x7f49c40a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 1 monclient: found mon.noname-a 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c37fe640 1 -- 172.21.15.98:0/2015618817 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c406f600 con 0x7f49c4139a70 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c27fc640 1 -- 172.21.15.98:0/2015618817 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485593488 0 0) 0x7f49c406f090 con 0x7f49c406f9a0 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c27fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c3fff640 10 monclient: get_auth_request con 0x7f49c4071af0 auth_method 0 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.368+0000 7f49c3fff640 1 --2- 172.21.15.98:0/2015618817 >> v2:172.21.15.190:3300/0 conn(0x7f49c4071af0 0x7f49c40a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 1 --2- 172.21.15.98:0/2015618817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4139a70 0x7f49c4139e50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f49c4070a00 tx=0x7f49b401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc3b04671c8cff4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 1 -- 172.21.15.98:0/2015618817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49b40b2070 con 0x7f49c4139a70 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: dump: 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.378 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 5 monclient: authenticate success, global_id 44235 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 1 -- 172.21.15.98:0/2015618817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c40a6720 con 0x7f49c4139a70 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 1 -- 172.21.15.98:0/2015618817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49b40ab340 con 0x7f49c4139a70 2024-09-11T02:56:33.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 4 set_mon_vals no callback set 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c27fc640 10 monclient: dump: 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.380 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient: shutdown 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- 172.21.15.98:0/2015618817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4139a70 msgr2=0x7f49c4139e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 --2- 172.21.15.98:0/2015618817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4139a70 0x7f49c4139e50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f49c4070a00 tx=0x7f49b401c7f0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- 172.21.15.98:0/2015618817 shutdown_connections 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 --2- 172.21.15.98:0/2015618817 >> v2:172.21.15.190:3300/0 conn(0x7f49c4071af0 0x7f49c40a5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 --2- 172.21.15.98:0/2015618817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4139a70 0x7f49c4139e50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- 172.21.15.98:0/2015618817 >> 172.21.15.98:0/2015618817 conn(0x7f49c40aa4b0 msgr2=0x7f49c40a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- 172.21.15.98:0/2015618817 shutdown_connections 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- 172.21.15.98:0/2015618817 wait complete. 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient: monmap: 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:33.382 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 librados: starting msgr at 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 librados: starting objecter 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 Processor -- start 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- start start 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 librados: setting wanted keys 2024-09-11T02:56:33.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 librados: calling monclient init 2024-09-11T02:56:33.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient: init 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient: _add_conns ranks=[4,3,1,6,5,2,8,0,7] 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c406f9a0 0x7f49c4138140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f49c406f9a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49c4071af0 0x7f49c413a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): picked mon.noname-g con 0x7f49c4071af0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49c4139a70 0x7f49c413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): picked mon.noname-f con 0x7f49c4139a70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f49c41444a0 con 0x7f49c4139a70 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f49c4144320 con 0x7f49c4071af0 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f49c4144620 con 0x7f49c406f9a0 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c9e9a640 10 monclient(hunting): authenticate will time out at 1347.255740s 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c406f9a0 0x7f49c4138140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c406f9a0 0x7f49c4138140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38216/0 (socket says 172.21.15.98:38216) 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 1 -- 172.21.15.98:0/183832107 learned_addr learned my addr 172.21.15.98:0/183832107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 10 monclient(hunting): get_auth_request con 0x7f49c406f9a0 auth_method 0 2024-09-11T02:56:33.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.372+0000 7f49c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49c4071af0 0x7f49c413a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c3fff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49c4139a70 0x7f49c413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 10 monclient(hunting): handle_auth_done global_id 23800 payload 824 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49c4071af0 msgr2=0x7f49c413a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f49c4071af0 0x7f49c413a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49c4139a70 msgr2=0x7f49c413aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49c4139a70 0x7f49c413aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 1 monclient: found mon.noname-c 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 1 -- 172.21.15.98:0/183832107 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c4138ae0 con 0x7f49c406f9a0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c3fff640 10 monclient: get_auth_request con 0x7f49c4139a70 auth_method 0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c3fff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49c4139a70 0x7f49c413aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c37fe640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c406f9a0 0x7f49c4138140 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f49b40b0040 tx=0x7f49b4019590 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 client.?.objecter ms_handle_connect 0x7f49c406f9a0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49b40b2070 con 0x7f49c406f9a0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: dump: 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.394 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49b40bc070 con 0x7f49c406f9a0 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 5 monclient: authenticate success, global_id 23800 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 10 monclient: _renew_subs 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c413cd00 con 0x7f49c406f9a0 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49b40a5a30 con 0x7f49c406f9a0 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 10 client.23800.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c1ffb640 4 set_mon_vals no callback set 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: dump: 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.396 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 10 monclient: _renew_subs 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c4141450 con 0x7f49c406f9a0 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 1 librados: init done 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c413b020 0x7f49c40a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f49c406ca10 con 0x7f49c413b020 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c413b020 0x7f49c40a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f49b40a5bd0 con 0x7f49c406f9a0 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient: get_auth_request con 0x7f49c413b020 auth_method 0 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_done global_id 23809 payload 824 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 1 -- 172.21.15.98:0/183832107 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49c4072a40 con 0x7f49c413b020 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b003df40 0x7f49b0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(296..296 src has 1..296) ==== 7049+0+0 (secure 0 0 0) 0x7f49b40ea4e0 con 0x7f49c406f9a0 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 client.23800.objecter ms_dispatch 0x7f49c4000ba0 osd_map(296..296 src has 1..296) 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 3 client.23800.objecter handle_osd_map got epochs [296,296] > 0 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 3 client.23800.objecter handle_osd_map decoding full epoch 296 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c2ffd640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c413b020 0x7f49c40a1b70 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f49c4138090 tx=0x7f49b800d880 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 20 client.23800.objecter dump_active .. 0 homeless 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 client.23800.objecter ms_handle_connect 0x7f49c413b020 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 client.23800.objecter resend_mon_ops 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49b80047f0 con 0x7f49c413b020 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: dump: 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c3fff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b003df40 0x7f49b0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c3fff640 10 monclient: get_auth_request con 0x7f49b003df40 auth_method 0 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f49b80072e0 con 0x7f49c413b020 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c413b020 msgr2=0x7f49c40a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c413b020 0x7f49c40a1b70 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f49c4138090 tx=0x7f49b800d880 comp rx=0 tx=0).stop 2024-09-11T02:56:33.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c413b020 msgr2=0x7f49c40a1b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c413b020 0x7f49c40a1b70 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49abfff640 10 client.23800.objecter ms_handle_connect 0x7f49b003df40 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.376+0000 7f49c3fff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b003df40 0x7f49b0040400 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f49c41387a0 tx=0x7f49bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c40a20b0 0x7f49c40a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f49c406e870 con 0x7f49c40a20b0 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c40a20b0 0x7f49c40a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient: get_auth_request con 0x7f49c40a20b0 auth_method 0 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 10 monclient(hunting): handle_auth_done global_id 23818 payload 824 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 1 -- 172.21.15.98:0/183832107 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f49c40a6720 con 0x7f49c40a20b0 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 10 client.23800.objecter ms_handle_connect 0x7f49c40a20b0 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 10 client.23800.objecter resend_mon_ops 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c2ffd640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c40a20b0 0x7f49c40a24c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f49b8008fd0 tx=0x7f49b8000ee0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f49b801bbe0 con 0x7f49c40a20b0 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 10 monclient: dump: 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f49b8013420 con 0x7f49c40a20b0 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c40a20b0 msgr2=0x7f49c40a24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c40a20b0 0x7f49c40a24c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f49b8008fd0 tx=0x7f49b8000ee0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c40a20b0 msgr2=0x7f49c40a24c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49abfff640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c40a20b0 0x7f49c40a24c0 secure :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f49b8008fd0 tx=0x7f49b8000ee0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:33.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 409168, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:33.406 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.407 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.408 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:33.409 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.410 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.411 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.417 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.421 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:33.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.426 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:33.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:56:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:33.431 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 10 librados: watch_flush enter 2024-09-11T02:56:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 10 librados: watch_flush exit 2024-09-11T02:56:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 20 client.23800.objecter shutdown clearing up homeless session... 2024-09-11T02:56:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 10 client.23800.objecter successfully canceled tick 2024-09-11T02:56:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b003df40 msgr2=0x7f49b0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f49b003df40 0x7f49b0040400 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f49c41387a0 tx=0x7f49bc008040 comp rx=0 tx=0).stop 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 10 monclient: shutdown 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c406f9a0 msgr2=0x7f49c4138140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 --2- 172.21.15.98:0/183832107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f49c406f9a0 0x7f49c4138140 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f49b40b0040 tx=0x7f49b4019590 comp rx=0 tx=0).stop 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c3fff640 1 -- 172.21.15.98:0/183832107 reap_dead start 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c3fff640 1 -- 172.21.15.98:0/183832107 reap_dead start 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 shutdown_connections 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 >> 172.21.15.98:0/183832107 conn(0x7f49c40aa4b0 msgr2=0x7f49c4134b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 shutdown_connections 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 -- 172.21.15.98:0/183832107 wait complete. 2024-09-11T02:56:33.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.384+0000 7f49c9e9a640 1 librados: shutdown 2024-09-11T02:56:33.434 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:33.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:56:33.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:56:33.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.556+0000 7fb5c3619640 10 monclient: get_monmap_and_config 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.556+0000 7fb5c3619640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient: monmap: 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:33.561314+0000 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:33.561314+0000 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.560 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.561 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.561 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.561 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.561 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.561 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:33.561 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:33.561 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 Processor -- start 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 -- start start 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient: init 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient: _add_conns ranks=[10,5,9,3,2,4,0,6,1,8,7] 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fb5bc09f4a0 0x7fb5bc09f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): picked mon.noname-f con 0x7fb5bc09f4a0 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:33.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5bc0a2bd0 0x7fb5bc0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): picked mon.noname-h con 0x7fb5bc0a2bd0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5bc0738d0 0x7fb5bc0a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): picked mon.noname-a con 0x7fb5bc0738d0 addr v2:172.21.15.98:3300/0 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb5bc09cff0 con 0x7fb5bc0738d0 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb5bc0a3050 con 0x7fb5bc09f4a0 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb5bc070110 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c3619640 10 monclient(hunting): authenticate will time out at 1347.441697s 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5bc0738d0 0x7fb5bc0a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb5bc0738d0 0x7fb5bc0a4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57588/0 (socket says 172.21.15.98:57588) 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/623986528 learned_addr learned my addr 172.21.15.98:0/623986528 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): get_auth_request con 0x7fb5bc0738d0 auth_method 0 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c0bb4640 1 --2- 172.21.15.98:0/623986528 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5bc0a2bd0 0x7fb5bc0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c13b5640 1 --2- 172.21.15.98:0/623986528 >> v2:172.21.15.190:3300/0 conn(0x7fb5bc09f4a0 0x7fb5bc09f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c13b5640 10 monclient(hunting): get_auth_request con 0x7fb5bc09f4a0 auth_method 0 2024-09-11T02:56:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c13b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c13b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c13b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c0bb4640 10 monclient(hunting): get_auth_request con 0x7fb5bc0a2bd0 auth_method 0 2024-09-11T02:56:33.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c0bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c0bb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c0bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_done global_id 44253 payload 274 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/623986528 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5bc0a2bd0 msgr2=0x7fb5bc0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 --2- 172.21.15.98:0/623986528 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5bc0a2bd0 0x7fb5bc0a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/623986528 >> v2:172.21.15.190:3300/0 conn(0x7fb5bc09f4a0 msgr2=0x7fb5bc09f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 --2- 172.21.15.98:0/623986528 >> v2:172.21.15.190:3300/0 conn(0x7fb5bc09f4a0 0x7fb5bc09f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 monclient: found mon.noname-a 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.560+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/623986528 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc09fdc0 con 0x7fb5bc0738d0 2024-09-11T02:56:33.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 --2- 172.21.15.98:0/623986528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5bc0738d0 0x7fb5bc0a4be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc076d20 tx=0x7fb5b401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e9e7fb9156909b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 1 -- 172.21.15.98:0/623986528 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5b40090d0 con 0x7fb5bc0738d0 2024-09-11T02:56:33.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: dump: 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 1 -- 172.21.15.98:0/623986528 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b40a5ab0 con 0x7fb5bc0738d0 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 5 monclient: authenticate success, global_id 44253 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:33.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 1 -- 172.21.15.98:0/623986528 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5b40a5c60 con 0x7fb5bc0738d0 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 4 set_mon_vals no callback set 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b3fff640 10 monclient: dump: 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: shutdown 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- 172.21.15.98:0/623986528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5bc0738d0 msgr2=0x7fb5bc0a4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 --2- 172.21.15.98:0/623986528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5bc0738d0 0x7fb5bc0a4be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc076d20 tx=0x7fb5b401c8b0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- 172.21.15.98:0/623986528 shutdown_connections 2024-09-11T02:56:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 --2- 172.21.15.98:0/623986528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5bc0738d0 0x7fb5bc0a4be0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 --2- 172.21.15.98:0/623986528 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5bc0a2bd0 0x7fb5bc0a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 --2- 172.21.15.98:0/623986528 >> v2:172.21.15.190:3300/0 conn(0x7fb5bc09f4a0 0x7fb5bc09f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- 172.21.15.98:0/623986528 >> 172.21.15.98:0/623986528 conn(0x7fb5bc06eed0 msgr2=0x7fb5bc06f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- 172.21.15.98:0/623986528 shutdown_connections 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- 172.21.15.98:0/623986528 wait complete. 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: monmap: 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.581 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 librados: starting msgr at 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 librados: starting objecter 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 Processor -- start 2024-09-11T02:56:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- start start 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 librados: setting wanted keys 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 librados: calling monclient init 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: init 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: _add_conns ranks=[1,8,2,3,0,5,4,6,7] 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0738d0 0x7fb5bc1cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): picked mon.noname-f con 0x7fb5bc0738d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5bc09f4a0 0x7fb5bc1cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): picked mon.noname-e con 0x7fb5bc09f4a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5bc0a2bd0 0x7fb5bc1cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): picked mon.noname-b con 0x7fb5bc0a2bd0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc141330 con 0x7fb5bc0738d0 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb5bc0a5530 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb5bc1414b0 con 0x7fb5bc09f4a0 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient(hunting): authenticate will time out at 1347.446885s 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5bc09f4a0 0x7fb5bc1cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5bc0a2bd0 0x7fb5bc1cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5bc09f4a0 0x7fb5bc1cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44698/0 (socket says 172.21.15.98:44698) 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): get_auth_request con 0x7fb5bc0a2bd0 auth_method 0 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 1 -- 172.21.15.98:0/2149992325 learned_addr learned my addr 172.21.15.98:0/2149992325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 10 monclient(hunting): get_auth_request con 0x7fb5bc09f4a0 auth_method 0 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c13b5640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0738d0 0x7fb5bc1cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c13b5640 10 monclient(hunting): get_auth_request con 0x7fb5bc0738d0 auth_method 0 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c13b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c13b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c13b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient(hunting): handle_auth_done global_id 44201 payload 824 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5bc09f4a0 msgr2=0x7fb5bc1cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5bc09f4a0 0x7fb5bc1cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0738d0 msgr2=0x7fb5bc1cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5bc0738d0 0x7fb5bc1cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 monclient: found mon.noname-b 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/2149992325 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc13cf50 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c0bb4640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb5bc09f4a0 0x7fb5bc1cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c1bb6640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5bc0a2bd0 0x7fb5bc1cfe50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc138650 tx=0x7fb5b4019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 client.?.objecter ms_handle_connect 0x7fb5bc0a2bd0 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5b40bc030 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: dump: 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.587 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b4003e10 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5b40044b0 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 5 monclient: authenticate success, global_id 44201 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: _renew_subs 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5bc0a7ea0 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b37fe640 4 set_mon_vals no callback set 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:33.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 client.44201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5b1ffb640 10 monclient: dump: 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.590 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: _renew_subs 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5bc13c0e0 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.564+0000 7fb5c3619640 1 librados: init done 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c3619640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c3619640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ae0 0x7fb5bc02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb5bc06ca10 con 0x7fb5bc139ae0 2024-09-11T02:56:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb5b4004650 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ae0 0x7fb5bc02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient: get_auth_request con 0x7fb5bc139ae0 auth_method 0 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb5a003df40 0x7fb5a0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_done global_id 44223 payload 824 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 1 -- 172.21.15.98:0/2149992325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5bc09f880 con 0x7fb5bc139ae0 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(297..297 src has 1..297) ==== 7075+0+0 (secure 0 0 0) 0x7fb5b40bd070 con 0x7fb5bc0a2bd0 2024-09-11T02:56:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 client.44201.objecter ms_dispatch 0x7fb5bc000ba0 osd_map(297..297 src has 1..297) 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 3 client.44201.objecter handle_osd_map got epochs [297,297] > 0 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 3 client.44201.objecter handle_osd_map decoding full epoch 297 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 20 client.44201.objecter dump_active .. 0 homeless 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 client.44201.objecter ms_handle_connect 0x7fb5bc139ae0 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c13b5640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ae0 0x7fb5bc02d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800f4b0 tx=0x7fb5b800f970 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 client.44201.objecter resend_mon_ops 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c0bb4640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb5a003df40 0x7fb5a0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5b8019020 con 0x7fb5bc139ae0 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 monclient: dump: 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c0bb4640 10 monclient: get_auth_request con 0x7fb5a003df40 auth_method 0 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5b8014670 con 0x7fb5bc139ae0 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ae0 msgr2=0x7fb5bc02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ae0 0x7fb5bc02d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800f4b0 tx=0x7fb5b800f970 comp rx=0 tx=0).stop 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ae0 msgr2=0x7fb5bc02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ae0 0x7fb5bc02d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5b1ffb640 10 client.44201.objecter ms_handle_connect 0x7fb5a003df40 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.568+0000 7fb5c0bb4640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb5a003df40 0x7fb5a0040400 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc13aa80 tx=0x7fb5ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.572+0000 7fb5c3619640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.572+0000 7fb5c3619640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ec0 0x7fb5bc02d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.572+0000 7fb5c3619640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.572+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb5bc06e870 con 0x7fb5bc139ec0 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ec0 0x7fb5bc02d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient: get_auth_request con 0x7fb5bc139ec0 auth_method 0 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 10 monclient(hunting): handle_auth_done global_id 44232 payload 824 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 1 -- 172.21.15.98:0/2149992325 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5bc134d40 con 0x7fb5bc139ec0 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 10 client.44201.objecter ms_handle_connect 0x7fb5bc139ec0 2024-09-11T02:56:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 10 client.44201.objecter resend_mon_ops 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c13b5640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ec0 0x7fb5bc02d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800f4b0 tx=0x7fb5b80046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5b8019040 con 0x7fb5bc139ec0 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 10 monclient: dump: 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7fb5b8015490 con 0x7fb5bc139ec0 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ec0 msgr2=0x7fb5bc02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ec0 0x7fb5bc02d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800f4b0 tx=0x7fb5b80046d0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ec0 msgr2=0x7fb5bc02d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5b1ffb640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5bc139ec0 0x7fb5bc02d990 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800f4b0 tx=0x7fb5b80046d0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 20003, 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:33.599 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:33.600 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:33.601 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.602 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:33.603 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.604 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.609 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:33.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.611 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:33.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:33.621 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:33.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 10 librados: watch_flush enter 2024-09-11T02:56:33.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 10 librados: watch_flush exit 2024-09-11T02:56:33.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 20 client.44201.objecter shutdown clearing up homeless session... 2024-09-11T02:56:33.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 10 client.44201.objecter successfully canceled tick 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb5a003df40 msgr2=0x7fb5a0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb5a003df40 0x7fb5a0040400 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc13aa80 tx=0x7fb5ac008040 comp rx=0 tx=0).stop 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 10 monclient: shutdown 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5bc0a2bd0 msgr2=0x7fb5bc1cfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 --2- 172.21.15.98:0/2149992325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5bc0a2bd0 0x7fb5bc1cfe50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc138650 tx=0x7fb5b4019650 comp rx=0 tx=0).stop 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/2149992325 reap_dead start 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c1bb6640 1 -- 172.21.15.98:0/2149992325 reap_dead start 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 shutdown_connections 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 >> 172.21.15.98:0/2149992325 conn(0x7fb5bc06eed0 msgr2=0x7fb5bc09df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 shutdown_connections 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 -- 172.21.15.98:0/2149992325 wait complete. 2024-09-11T02:56:33.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.576+0000 7fb5c3619640 1 librados: shutdown 2024-09-11T02:56:33.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:33.625 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:56:33.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:56:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.744+0000 7f86309d1640 10 monclient: get_monmap_and_config 2024-09-11T02:56:33.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.744+0000 7f86309d1640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient: monmap: 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:33.749248+0000 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:33.749248+0000 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.748 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:33.749 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:33.749 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 1 Processor -- start 2024-09-11T02:56:33.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 1 -- start start 2024-09-11T02:56:33.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient: init 2024-09-11T02:56:33.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient: _add_conns ranks=[5,1,3,9,8,7,4,6,2,0,10] 2024-09-11T02:56:33.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f862c139b10 0x7f862c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient(hunting): picked mon.noname-h con 0x7f862c139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f862c139ef0 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c13a2d0 0x7f862c0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient(hunting): picked mon.noname-g con 0x7f862c13a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f862c0702b0 con 0x7f862c139ef0 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f862c071bf0 con 0x7f862c13a2d0 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f862c071350 con 0x7f862c139b10 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f86309d1640 10 monclient(hunting): authenticate will time out at 1347.629452s 2024-09-11T02:56:33.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c13a2d0 0x7f862c0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f862c139b10 0x7f862c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c13a2d0 0x7f862c0a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:42532/0 (socket says 172.21.15.98:42532) 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 1 -- 172.21.15.98:0/288746997 learned_addr learned my addr 172.21.15.98:0/288746997 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f8629d9b640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f862c139ef0 0x7f862c0a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:53772/0 (socket says 172.21.15.98:53772) 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 10 monclient(hunting): get_auth_request con 0x7f862c13a2d0 auth_method 0 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): get_auth_request con 0x7f862c139b10 auth_method 0 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient(hunting): handle_auth_done global_id 19931 payload 274 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 1 -- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c13a2d0 msgr2=0x7f862c0a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 1 --2- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c13a2d0 0x7f862c0a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 1 -- 172.21.15.98:0/288746997 >> v1:172.21.15.190:6789/0 conn(0x7f862c139ef0 legacy=0x7f862c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 1 monclient: found mon.noname-h 2024-09-11T02:56:33.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:56:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862959a640 1 -- 172.21.15.98:0/288746997 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1207853972 0 0) 0x7f8614001120 con 0x7f862c139ef0 2024-09-11T02:56:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862a59c640 1 -- 172.21.15.98:0/288746997 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c0a9620 con 0x7f862c139b10 2024-09-11T02:56:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:56:33.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.748+0000 7f862ad9d640 1 --2- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c13a2d0 0x7f862c0a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862a59c640 1 --2- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f862c139b10 0x7f862c072600 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8624009df0 tx=0x7f862401c8e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=f1018408f1048ef3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 1 -- 172.21.15.98:0/288746997 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86240b2070 con 0x7f862c139b10 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: dump: 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 1 -- 172.21.15.98:0/288746997 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86240a59d0 con 0x7f862c139b10 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 1 -- 172.21.15.98:0/288746997 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86240a5b80 con 0x7f862c139b10 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862959a640 10 monclient: dump: 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.764 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 5 monclient: authenticate success, global_id 19931 2024-09-11T02:56:33.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 4 set_mon_vals no callback set 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient: shutdown 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f862c139b10 msgr2=0x7f862c072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 --2- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f862c139b10 0x7f862c072600 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8624009df0 tx=0x7f862401c8e0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- 172.21.15.98:0/288746997 shutdown_connections 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 --2- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c13a2d0 0x7f862c0a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 --2- 172.21.15.98:0/288746997 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f862c139b10 0x7f862c072600 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- 172.21.15.98:0/288746997 >> 172.21.15.98:0/288746997 conn(0x7f862c0ac790 msgr2=0x7f862c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- 172.21.15.98:0/288746997 shutdown_connections 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- 172.21.15.98:0/288746997 wait complete. 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient: monmap: 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.767 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 librados: starting msgr at 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 librados: starting objecter 2024-09-11T02:56:33.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 Processor -- start 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- start start 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 librados: setting wanted keys 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 librados: calling monclient init 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient: init 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient: _add_conns ranks=[3,7,0,6,2,4,5,8,1] 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c139b10 0x7f862c1d1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): picked mon.noname-g con 0x7f862c139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f862c139ef0 0x7f862c1d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): picked mon.noname-i con 0x7f862c139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862c13a2d0 0x7f862c1d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): picked mon.noname-a con 0x7f862c13a2d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f862c144470 con 0x7f862c13a2d0 2024-09-11T02:56:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f862c0a8870 con 0x7f862c139b10 2024-09-11T02:56:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f862c1445f0 con 0x7f862c139ef0 2024-09-11T02:56:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f86309d1640 10 monclient(hunting): authenticate will time out at 1347.635216s 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862c13a2d0 0x7f862c1d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862c13a2d0 0x7f862c1d1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57600/0 (socket says 172.21.15.98:57600) 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 -- 172.21.15.98:0/4129625409 learned_addr learned my addr 172.21.15.98:0/4129625409 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): get_auth_request con 0x7f862c13a2d0 auth_method 0 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862a59c640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c139b10 0x7f862c1d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862a59c640 10 monclient(hunting): get_auth_request con 0x7f862c139b10 auth_method 0 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f8629d9b640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f862c139ef0 0x7f862c1d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f8629d9b640 10 monclient(hunting): get_auth_request con 0x7f862c139ef0 auth_method 0 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f8629d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f8629d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f8629d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient(hunting): handle_auth_done global_id 44262 payload 824 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f862c139ef0 msgr2=0x7f862c1d1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f862c139ef0 0x7f862c1d1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c139b10 msgr2=0x7f862c1d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c139b10 0x7f862c1d1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 monclient: found mon.noname-a 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 -- 172.21.15.98:0/4129625409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c0a5b40 con 0x7f862c13a2d0 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862a59c640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f862c139b10 0x7f862c1d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f862ad9d640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862c13a2d0 0x7f862c1d1f80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f861c00e930 tx=0x7f861c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f8622ffd640 10 client.?.objecter ms_handle_connect 0x7f862c13a2d0 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.752+0000 7f8622ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f861c017070 con 0x7f862c13a2d0 2024-09-11T02:56:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: dump: 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861c00cdb0 con 0x7f862c13a2d0 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 5 monclient: authenticate success, global_id 44262 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 10 monclient: _renew_subs 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862c0a9f00 con 0x7f862c13a2d0 2024-09-11T02:56:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f861c0138e0 con 0x7f862c13a2d0 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8628d99640 4 set_mon_vals no callback set 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 10 client.44262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: dump: 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.777 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 10 monclient: _renew_subs 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f862c134890 con 0x7f862c13a2d0 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 1 librados: init done 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13a740 0x7f862c13ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f862c06ca10 con 0x7f862c13a740 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f861c013470 con 0x7f862c13a2d0 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13a740 0x7f862c13ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient: get_auth_request con 0x7f862c13a740 auth_method 0 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f860c03df40 0x7f860c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(297..297 src has 1..297) ==== 7075+0+0 (secure 0 0 0) 0x7f861c00f040 con 0x7f862c13a2d0 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 client.44262.objecter ms_dispatch 0x7f862c000ba0 osd_map(297..297 src has 1..297) 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 3 client.44262.objecter handle_osd_map got epochs [297,297] > 0 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 3 client.44262.objecter handle_osd_map decoding full epoch 297 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 10 monclient(hunting): handle_auth_done global_id 24272 payload 824 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 20 client.44262.objecter dump_active .. 0 homeless 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 1 -- 172.21.15.98:0/4129625409 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f862c06e680 con 0x7f862c13a740 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8629d9b640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f860c03df40 0x7f860c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8629d9b640 10 monclient: get_auth_request con 0x7f860c03df40 auth_method 0 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 client.44262.objecter ms_handle_connect 0x7f862c13a740 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f862a59c640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13a740 0x7f862c13ab20 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f862c0a9590 tx=0x7f8624019680 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 client.44262.objecter resend_mon_ops 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86240bd030 con 0x7f862c13a740 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: dump: 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f86240b3040 con 0x7f862c13a740 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13a740 msgr2=0x7f862c13ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13a740 0x7f862c13ab20 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f862c0a9590 tx=0x7f8624019680 comp rx=0 tx=0).stop 2024-09-11T02:56:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13a740 msgr2=0x7f862c13ab20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13a740 0x7f862c13ab20 secure :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f862c0a9590 tx=0x7f8624019680 comp rx=0 tx=0).stop 2024-09-11T02:56:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8629d9b640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f860c03df40 0x7f860c040400 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f862c141830 tx=0x7f8614007a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.756+0000 7f8622ffd640 10 client.44262.objecter ms_handle_connect 0x7f860c03df40 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13b060 0x7f862c0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f862c06e870 con 0x7f862c13b060 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13b060 0x7f862c0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient: get_auth_request con 0x7f862c13b060 auth_method 0 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 10 monclient(hunting): handle_auth_done global_id 24281 payload 824 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 1 -- 172.21.15.98:0/4129625409 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f862c072600 con 0x7f862c13b060 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 10 client.44262.objecter ms_handle_connect 0x7f862c13b060 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862a59c640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13b060 0x7f862c0a0e60 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f862401af90 tx=0x7f86240a7000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 10 client.44262.objecter resend_mon_ops 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86240bd070 con 0x7f862c13b060 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 10 monclient: dump: 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f86240c3910 con 0x7f862c13b060 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13b060 msgr2=0x7f862c0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13b060 0x7f862c0a0e60 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f862401af90 tx=0x7f86240a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13b060 msgr2=0x7f862c0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f8622ffd640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f862c13b060 0x7f862c0a0e60 secure :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f862401af90 tx=0x7f86240a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:33.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 409556, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:33.788 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 7, 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.789 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.790 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.792 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:33.810 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 10 librados: watch_flush enter 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 10 librados: watch_flush exit 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 20 client.44262.objecter shutdown clearing up homeless session... 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 10 client.44262.objecter successfully canceled tick 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f860c03df40 msgr2=0x7f860c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f860c03df40 0x7f860c040400 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f862c141830 tx=0x7f8614007a30 comp rx=0 tx=0).stop 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 10 monclient: shutdown 2024-09-11T02:56:33.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862c13a2d0 msgr2=0x7f862c1d1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f862ad9d640 1 -- 172.21.15.98:0/4129625409 reap_dead start 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862c13a2d0 0x7f862c1d1f80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f861c00e930 tx=0x7f861c00edf0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 shutdown_connections 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 --2- 172.21.15.98:0/4129625409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f862c13a2d0 0x7f862c1d1f80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 >> 172.21.15.98:0/4129625409 conn(0x7f862c0ac790 msgr2=0x7f862c0a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 shutdown_connections 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 -- 172.21.15.98:0/4129625409 wait complete. 2024-09-11T02:56:33.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.764+0000 7f86309d1640 1 librados: shutdown 2024-09-11T02:56:33.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:33.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:56:33.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient: get_monmap_and_config 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient: monmap: 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:33.937679+0000 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:33.937679+0000 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.936 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:33.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 Processor -- start 2024-09-11T02:56:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 -- start start 2024-09-11T02:56:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient: init 2024-09-11T02:56:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient: _add_conns ranks=[8,5,10,6,2,1,4,3,0,7,9] 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e340a46a0 0x7f3e340a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): picked mon.noname-e con 0x7f3e340a46a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e34139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): picked mon.noname-h con 0x7f3e340a4fc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3e34139e70 0x7f3e3413a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): picked mon.noname-f con 0x7f3e34139e70 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3e340a0750 con 0x7f3e34139e70 2024-09-11T02:56:33.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3e340a17b0 con 0x7f3e340a4fc0 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3e340775d0 con 0x7f3e340a46a0 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3c4ef640 10 monclient(hunting): authenticate will time out at 1347.817986s 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e340a46a0 0x7f3e340a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e340a46a0 0x7f3e340a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44724/0 (socket says 172.21.15.98:44724) 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 -- 172.21.15.98:0/509430256 learned_addr learned my addr 172.21.15.98:0/509430256 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): get_auth_request con 0x7f3e340a46a0 auth_method 0 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3aa8c640 1 --2- 172.21.15.98:0/509430256 >> v2:172.21.15.190:3300/0 conn(0x7f3e34139e70 0x7f3e3413a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e39a8a640 1 --2- 172.21.15.98:0/509430256 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e34139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_done global_id 24290 payload 274 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 -- 172.21.15.98:0/509430256 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 msgr2=0x7f3e34139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 --2- 172.21.15.98:0/509430256 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e34139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 -- 172.21.15.98:0/509430256 >> v2:172.21.15.190:3300/0 conn(0x7f3e34139e70 msgr2=0x7f3e3413a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 --2- 172.21.15.98:0/509430256 >> v2:172.21.15.190:3300/0 conn(0x7f3e34139e70 0x7f3e3413a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 monclient: found mon.noname-e 2024-09-11T02:56:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3a28b640 1 -- 172.21.15.98:0/509430256 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e340a0200 con 0x7f3e340a46a0 2024-09-11T02:56:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3aa8c640 10 monclient: get_auth_request con 0x7f3e34139e70 auth_method 0 2024-09-11T02:56:33.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.936+0000 7f3e3aa8c640 1 --2- 172.21.15.98:0/509430256 >> v2:172.21.15.190:3300/0 conn(0x7f3e34139e70 0x7f3e3413a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 1 --2- 172.21.15.98:0/509430256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e340a46a0 0x7f3e340a4a80 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009870 tx=0x7f3e2c01c420 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a2c431ccf234bec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 1 -- 172.21.15.98:0/509430256 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e2c0b0070 con 0x7f3e340a46a0 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: dump: 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 5 monclient: authenticate success, global_id 24290 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 1 -- 172.21.15.98:0/509430256 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e2c0aa810 con 0x7f3e340a46a0 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 4 set_mon_vals no callback set 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 1 -- 172.21.15.98:0/509430256 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e2c0a9300 con 0x7f3e340a46a0 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39289640 10 monclient: dump: 2024-09-11T02:56:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient: shutdown 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/509430256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e340a46a0 msgr2=0x7f3e340a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/509430256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e340a46a0 0x7f3e340a4a80 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c009870 tx=0x7f3e2c01c420 comp rx=0 tx=0).stop 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/509430256 shutdown_connections 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/509430256 >> v2:172.21.15.190:3300/0 conn(0x7f3e34139e70 0x7f3e3413a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/509430256 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e34139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/509430256 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3e340a46a0 0x7f3e340a4a80 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/509430256 >> 172.21.15.98:0/509430256 conn(0x7f3e3406ed00 msgr2=0x7f3e3406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/509430256 shutdown_connections 2024-09-11T02:56:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/509430256 wait complete. 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient: build_initial_monmap 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient: monmap: 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 librados: starting msgr at 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 librados: starting objecter 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 Processor -- start 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- start start 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 librados: setting wanted keys 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 librados: calling monclient init 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient: init 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient: _add_conns ranks=[0,5,2,3,6,1,8,4,7] 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e340a46a0 0x7f3e3413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): picked mon.noname-a con 0x7f3e340a46a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e341d19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): picked mon.noname-h con 0x7f3e340a4fc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e34139e70 0x7f3e3413c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): picked mon.noname-b con 0x7f3e34139e70 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3e340779d0 con 0x7f3e340a46a0 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3e34077850 con 0x7f3e34139e70 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3e34143410 con 0x7f3e340a4fc0 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3c4ef640 10 monclient(hunting): authenticate will time out at 1347.823476s 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e340a46a0 0x7f3e3413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e34139e70 0x7f3e3413c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e340a46a0 0x7f3e3413ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57604/0 (socket says 172.21.15.98:57604) 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 1 -- 172.21.15.98:0/2597526676 learned_addr learned my addr 172.21.15.98:0/2597526676 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): get_auth_request con 0x7f3e340a46a0 auth_method 0 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 10 monclient(hunting): get_auth_request con 0x7f3e34139e70 auth_method 0 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3aa8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39a8a640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e341d19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39a8a640 10 monclient(hunting): get_auth_request con 0x7f3e340a4fc0 auth_method 0 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39a8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39a8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e39a8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 10 monclient(hunting): handle_auth_done global_id 44271 payload 824 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.940+0000 7f3e3a28b640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 msgr2=0x7f3e341d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e341d19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e34139e70 msgr2=0x7f3e3413c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e34139e70 0x7f3e3413c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 10 monclient: _finish_hunting 0 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 1 monclient: found mon.noname-a 2024-09-11T02:56:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 1 -- 172.21.15.98:0/2597526676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e341236f0 con 0x7f3e340a46a0 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3aa8c640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e34139e70 0x7f3e3413c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e340a4fc0 0x7f3e341d19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3a28b640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e340a46a0 0x7f3e3413ec70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0ae040 tx=0x7f3e2c0191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 client.?.objecter ms_handle_connect 0x7f3e340a46a0 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e2c0b9070 con 0x7f3e340a46a0 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: dump: 2024-09-11T02:56:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: _finish_auth 0 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e2c0044b0 con 0x7f3e340a46a0 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e2c0b0040 con 0x7f3e340a46a0 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 5 monclient: authenticate success, global_id 44271 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 10 monclient: _renew_subs 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e34123f50 con 0x7f3e340a46a0 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e38a88640 4 set_mon_vals no callback set 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: dump: 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.962 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 10 client.44271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 10 monclient: _renew_subs 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e340a87e0 con 0x7f3e340a46a0 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 1 librados: init done 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3e2c004040 con 0x7f3e340a46a0 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e341d1ee0 0x7f3e3402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e341c82a0 con 0x7f3e341d1ee0 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e2003df40 0x7f3e20040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(297..297 src has 1..297) ==== 7075+0+0 (secure 0 0 0) 0x7f3e2c0e8dd0 con 0x7f3e340a46a0 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 client.44271.objecter ms_dispatch 0x7f3e34000ba0 osd_map(297..297 src has 1..297) 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 3 client.44271.objecter handle_osd_map got epochs [297,297] > 0 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 3 client.44271.objecter handle_osd_map decoding full epoch 297 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e341d1ee0 0x7f3e3402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 20 client.44271.objecter dump_active .. 0 homeless 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient: get_auth_request con 0x7f3e341d1ee0 auth_method 0 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3aa8c640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e2003df40 0x7f3e20040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3aa8c640 10 monclient: get_auth_request con 0x7f3e2003df40 auth_method 0 2024-09-11T02:56:33.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_done global_id 25183 payload 824 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 1 -- 172.21.15.98:0/2597526676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e34139930 con 0x7f3e341d1ee0 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e3aa8c640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e2003df40 0x7f3e20040400 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f3e34140420 tx=0x7f3e24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 client.44271.objecter ms_handle_connect 0x7f3e2003df40 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 client.44271.objecter ms_handle_connect 0x7f3e341d1ee0 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 client.44271.objecter resend_mon_ops 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e39a8a640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e341d1ee0 0x7f3e3402d450 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f3e3413e780 tx=0x7f3e3000c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e3000fa80 con 0x7f3e341d1ee0 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: dump: 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e3001a480 con 0x7f3e341d1ee0 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e341d1ee0 msgr2=0x7f3e3402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e341d1ee0 0x7f3e3402d450 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f3e3413e780 tx=0x7f3e3000c6e0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e341d1ee0 msgr2=0x7f3e3402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.944+0000 7f3e2affd640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e341d1ee0 0x7f3e3402d450 secure :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f3e3413e780 tx=0x7f3e3000c6e0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e3c4ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e3402d990 0x7f3e3402dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e3c4ef640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3406ca10 con 0x7f3e3402d990 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e3402d990 0x7f3e3402dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient: get_auth_request con 0x7f3e3402d990 auth_method 0 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 10 monclient(hunting): handle_auth_done global_id 25192 payload 824 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 1 -- 172.21.15.98:0/2597526676 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e3413f600 con 0x7f3e3402d990 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e2affd640 10 client.44271.objecter ms_handle_connect 0x7f3e3402d990 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e39a8a640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e3402d990 0x7f3e3402dda0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f3e30019000 tx=0x7f3e30004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e2affd640 10 client.44271.objecter resend_mon_ops 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e3001ae10 con 0x7f3e3402d990 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e2affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.952+0000 7f3e2affd640 10 monclient: dump: 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:33.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7f3e3001ae10 con 0x7f3e3402d990 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e2affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e2affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e3402d990 msgr2=0x7f3e3402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e2affd640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e3402d990 0x7f3e3402dda0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7f3e30019000 tx=0x7f3e30004a50 comp rx=0 tx=0).stop 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e2affd640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e3402d990 msgr2=0x7f3e3402dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e2affd640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e3402d990 0x7f3e3402dda0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:33.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 409721, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.974 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.975 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:33.976 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:33.977 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.979 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:33.980 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.981 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.983 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.984 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:33.989 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.991 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:33.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:33.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.994 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:33.995 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 10 librados: watch_flush enter 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 10 librados: watch_flush exit 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 20 client.44271.objecter shutdown clearing up homeless session... 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 10 client.44271.objecter successfully canceled tick 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e2003df40 msgr2=0x7f3e20040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e2003df40 0x7f3e20040400 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f3e34140420 tx=0x7f3e24008040 comp rx=0 tx=0).stop 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 10 monclient: shutdown 2024-09-11T02:56:33.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e340a46a0 msgr2=0x7f3e3413ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 --2- 172.21.15.98:0/2597526676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3e340a46a0 0x7f3e3413ec70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0ae040 tx=0x7f3e2c0191c0 comp rx=0 tx=0).stop 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3aa8c640 1 -- 172.21.15.98:0/2597526676 reap_dead start 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3aa8c640 1 -- 172.21.15.98:0/2597526676 reap_dead start 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 shutdown_connections 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 >> 172.21.15.98:0/2597526676 conn(0x7f3e3406ed00 msgr2=0x7f3e341404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 shutdown_connections 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 -- 172.21.15.98:0/2597526676 wait complete. 2024-09-11T02:56:33.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:33.956+0000 7f3e3c4ef640 1 librados: shutdown 2024-09-11T02:56:33.998 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:33.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:56:33.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:56:34.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient: get_monmap_and_config 2024-09-11T02:56:34.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient: monmap: 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:34.114164+0000 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:34.114164+0000 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.113 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:34.114 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:34.114 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 1 Processor -- start 2024-09-11T02:56:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 1 -- start start 2024-09-11T02:56:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient: init 2024-09-11T02:56:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient: _add_conns ranks=[6,1,8,2,5,10,4,3,0,9,7] 2024-09-11T02:56:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee0139b80 0x7f8ee0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient(hunting): picked mon.noname-d con 0x7f8ee0139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8ee006fe50 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee0072640 0x7f8ee00a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient(hunting): picked mon.noname-e con 0x7f8ee0072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ee0071370 con 0x7f8ee006fe50 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8ee00719c0 con 0x7f8ee0139b80 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8ee0071c10 con 0x7f8ee0072640 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8ee5e9b640 10 monclient(hunting): authenticate will time out at 1347.994413s 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee0072640 0x7f8ee00a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee0139b80 0x7f8ee0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee0139b80 0x7f8ee0139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:52090/0 (socket says 172.21.15.98:52090) 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3619856148 learned_addr learned my addr 172.21.15.98:0/3619856148 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee0072640 0x7f8ee00a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44736/0 (socket says 172.21.15.98:44736) 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): get_auth_request con 0x7f8ee0139b80 auth_method 0 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 10 monclient(hunting): get_auth_request con 0x7f8ee0072640 auth_method 0 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_done global_id 44241 payload 274 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee0072640 msgr2=0x7f8ee00a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 --2- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee0072640 0x7f8ee00a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3619856148 >> v1:172.21.15.190:6789/0 conn(0x7f8ee006fe50 legacy=0x7f8ee00a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 monclient: found mon.noname-d 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:56:34.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3619856148 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee00a9600 con 0x7f8ee0139b80 2024-09-11T02:56:34.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.112+0000 7f8edffff640 1 --2- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee0072640 0x7f8ee00a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:34.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 1 --2- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee0139b80 0x7f8ee0139f60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00098a0 tx=0x7f8ed001c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=5ca1793267b345b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 1 -- 172.21.15.98:0/3619856148 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed00b1070 con 0x7f8ee0139b80 2024-09-11T02:56:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:56:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: dump: 2024-09-11T02:56:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.126 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.127 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 5 monclient: authenticate success, global_id 44241 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 1 -- 172.21.15.98:0/3619856148 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed00aa7b0 con 0x7f8ee0139b80 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 4 set_mon_vals no callback set 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 1 -- 172.21.15.98:0/3619856148 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed00aae50 con 0x7f8ee0139b80 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edeffd640 10 monclient: dump: 2024-09-11T02:56:34.128 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.129 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient: shutdown 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee0139b80 msgr2=0x7f8ee0139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 --2- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee0139b80 0x7f8ee0139f60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00098a0 tx=0x7f8ed001c440 comp rx=0 tx=0).stop 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3619856148 shutdown_connections 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 --2- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee0072640 0x7f8ee00a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 --2- 172.21.15.98:0/3619856148 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee0139b80 0x7f8ee0139f60 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3619856148 >> 172.21.15.98:0/3619856148 conn(0x7f8ee00ac730 msgr2=0x7f8ee00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3619856148 shutdown_connections 2024-09-11T02:56:34.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3619856148 wait complete. 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient: monmap: 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.131 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 librados: starting msgr at 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 librados: starting objecter 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 Processor -- start 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- start start 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 librados: setting wanted keys 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 librados: calling monclient init 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient: init 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient: _add_conns ranks=[8,0,1,3,5,6,4,2,7] 2024-09-11T02:56:34.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee006fe50 0x7f8ee0133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): picked mon.noname-e con 0x7f8ee006fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee0072640 0x7f8ee0133ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): picked mon.noname-a con 0x7f8ee0072640 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ee0139b80 0x7f8ee013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): picked mon.noname-f con 0x7f8ee0139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee0144450 con 0x7f8ee0072640 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee00a8870 con 0x7f8ee0139b80 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8ee01445d0 con 0x7f8ee006fe50 2024-09-11T02:56:34.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ee5e9b640 10 monclient(hunting): authenticate will time out at 1347.999632s 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee006fe50 0x7f8ee0133ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee006fe50 0x7f8ee0133ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44748/0 (socket says 172.21.15.98:44748) 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3597318698 learned_addr learned my addr 172.21.15.98:0/3597318698 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): get_auth_request con 0x7f8ee006fe50 auth_method 0 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee0072640 0x7f8ee0133ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ee0072640 auth_method 0 2024-09-11T02:56:34.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edffff640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ee0139b80 0x7f8ee013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.116+0000 7f8edf7fe640 10 monclient(hunting): handle_auth_done global_id 24308 payload 824 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ee0139b80 msgr2=0x7f8ee013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ee0139b80 0x7f8ee013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee0072640 msgr2=0x7f8ee0133ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee0072640 0x7f8ee0133ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 1 monclient: found mon.noname-e 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 1 -- 172.21.15.98:0/3597318698 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee00a8300 con 0x7f8ee006fe50 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edffff640 10 monclient: get_auth_request con 0x7f8ee0139b80 auth_method 0 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edffff640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ee0139b80 0x7f8ee013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edf7fe640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee006fe50 0x7f8ee0133ab0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00af040 tx=0x7f8ed00191e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 client.?.objecter ms_handle_connect 0x7f8ee006fe50 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed00b1040 con 0x7f8ee006fe50 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: dump: 2024-09-11T02:56:34.136 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 5 monclient: authenticate success, global_id 24308 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 10 monclient: _renew_subs 2024-09-11T02:56:34.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed00aadd0 con 0x7f8ee006fe50 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee013d040 con 0x7f8ee006fe50 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed0004320 con 0x7f8ee006fe50 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ede7fc640 4 set_mon_vals no callback set 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 10 client.24308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: dump: 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.139 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 10 monclient: _renew_subs 2024-09-11T02:56:34.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee00aa190 con 0x7f8ee006fe50 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 1 librados: init done 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002d450 0x7f8ee002d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee013bb20 con 0x7f8ee002d450 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8ed00b4070 con 0x7f8ee006fe50 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ec003df40 0x7f8ec0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002d450 0x7f8ee002d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient: get_auth_request con 0x7f8ee002d450 auth_method 0 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(297..297 src has 1..297) ==== 7075+0+0 (secure 0 0 0) 0x7f8ed00e9800 con 0x7f8ee006fe50 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 client.24308.objecter ms_dispatch 0x7f8ee0000ba0 osd_map(297..297 src has 1..297) 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 3 client.24308.objecter handle_osd_map got epochs [297,297] > 0 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 3 client.24308.objecter handle_osd_map decoding full epoch 297 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edffff640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ec003df40 0x7f8ec0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 20 client.24308.objecter dump_active .. 0 homeless 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edffff640 10 monclient: get_auth_request con 0x7f8ec003df40 auth_method 0 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_done global_id 44175 payload 824 2024-09-11T02:56:34.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 1 -- 172.21.15.98:0/3597318698 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ee0141200 con 0x7f8ee002d450 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 client.24308.objecter ms_handle_connect 0x7f8ee002d450 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 client.24308.objecter resend_mon_ops 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8ed6ffd640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002d450 0x7f8ee002d830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00a4db0 tx=0x7f8ec800fc00 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 client.24308.objecter ms_handle_connect 0x7f8ec003df40 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edffff640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ec003df40 0x7f8ec0040400 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8007520 tx=0x7f8ed800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ec800d6d0 con 0x7f8ee002d450 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: dump: 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ec80050d0 con 0x7f8ee002d450 2024-09-11T02:56:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002d450 msgr2=0x7f8ee002d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002d450 0x7f8ee002d830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00a4db0 tx=0x7f8ec800fc00 comp rx=0 tx=0).stop 2024-09-11T02:56:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002d450 msgr2=0x7f8ee002d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.120+0000 7f8edcff9640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002d450 0x7f8ee002d830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ee5e9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ee5e9b640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002dd70 0x7f8ee002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ee5e9b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee006ca10 con 0x7f8ee002dd70 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002dd70 0x7f8ee002e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient: get_auth_request con 0x7f8ee002dd70 auth_method 0 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 10 monclient(hunting): handle_auth_done global_id 44184 payload 824 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 1 -- 172.21.15.98:0/3597318698 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ee0138b80 con 0x7f8ee002dd70 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ed6ffd640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002dd70 0x7f8ee002e180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00a4db0 tx=0x7f8ec8000ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 10 client.24308.objecter ms_handle_connect 0x7f8ee002dd70 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 10 client.24308.objecter resend_mon_ops 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ec8003600 con 0x7f8ee002dd70 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 10 monclient: dump: 2024-09-11T02:56:34.148 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.149 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f8ec800da20 con 0x7f8ee002dd70 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002dd70 msgr2=0x7f8ee002e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002dd70 0x7f8ee002e180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00a4db0 tx=0x7f8ec8000ee0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002dd70 msgr2=0x7f8ee002e180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8edcff9640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8ee002dd70 0x7f8ee002e180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:34.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 20555, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:34.151 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.152 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.153 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:34.154 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.155 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:34.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:34.159 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.162 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.163 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.164 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.166 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.171 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:34.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:34.173 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.128+0000 7f8ee5e9b640 10 librados: watch_flush enter 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 10 librados: watch_flush exit 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 20 client.24308.objecter shutdown clearing up homeless session... 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 10 client.24308.objecter successfully canceled tick 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ec003df40 msgr2=0x7f8ec0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ec003df40 0x7f8ec0040400 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8007520 tx=0x7f8ed800b040 comp rx=0 tx=0).stop 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 10 monclient: shutdown 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee006fe50 msgr2=0x7f8ee0133ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 --2- 172.21.15.98:0/3597318698 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ee006fe50 0x7f8ee0133ab0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00af040 tx=0x7f8ed00191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8edffff640 1 -- 172.21.15.98:0/3597318698 reap_dead start 2024-09-11T02:56:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8edffff640 1 -- 172.21.15.98:0/3597318698 reap_dead start 2024-09-11T02:56:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 shutdown_connections 2024-09-11T02:56:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 >> 172.21.15.98:0/3597318698 conn(0x7f8ee00ac730 msgr2=0x7f8ee0134aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 shutdown_connections 2024-09-11T02:56:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 -- 172.21.15.98:0/3597318698 wait complete. 2024-09-11T02:56:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.132+0000 7f8ee5e9b640 1 librados: shutdown 2024-09-11T02:56:34.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:34.176 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:56:34.177 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:56:34.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient: get_monmap_and_config 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient: monmap: 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:34.298062+0000 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:34.298062+0000 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.297 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:34.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 1 Processor -- start 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 1 -- start start 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient: init 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient: _add_conns ranks=[3,4,0,5,9,1,10,2,8,6,7] 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f057c139b80 0x7f057c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient(hunting): picked mon.noname-g con 0x7f057c139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c06fce0 0x7f057c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient(hunting): picked mon.noname-c con 0x7f057c06fce0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f057c0724b0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057c0a94a0 con 0x7f057c0724b0 2024-09-11T02:56:34.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f057c071850 con 0x7f057c139b80 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f057c071aa0 con 0x7f057c06fce0 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f058405d640 10 monclient(hunting): authenticate will time out at 1348.178563s 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05825fa640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f057c0724b0 0x7f057c0a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54394/0 (socket says 172.21.15.98:54394) 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05825fa640 1 -- 172.21.15.98:0/1332185441 learned_addr learned my addr 172.21.15.98:0/1332185441 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 1 --2- 172.21.15.98:0/1332185441 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c06fce0 0x7f057c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): get_auth_request con 0x7f057c06fce0 auth_method 0 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 1 --2- 172.21.15.98:0/1332185441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f057c139b80 0x7f057c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 10 monclient(hunting): get_auth_request con 0x7f057c139b80 auth_method 0 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient(hunting): handle_auth_done global_id 23827 payload 274 2024-09-11T02:56:34.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 1 -- 172.21.15.98:0/1332185441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f057c139b80 msgr2=0x7f057c139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 1 --2- 172.21.15.98:0/1332185441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f057c139b80 0x7f057c139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 1 -- 172.21.15.98:0/1332185441 >> v1:172.21.15.98:6789/0 conn(0x7f057c0724b0 legacy=0x7f057c0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0580df7640 1 -- 172.21.15.98:0/1332185441 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732215874 0 0) 0x7f057c0a94a0 con 0x7f057c0724b0 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 1 monclient: found mon.noname-c 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f05815f8640 1 -- 172.21.15.98:0/1332185441 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057c13a4a0 con 0x7f057c06fce0 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.296+0000 7f0580df7640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f05815f8640 1 --2- 172.21.15.98:0/1332185441 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c06fce0 0x7f057c0a7a10 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f056c009820 tx=0x7f056c01c380 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e6601245c125279 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 1 -- 172.21.15.98:0/1332185441 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f056c0b0070 con 0x7f057c06fce0 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:34.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: dump: 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.312 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 5 monclient: authenticate success, global_id 23827 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 1 -- 172.21.15.98:0/1332185441 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f056c0aa810 con 0x7f057c06fce0 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 4 set_mon_vals no callback set 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 1 -- 172.21.15.98:0/1332185441 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f056c0a9300 con 0x7f057c06fce0 2024-09-11T02:56:34.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f0580df7640 10 monclient: dump: 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.314 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient: shutdown 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- 172.21.15.98:0/1332185441 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c06fce0 msgr2=0x7f057c0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 --2- 172.21.15.98:0/1332185441 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c06fce0 0x7f057c0a7a10 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f056c009820 tx=0x7f056c01c380 comp rx=0 tx=0).stop 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- 172.21.15.98:0/1332185441 shutdown_connections 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 --2- 172.21.15.98:0/1332185441 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c06fce0 0x7f057c0a7a10 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 --2- 172.21.15.98:0/1332185441 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f057c139b80 0x7f057c139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- 172.21.15.98:0/1332185441 >> 172.21.15.98:0/1332185441 conn(0x7f057c0ac6b0 msgr2=0x7f057c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- 172.21.15.98:0/1332185441 shutdown_connections 2024-09-11T02:56:34.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- 172.21.15.98:0/1332185441 wait complete. 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient: monmap: 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.316 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 librados: starting msgr at 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 librados: starting objecter 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 Processor -- start 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- start start 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 librados: setting wanted keys 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 librados: calling monclient init 2024-09-11T02:56:34.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient: init 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient: _add_conns ranks=[1,4,5,0,7,3,2,6,8] 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057c06fce0 0x7f057c13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): picked mon.noname-f con 0x7f057c06fce0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c0724b0 0x7f057c13ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): picked mon.noname-c con 0x7f057c0724b0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c139b80 0x7f057c13b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): picked mon.noname-h con 0x7f057c139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f057c0a8870 con 0x7f057c06fce0 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f057c070160 con 0x7f057c0724b0 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f057c144450 con 0x7f057c139b80 2024-09-11T02:56:34.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f058405d640 10 monclient(hunting): authenticate will time out at 1348.183849s 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f05815f8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c0724b0 0x7f057c13ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.300+0000 7f05815f8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c0724b0 0x7f057c13ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38242/0 (socket says 172.21.15.98:38242) 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 -- 172.21.15.98:0/1748662025 learned_addr learned my addr 172.21.15.98:0/1748662025 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): get_auth_request con 0x7f057c0724b0 auth_method 0 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05825fa640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c139b80 0x7f057c13b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057c06fce0 0x7f057c13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient(hunting): handle_auth_done global_id 23836 payload 824 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c139b80 msgr2=0x7f057c13b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c139b80 0x7f057c13b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057c06fce0 msgr2=0x7f057c13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f057c06fce0 0x7f057c13a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 monclient: found mon.noname-c 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 -- 172.21.15.98:0/1748662025 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057c0a5580 con 0x7f057c0724b0 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05825fa640 10 monclient: get_auth_request con 0x7f057c139b80 auth_method 0 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05825fa640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c139b80 0x7f057c13b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:34.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05815f8640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c0724b0 0x7f057c13ad10 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f056c009710 tx=0x7f056c019120 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 client.?.objecter ms_handle_connect 0x7f057c0724b0 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f056c0b9070 con 0x7f057c0724b0 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: dump: 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.321 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f056c0044a0 con 0x7f057c0724b0 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f056c0b0040 con 0x7f057c0724b0 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0573fff640 4 set_mon_vals no callback set 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: dump: 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.323 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 5 monclient: authenticate success, global_id 23836 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 10 monclient: _renew_subs 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f057c0a7890 con 0x7f057c0724b0 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 10 client.23836.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 10 monclient: _renew_subs 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:56:34.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f057c13d090 con 0x7f057c0724b0 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 1 librados: init done 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c13c7f0 0x7f057c1337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f057c13b890 con 0x7f057c13c7f0 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f056c0b3070 con 0x7f057c0724b0 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c13c7f0 0x7f057c1337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient: get_auth_request con 0x7f057c13c7f0 auth_method 0 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056003df40 0x7f0560040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(297..297 src has 1..297) ==== 7075+0+0 (secure 0 0 0) 0x7f056c0e8a60 con 0x7f057c0724b0 2024-09-11T02:56:34.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 client.23836.objecter ms_dispatch 0x7f057c000ba0 osd_map(297..297 src has 1..297) 2024-09-11T02:56:34.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 3 client.23836.objecter handle_osd_map got epochs [297,297] > 0 2024-09-11T02:56:34.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 3 client.23836.objecter handle_osd_map decoding full epoch 297 2024-09-11T02:56:34.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 20 client.23836.objecter dump_active .. 0 homeless 2024-09-11T02:56:34.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 10 monclient(hunting): handle_auth_done global_id 19949 payload 824 2024-09-11T02:56:34.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 1 -- 172.21.15.98:0/1748662025 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f057c0a94a0 con 0x7f057c13c7f0 2024-09-11T02:56:34.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05825fa640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056003df40 0x7f0560040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05825fa640 10 monclient: get_auth_request con 0x7f056003df40 auth_method 0 2024-09-11T02:56:34.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 client.23836.objecter ms_handle_connect 0x7f057c13c7f0 2024-09-11T02:56:34.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 client.23836.objecter resend_mon_ops 2024-09-11T02:56:34.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f0581df9640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c13c7f0 0x7f057c1337f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f05780107c0 tx=0x7f0578010c80 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f057800f040 con 0x7f057c13c7f0 2024-09-11T02:56:34.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:34.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 monclient: dump: 2024-09-11T02:56:34.329 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.329 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.329 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.329 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.330 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.330 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.330 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.330 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.330 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.331 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.331 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.331 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.331 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.331 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.331 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.332 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05727fc640 10 client.23836.objecter ms_handle_connect 0x7f056003df40 2024-09-11T02:56:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.304+0000 7f05825fa640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056003df40 0x7f0560040400 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7f05740012f0 tx=0x7f0574009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.308+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f057801c400 con 0x7f057c13c7f0 2024-09-11T02:56:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.308+0000 7f05727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.308+0000 7f05727fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.308+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c13c7f0 msgr2=0x7f057c1337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.308+0000 7f05727fc640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c13c7f0 0x7f057c1337f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f05780107c0 tx=0x7f0578010c80 comp rx=0 tx=0).stop 2024-09-11T02:56:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.308+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c13c7f0 msgr2=0x7f057c1337f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.308+0000 7f05727fc640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c13c7f0 0x7f057c1337f0 secure :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f05780107c0 tx=0x7f0578010c80 comp rx=0 tx=0).stop 2024-09-11T02:56:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f058405d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f058405d640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c133d30 0x7f057c134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f058405d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f057c134680 con 0x7f057c133d30 2024-09-11T02:56:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c133d30 0x7f057c134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient: get_auth_request con 0x7f057c133d30 auth_method 0 2024-09-11T02:56:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 10 monclient(hunting): handle_auth_done global_id 19958 payload 824 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 1 -- 172.21.15.98:0/1748662025 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f057c0732c0 con 0x7f057c133d30 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f0581df9640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c133d30 0x7f057c134140 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f05780107c0 tx=0x7f05780046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f05727fc640 10 client.23836.objecter ms_handle_connect 0x7f057c133d30 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f05727fc640 10 client.23836.objecter resend_mon_ops 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f057801cda0 con 0x7f057c133d30 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f05727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f05727fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:34.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.312+0000 7f05727fc640 10 monclient: dump: 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f0578025070 con 0x7f057c133d30 2024-09-11T02:56:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05727fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c133d30 msgr2=0x7f057c134140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05727fc640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c133d30 0x7f057c134140 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f05780107c0 tx=0x7f05780046d0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05727fc640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c133d30 msgr2=0x7f057c134140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05727fc640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f057c133d30 0x7f057c134140 secure :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f05780107c0 tx=0x7f05780046d0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.343 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:34.343 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 410061, 2024-09-11T02:56:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:34.344 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T02:56:34.345 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.346 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.347 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:34.348 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.349 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.351 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:34.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:34.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:34.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:34.357 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:34.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:34.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:34.364 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:34.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:34.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:34.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.371 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:34.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:34.373 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 10 librados: watch_flush enter 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 10 librados: watch_flush exit 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 20 client.23836.objecter shutdown clearing up homeless session... 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 10 client.23836.objecter successfully canceled tick 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056003df40 msgr2=0x7f0560040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056003df40 0x7f0560040400 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7f05740012f0 tx=0x7f0574009040 comp rx=0 tx=0).stop 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 10 monclient: shutdown 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c0724b0 msgr2=0x7f057c13ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 --2- 172.21.15.98:0/1748662025 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f057c0724b0 0x7f057c13ad10 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f056c009710 tx=0x7f056c019120 comp rx=0 tx=0).stop 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05825fa640 1 -- 172.21.15.98:0/1748662025 reap_dead start 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f05825fa640 1 -- 172.21.15.98:0/1748662025 reap_dead start 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 shutdown_connections 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 >> 172.21.15.98:0/1748662025 conn(0x7f057c0ac6b0 msgr2=0x7f057c0a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 shutdown_connections 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 -- 172.21.15.98:0/1748662025 wait complete. 2024-09-11T02:56:34.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.316+0000 7f058405d640 1 librados: shutdown 2024-09-11T02:56:34.376 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:34.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:56:34.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:56:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.496+0000 7f16f991e640 10 monclient: get_monmap_and_config 2024-09-11T02:56:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.496+0000 7f16f991e640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.496+0000 7f16f991e640 10 monclient: monmap: 2024-09-11T02:56:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:34.500754+0000 2024-09-11T02:56:34.499 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:34.500754+0000 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.500 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.501 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.501 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:34.501 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:34.501 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 1 Processor -- start 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 1 -- start start 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient: init 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient: _add_conns ranks=[3,0,7,8,1,6,10,2,5,9,4] 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 0x7f16f4072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient(hunting): picked mon.noname-g con 0x7f16f4139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16f4139ef0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413a2d0 0x7f16f40a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient(hunting): picked mon.noname-i con 0x7f16f413a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f40702b0 con 0x7f16f4139ef0 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f16f4071bf0 con 0x7f16f4139b10 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f16f4071350 con 0x7f16f413a2d0 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 10 monclient(hunting): authenticate will time out at 1348.381036s 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16ebfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f16f4139ef0 0x7f16f40a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54396/0 (socket says 172.21.15.98:54396) 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16ebfff640 1 -- 172.21.15.98:0/2262896408 learned_addr learned my addr 172.21.15.98:0/2262896408 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 0x7f16f4072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413a2d0 0x7f16f40a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request con 0x7f16f4139b10 auth_method 0 2024-09-11T02:56:34.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2659269098 0 0) 0x7f16dc001120 con 0x7f16f4139ef0 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): _negotiate 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): my global_id is 44298 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16e0003530 con 0x7f16f4139ef0 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): get_auth_request con 0x7f16f413a2d0 auth_method 0 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1156355441 0 0) 0x7f16e0003530 con 0x7f16f4139ef0 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413a2d0 msgr2=0x7f16f40a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413a2d0 0x7f16f40a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 msgr2=0x7f16f4072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 0x7f16f4072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f40a9620 con 0x7f16f4139ef0 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f16dc001b40 con 0x7f16f4139ef0 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: dump: 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.516 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.517 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 >> v1:172.21.15.98:6789/0 conn(0x7f16f4139ef0 legacy=0x7f16f40a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient: _add_conns ranks=[8,3,6,0,5,2,7,1,4] 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16e0004240 0x7f16e0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): picked mon.e con 0x7f16e0004240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 0x7f16e0004d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): picked mon.g con 0x7f16f4139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16e0005340 0x7f16e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): picked mon.d con 0x7f16e0005340 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f16f4071bf0 con 0x7f16f4139b10 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f16f4071350 con 0x7f16e0005340 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f16e000c210 con 0x7f16e0004240 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f27fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16e0005340 0x7f16e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16e0004240 0x7f16e0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): get_auth_request con 0x7f16e0005340 auth_method 0 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request con 0x7f16e0004240 auth_method 0 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16ebfff640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 0x7f16e0004d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16ebfff640 10 monclient(hunting): get_auth_request con 0x7f16f4139b10 auth_method 0 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient(hunting): handle_auth_done global_id 44298 payload 293 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 -- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16e0004240 msgr2=0x7f16e0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16e0004240 0x7f16e0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 -- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 msgr2=0x7f16e0004d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16f4139b10 0x7f16e0004d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 monclient: found mon.d 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:56:34.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 1 -- 172.21.15.98:0/2262896408 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e0003530 con 0x7f16e0005340 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16e0004240 0x7f16e0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 5 monclient: authenticate success, global_id 44298 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.500+0000 7f16f991e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16e0005340 0x7f16e000b9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009fc0 tx=0x7f16ec01cca0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=faa551c10ca33806 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16ec0b2070 con 0x7f16e0005340 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 10 monclient: dump: 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.522 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.523 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec0a5d00 con 0x7f16e0005340 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 1 -- 172.21.15.98:0/2262896408 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16ec009c10 con 0x7f16e0005340 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 4 set_mon_vals no callback set 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f27fc640 10 monclient: dump: 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.524 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.525 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: shutdown 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16e0005340 msgr2=0x7f16e000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 --2- 172.21.15.98:0/2262896408 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16e0005340 0x7f16e000b9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009fc0 tx=0x7f16ec01cca0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 1 -- 172.21.15.98:0/2262896408 reap_dead start 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- 172.21.15.98:0/2262896408 shutdown_connections 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- 172.21.15.98:0/2262896408 >> 172.21.15.98:0/2262896408 conn(0x7f16f40ac790 msgr2=0x7f16f40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- 172.21.15.98:0/2262896408 shutdown_connections 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- 172.21.15.98:0/2262896408 wait complete. 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: monmap: 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.526 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.527 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 librados: starting msgr at 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 librados: starting objecter 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 Processor -- start 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- start start 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 librados: setting wanted keys 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 librados: calling monclient init 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: init 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: _add_conns ranks=[7,2,8,1,4,3,5,6,0] 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16e0005340 0x7f16f41d1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): picked mon.noname-i con 0x7f16e0005340 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:34.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f4137920 0x7f16f4137d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): picked mon.noname-b con 0x7f16f4137920 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16f40a3fb0 0x7f16f40a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): picked mon.noname-e con 0x7f16f40a3fb0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f16f40517e0 con 0x7f16f4137920 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f16f40a8870 con 0x7f16e0005340 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f16f4142250 con 0x7f16f40a3fb0 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient(hunting): authenticate will time out at 1348.386795s 2024-09-11T02:56:34.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f4137920 0x7f16f4137d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16f40a3fb0 0x7f16f40a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f4137920 0x7f16f4137d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:35664/0 (socket says 172.21.15.98:35664) 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 -- 172.21.15.98:0/320329020 learned_addr learned my addr 172.21.15.98:0/320329020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): get_auth_request con 0x7f16f4137920 auth_method 0 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 10 monclient(hunting): get_auth_request con 0x7f16f40a3fb0 auth_method 0 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16e0005340 0x7f16f41d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request con 0x7f16e0005340 auth_method 0 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient(hunting): handle_auth_done global_id 44219 payload 824 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16f40a3fb0 msgr2=0x7f16f40a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16f40a3fb0 0x7f16f40a4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16e0005340 msgr2=0x7f16f41d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16e0005340 0x7f16f41d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 monclient: found mon.noname-b 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 -- 172.21.15.98:0/320329020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f41242d0 con 0x7f16f4137920 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f37fe640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16f40a3fb0 0x7f16f40a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16ebfff640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f4137920 0x7f16f4137d60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00e8b0 tx=0x7f16dc00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 client.?.objecter ms_handle_connect 0x7f16f4137920 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:34.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16dc00cce0 con 0x7f16f4137920 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: dump: 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.533 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16dc018070 con 0x7f16f4137920 2024-09-11T02:56:34.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 5 monclient: authenticate success, global_id 44219 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: _renew_subs 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f4140ae0 con 0x7f16f4137920 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16dc013890 con 0x7f16f4137920 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f1ffb640 4 set_mon_vals no callback set 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f0ff9640 10 monclient: dump: 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.535 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.536 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 client.44219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: _renew_subs 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f4122ea0 con 0x7f16f4137920 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.504+0000 7f16f991e640 1 librados: init done 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f991e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f991e640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f40a4950 0x7f16f413ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f991e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f16f406ca10 con 0x7f16f40a4950 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f16dc013420 con 0x7f16f4137920 2024-09-11T02:56:34.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f40a4950 0x7f16f413ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient: get_auth_request con 0x7f16f40a4950 auth_method 0 2024-09-11T02:56:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f16d403df70 0x7f16d4040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(298..298 src has 1..298) ==== 6658+0+0 (secure 0 0 0) 0x7f16dc010040 con 0x7f16f4137920 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 client.44219.objecter ms_dispatch 0x7f16f4000ba0 osd_map(298..298 src has 1..298) 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 3 client.44219.objecter handle_osd_map got epochs [298,298] > 0 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 3 client.44219.objecter handle_osd_map decoding full epoch 298 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 20 client.44219.objecter dump_active .. 0 homeless 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f37fe640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f16d403df70 0x7f16d4040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f37fe640 10 monclient: get_auth_request con 0x7f16d403df70 auth_method 0 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_done global_id 19564 payload 824 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 1 -- 172.21.15.98:0/320329020 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16f406e680 con 0x7f16f40a4950 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 client.44219.objecter ms_handle_connect 0x7f16f40a4950 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 client.44219.objecter resend_mon_ops 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f40a4950 0x7f16f413ea20 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f16e4014a60 tx=0x7f16e4014f20 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 client.44219.objecter ms_handle_connect 0x7f16d403df70 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f37fe640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f16d403df70 0x7f16d4040430 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0af000 tx=0x7f16ec004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e4012c60 con 0x7f16f40a4950 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:56:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 monclient: dump: 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f16e4012c60 con 0x7f16f40a4950 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f40a4950 msgr2=0x7f16f413ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f40a4950 0x7f16f413ea20 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f16e4014a60 tx=0x7f16e4014f20 comp rx=0 tx=0).stop 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f40a4950 msgr2=0x7f16f413ea20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.508+0000 7f16f0ff9640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f40a4950 0x7f16f413ea20 secure :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f16e4014a60 tx=0x7f16e4014f20 comp rx=0 tx=0).stop 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413ef60 0x7f16f413f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f16f406e870 con 0x7f16f413ef60 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413ef60 0x7f16f413f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient: get_auth_request con 0x7f16f413ef60 auth_method 0 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 10 monclient(hunting): handle_auth_done global_id 19573 payload 824 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 1 -- 172.21.15.98:0/320329020 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16f40a7a10 con 0x7f16f413ef60 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f2ffd640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413ef60 0x7f16f413f370 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f16e4014a60 tx=0x7f16e4004740 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 10 client.44219.objecter ms_handle_connect 0x7f16f413ef60 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 10 client.44219.objecter resend_mon_ops 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e4016e00 con 0x7f16f413ef60 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 10 monclient: dump: 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.546 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f16e4016990 con 0x7f16f413ef60 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413ef60 msgr2=0x7f16f413f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413ef60 0x7f16f413f370 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f16e4014a60 tx=0x7f16e4004740 comp rx=0 tx=0).stop 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413ef60 msgr2=0x7f16f413f370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f0ff9640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f16f413ef60 0x7f16f413f370 secure :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f16e4014a60 tx=0x7f16e4004740 comp rx=0 tx=0).stop 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 56, 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 410285, 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:56:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.549 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.550 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:34.551 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:34.552 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:34.553 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:34.558 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:34.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:34.560 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:34.561 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.567 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:34.568 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.569 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.570 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:34.571 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:34.572 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:34.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 10 librados: watch_flush enter 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 10 librados: watch_flush exit 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 20 client.44219.objecter shutdown clearing up homeless session... 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 10 client.44219.objecter successfully canceled tick 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f16d403df70 msgr2=0x7f16d4040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f16d403df70 0x7f16d4040430 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0af000 tx=0x7f16ec004ac0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 10 monclient: shutdown 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f4137920 msgr2=0x7f16f4137d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f991e640 1 --2- 172.21.15.98:0/320329020 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f16f4137920 0x7f16f4137d60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f16dc00e8b0 tx=0x7f16dc00ed70 comp rx=0 tx=0).stop 2024-09-11T02:56:34.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f37fe640 1 -- 172.21.15.98:0/320329020 reap_dead start 2024-09-11T02:56:34.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.516+0000 7f16f37fe640 1 -- 172.21.15.98:0/320329020 reap_dead start 2024-09-11T02:56:34.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.520+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 shutdown_connections 2024-09-11T02:56:34.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.520+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 >> 172.21.15.98:0/320329020 conn(0x7f16f40ac790 msgr2=0x7f16f41389c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.520+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 shutdown_connections 2024-09-11T02:56:34.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.520+0000 7f16f991e640 1 -- 172.21.15.98:0/320329020 wait complete. 2024-09-11T02:56:34.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.520+0000 7f16f991e640 1 librados: shutdown 2024-09-11T02:56:34.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:56:34.576 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b', 'g'] 2024-09-11T02:56:34.576 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['f', 'h'] 2024-09-11T02:56:34.576 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T02:56:34.576 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-11T02:56:34.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-11T02:56:34.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient: get_monmap_and_config 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient: monmap: 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:34.699483+0000 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:34.699483+0000 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:34.698 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:34.699 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 1 Processor -- start 2024-09-11T02:56:34.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 1 -- start start 2024-09-11T02:56:34.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient: init 2024-09-11T02:56:34.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient: _add_conns ranks=[1,8,2,7,4,3,5,10,9,0,6] 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f41d0139b10 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d006fe30 0x7f41d00725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient(hunting): picked mon.noname-e con 0x7f41d006fe30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d013ea00 0x7f41d0140df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient(hunting): picked mon.noname-b con 0x7f41d013ea00 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41d013a600 con 0x7f41d0139b10 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f41d00a9650 con 0x7f41d013ea00 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f41d0071350 con 0x7f41d006fe30 2024-09-11T02:56:34.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d7b1e640 10 monclient(hunting): authenticate will time out at 1348.579859s 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.696+0000 7f41d50b9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d006fe30 0x7f41d00725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d013ea00 0x7f41d0140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d006fe30 0x7f41d00725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44784/0 (socket says 172.21.15.98:44784) 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 -- 172.21.15.98:0/2977333749 learned_addr learned my addr 172.21.15.98:0/2977333749 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 10 monclient(hunting): get_auth_request con 0x7f41d013ea00 auth_method 0 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): get_auth_request con 0x7f41d006fe30 auth_method 0 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient(hunting): handle_auth_done global_id 24326 payload 274 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 -- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d013ea00 msgr2=0x7f41d0140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 --2- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d013ea00 0x7f41d0140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 -- 172.21.15.98:0/2977333749 >> v1:172.21.15.190:6789/0 conn(0x7f41d0139b10 legacy=0x7f41d0139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 monclient: found mon.noname-e 2024-09-11T02:56:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 -- 172.21.15.98:0/2977333749 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d0071630 con 0x7f41d006fe30 2024-09-11T02:56:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d60bb640 1 --2- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d013ea00 0x7f41d0140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:34.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d50b9640 1 --2- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d006fe30 0x7f41d00725d0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f41c00098d0 tx=0x7f41c001c120 comp rx=0 tx=0).ready entity=mon.8 client_cookie=dd05ac794ae97ad3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 1 -- 172.21.15.98:0/2977333749 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41c00b1070 con 0x7f41d006fe30 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: dump: 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 5 monclient: authenticate success, global_id 24326 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 1 -- 172.21.15.98:0/2977333749 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c00a4af0 con 0x7f41d006fe30 2024-09-11T02:56:34.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 1 -- 172.21.15.98:0/2977333749 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41c00a4d20 con 0x7f41d006fe30 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 4 set_mon_vals no callback set 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d48b8640 10 monclient: dump: 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.714 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 10 monclient: shutdown 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 -- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d006fe30 msgr2=0x7f41d00725d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d006fe30 0x7f41d00725d0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f41c00098d0 tx=0x7f41c001c120 comp rx=0 tx=0).stop 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 -- 172.21.15.98:0/2977333749 shutdown_connections 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d013ea00 0x7f41d0140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/2977333749 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d006fe30 0x7f41d00725d0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 -- 172.21.15.98:0/2977333749 >> 172.21.15.98:0/2977333749 conn(0x7f41d00ac770 msgr2=0x7f41d00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 -- 172.21.15.98:0/2977333749 shutdown_connections 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 1 -- 172.21.15.98:0/2977333749 wait complete. 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.700+0000 7f41d7b1e640 10 monclient: monmap: 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:34.716 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 librados: starting msgr at 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 librados: starting objecter 2024-09-11T02:56:34.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 Processor -- start 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 -- start start 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 librados: setting wanted keys 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 librados: calling monclient init 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient: init 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient: _add_conns ranks=[0,5,8,7,3,1,6,4,2] 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d0139b10 0x7f41d0138360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): picked mon.noname-a con 0x7f41d0139b10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f41d013ea00 0x7f41d01388a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): picked mon.noname-h con 0x7f41d013ea00 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d00a2e60 0x7f41d00a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): picked mon.noname-e con 0x7f41d00a2e60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f41d0144700 con 0x7f41d0139b10 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f41d0144580 con 0x7f41d013ea00 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f41d0144880 con 0x7f41d00a2e60 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): authenticate will time out at 1348.584916s 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d00a2e60 0x7f41d00a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d0139b10 0x7f41d0138360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d00a2e60 0x7f41d00a32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44800/0 (socket says 172.21.15.98:44800) 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d0139b10 0x7f41d0138360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57634/0 (socket says 172.21.15.98:57634) 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 -- 172.21.15.98:0/3496162052 learned_addr learned my addr 172.21.15.98:0/3496162052 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): get_auth_request con 0x7f41d00a2e60 auth_method 0 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): get_auth_request con 0x7f41d0139b10 auth_method 0 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d50b9640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f41d013ea00 0x7f41d01388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient(hunting): handle_auth_done global_id 24335 payload 824 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f41d013ea00 msgr2=0x7f41d01388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f41d013ea00 0x7f41d01388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d0139b10 msgr2=0x7f41d0138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d0139b10 0x7f41d0138360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 monclient: found mon.noname-e 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 -- 172.21.15.98:0/3496162052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41d0122f10 con 0x7f41d00a2e60 2024-09-11T02:56:34.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d50b9640 10 monclient: get_auth_request con 0x7f41d013ea00 auth_method 0 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d50b9640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f41d013ea00 0x7f41d01388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41d0139b10 0x7f41d0138360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d60bb640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d00a2e60 0x7f41d00a32e0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009e10 tx=0x7f41c800c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 client.?.objecter ms_handle_connect 0x7f41d00a2e60 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41c8018020 con 0x7f41d00a2e60 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: dump: 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.722 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c800cdf0 con 0x7f41d00a2e60 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41c8014d60 con 0x7f41d00a2e60 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c7fff640 4 set_mon_vals no callback set 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: dump: 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.724 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 5 monclient: authenticate success, global_id 24335 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient: _renew_subs 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41d01231f0 con 0x7f41d00a2e60 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 client.24335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient: _renew_subs 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f41d00a8a20 con 0x7f41d00a2e60 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 librados: init done 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d006fe10 0x7f41d002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f41d006ca10 con 0x7f41d006fe10 2024-09-11T02:56:34.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d006fe10 0x7f41d002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient: get_auth_request con 0x7f41d006fe10 auth_method 0 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f41c8014480 con 0x7f41d00a2e60 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 10 monclient(hunting): handle_auth_done global_id 44237 payload 824 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 1 -- 172.21.15.98:0/3496162052 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41d00725d0 con 0x7f41d006fe10 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f41b403df40 0x7f41b4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(298..298 src has 1..298) ==== 6658+0+0 (secure 0 0 0) 0x7f41c80508d0 con 0x7f41d00a2e60 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 client.24335.objecter ms_dispatch 0x7f41d0000ba0 osd_map(298..298 src has 1..298) 2024-09-11T02:56:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 3 client.24335.objecter handle_osd_map got epochs [298,298] > 0 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 3 client.24335.objecter handle_osd_map decoding full epoch 298 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d58ba640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d006fe10 0x7f41d002d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00fae0 tx=0x7f41cc004510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 20 client.24335.objecter dump_active .. 0 homeless 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 client.24335.objecter ms_handle_connect 0x7f41d006fe10 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 client.24335.objecter resend_mon_ops 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41cc005030 con 0x7f41d006fe10 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: dump: 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d50b9640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f41b403df40 0x7f41b4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41d50b9640 10 monclient: get_auth_request con 0x7f41b403df40 auth_method 0 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41cc00dad0 con 0x7f41d006fe10 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d006fe10 msgr2=0x7f41d002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d006fe10 0x7f41d002d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00fae0 tx=0x7f41cc004510 comp rx=0 tx=0).stop 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d006fe10 msgr2=0x7f41d002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.704+0000 7f41c67fc640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d006fe10 0x7f41d002d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.708+0000 7f41c67fc640 10 client.24335.objecter ms_handle_connect 0x7f41b403df40 2024-09-11T02:56:34.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.708+0000 7f41d50b9640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f41b403df40 0x7f41b4040400 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f41c00ae000 tx=0x7f41c00a6070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d7b1e640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d00a3fe0 0x7f41d002d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d7b1e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f41d006e870 con 0x7f41d00a3fe0 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d00a3fe0 0x7f41d002d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient: get_auth_request con 0x7f41d00a3fe0 auth_method 0 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 10 monclient(hunting): handle_auth_done global_id 44246 payload 824 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 1 -- 172.21.15.98:0/3496162052 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f41d00a55f0 con 0x7f41d00a3fe0 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41c67fc640 10 client.24335.objecter ms_handle_connect 0x7f41d00a3fe0 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41c67fc640 10 client.24335.objecter resend_mon_ops 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41d58ba640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d00a3fe0 0x7f41d002d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00fae0 tx=0x7f41cc003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41cc02c080 con 0x7f41d00a3fe0 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:56:34.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.712+0000 7f41c67fc640 10 monclient: dump: 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.734 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f41cc00de10 con 0x7f41d00a3fe0 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41c67fc640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41c67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d00a3fe0 msgr2=0x7f41d002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41c67fc640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d00a3fe0 0x7f41d002d990 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00fae0 tx=0x7f41cc003770 comp rx=0 tx=0).stop 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41c67fc640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d00a3fe0 msgr2=0x7f41d002d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41c67fc640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f41d00a3fe0 0x7f41d002d990 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00fae0 tx=0x7f41cc003770 comp rx=0 tx=0).stop 2024-09-11T02:56:34.735 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-09-11T02:56:34.736 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T02:56:34.736 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:34.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 10 librados: watch_flush enter 2024-09-11T02:56:34.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 10 librados: watch_flush exit 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 20 client.24335.objecter shutdown clearing up homeless session... 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 10 client.24335.objecter successfully canceled tick 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f41b403df40 msgr2=0x7f41b4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f41b403df40 0x7f41b4040400 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f41c00ae000 tx=0x7f41c00a6070 comp rx=0 tx=0).stop 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 10 monclient: shutdown 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d00a2e60 msgr2=0x7f41d00a32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d00a2e60 0x7f41d00a32e0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009e10 tx=0x7f41c800c490 comp rx=0 tx=0).stop 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d60bb640 1 -- 172.21.15.98:0/3496162052 reap_dead start 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 shutdown_connections 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 --2- 172.21.15.98:0/3496162052 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f41d00a2e60 0x7f41d00a32e0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 >> 172.21.15.98:0/3496162052 conn(0x7f41d00ac770 msgr2=0x7f41d00ab430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 shutdown_connections 2024-09-11T02:56:34.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 -- 172.21.15.98:0/3496162052 wait complete. 2024-09-11T02:56:34.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.716+0000 7f41d7b1e640 1 librados: shutdown 2024-09-11T02:56:34.739 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T02:56:34.739 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T02:56:34.739 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:56:34.750 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T02:56:34.750 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-09-11T02:56:34.750 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g store 2024-09-11T02:56:34.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g sync_force --yes-i-really-mean-it 2024-09-11T02:56:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.912+0000 7ff7674b2640 10 monclient: get_monmap_and_config 2024-09-11T02:56:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.912+0000 7ff7674b2640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.912+0000 7ff7674b2640 10 monclient: monmap: 2024-09-11T02:56:34.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:34.916106+0000 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:34.916106+0000 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:34.916 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:34.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 1 Processor -- start 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 1 -- start start 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient: init 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient: _add_conns ranks=[4,0,1,8,7,2,9,6,3,10,5] 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff760139b80 0x7ff760072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient(hunting): picked mon.noname-c con 0x7ff760139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff760139f60 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:34.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff76013a380 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7600a8830 con 0x7ff760139f60 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff760070320 con 0x7ff76013a380 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff7600713c0 con 0x7ff760139b80 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff7674b2640 10 monclient(hunting): authenticate will time out at 1348.797349s 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff760139b80 0x7ff760072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff760139b80 0x7ff760072690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:38244/0 (socket says 172.21.15.98:38244) 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 -- 172.21.15.98:0/228401641 learned_addr learned my addr 172.21.15.98:0/228401641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): get_auth_request con 0x7ff760139b80 auth_method 0 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff764a4d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff760139f60 0x7ff7600a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54404/0 (socket says 172.21.15.98:54404) 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient(hunting): handle_auth_done global_id 23845 payload 274 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 -- 172.21.15.98:0/228401641 >> v1:172.21.15.190:6789/0 conn(0x7ff76013a380 legacy=0x7ff7600a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 -- 172.21.15.98:0/228401641 >> v1:172.21.15.98:6789/0 conn(0x7ff760139f60 legacy=0x7ff7600a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 monclient: found mon.noname-c 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 -- 172.21.15.98:0/228401641 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff760072bd0 con 0x7ff760139b80 2024-09-11T02:56:34.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff757fff640 1 -- 172.21.15.98:0/228401641 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3606789880 0 0) 0x7ff7600a8830 con 0x7ff760139f60 2024-09-11T02:56:34.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff757fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:56:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.916+0000 7ff76524e640 1 --2- 172.21.15.98:0/228401641 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff760139b80 0x7ff760072690 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098a0 tx=0x7ff75c01c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1d88175ead4cbe68 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 1 -- 172.21.15.98:0/228401641 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75c0b0070 con 0x7ff760139b80 2024-09-11T02:56:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: dump: 2024-09-11T02:56:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.928 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.929 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 1 -- 172.21.15.98:0/228401641 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75c0aa810 con 0x7ff760139b80 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 5 monclient: authenticate success, global_id 23845 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 1 -- 172.21.15.98:0/228401641 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75c0a9300 con 0x7ff760139b80 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 4 set_mon_vals no callback set 2024-09-11T02:56:34.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff757fff640 10 monclient: dump: 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: shutdown 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- 172.21.15.98:0/228401641 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff760139b80 msgr2=0x7ff760072690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 --2- 172.21.15.98:0/228401641 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff760139b80 0x7ff760072690 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098a0 tx=0x7ff75c01c440 comp rx=0 tx=0).stop 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- 172.21.15.98:0/228401641 shutdown_connections 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 --2- 172.21.15.98:0/228401641 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff760139b80 0x7ff760072690 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- 172.21.15.98:0/228401641 >> 172.21.15.98:0/228401641 conn(0x7ff7600ac7f0 msgr2=0x7ff7600a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- 172.21.15.98:0/228401641 shutdown_connections 2024-09-11T02:56:34.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- 172.21.15.98:0/228401641 wait complete. 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: build_initial_monmap 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: monmap: 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:34.933 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 librados: starting msgr at 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 librados: starting objecter 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 Processor -- start 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- start start 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 librados: setting wanted keys 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 librados: calling monclient init 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: init 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:34.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: _add_conns ranks=[1,3,8,4,6,2,0,5,7] 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760139b80 0x7ff7601d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): picked mon.noname-f con 0x7ff760139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff760139f60 0x7ff7601d2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): picked mon.noname-g con 0x7ff760139f60 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff76013a380 0x7ff7600a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): picked mon.noname-e con 0x7ff76013a380 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff760145a80 con 0x7ff760139b80 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff760145900 con 0x7ff760139f60 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff760145c00 con 0x7ff76013a380 2024-09-11T02:56:34.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient(hunting): authenticate will time out at 1348.802339s 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff76013a380 0x7ff7600a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff76013a380 0x7ff7600a3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44812/0 (socket says 172.21.15.98:44812) 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 -- 172.21.15.98:0/1679491104 learned_addr learned my addr 172.21.15.98:0/1679491104 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): get_auth_request con 0x7ff76013a380 auth_method 0 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff76524e640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760139b80 0x7ff7601d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff764a4d640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff760139f60 0x7ff7601d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff76524e640 10 monclient(hunting): get_auth_request con 0x7ff760139b80 auth_method 0 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff76524e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff76524e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff76524e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff764a4d640 10 monclient(hunting): get_auth_request con 0x7ff760139f60 auth_method 0 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff764a4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff764a4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff764a4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient(hunting): handle_auth_done global_id 24344 payload 824 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff760139f60 msgr2=0x7ff7601d2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff760139f60 0x7ff7601d2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760139b80 msgr2=0x7ff7601d1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff760139b80 0x7ff7601d1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient: _finish_hunting 0 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 monclient: found mon.noname-e 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 -- 172.21.15.98:0/1679491104 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff760135de0 con 0x7ff76013a380 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff765a4f640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff76013a380 0x7ff7600a3060 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7ff75800b9e0 tx=0x7ff75800bea0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 client.?.objecter ms_handle_connect 0x7ff76013a380 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75800cd90 con 0x7ff76013a380 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: dump: 2024-09-11T02:56:34.938 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.939 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 5 monclient: authenticate success, global_id 24344 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff758014440 con 0x7ff76013a380 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: _renew_subs 2024-09-11T02:56:34.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7600a4a20 con 0x7ff76013a380 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff758014ae0 con 0x7ff76013a380 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 client.24344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7577fe640 4 set_mon_vals no callback set 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff755ffb640 10 monclient: dump: 2024-09-11T02:56:34.941 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.942 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: _renew_subs 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7600a9140 con 0x7ff76013a380 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.920+0000 7ff7674b2640 1 librados: init done 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff7674b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff7674b2640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a5ce0 0x7ff7600a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff7674b2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff76006ca10 con 0x7ff7600a5ce0 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ff758014c80 con 0x7ff76013a380 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a5ce0 0x7ff7600a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient: get_auth_request con 0x7ff7600a5ce0 auth_method 0 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff74403df40 0x7ff744040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(298..298 src has 1..298) ==== 6658+0+0 (secure 0 0 0) 0x7ff75804f880 con 0x7ff76013a380 2024-09-11T02:56:34.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 client.24344.objecter ms_dispatch 0x7ff760000ba0 osd_map(298..298 src has 1..298) 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 3 client.24344.objecter handle_osd_map got epochs [298,298] > 0 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 3 client.24344.objecter handle_osd_map decoding full epoch 298 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 20 client.24344.objecter dump_active .. 0 homeless 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff764a4d640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff74403df40 0x7ff744040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff764a4d640 10 monclient: get_auth_request con 0x7ff74403df40 auth_method 0 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 10 monclient(hunting): handle_auth_done global_id 44220 payload 824 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 1 -- 172.21.15.98:0/1679491104 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff760069670 con 0x7ff7600a5ce0 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 client.24344.objecter ms_handle_connect 0x7ff7600a5ce0 2024-09-11T02:56:34.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 client.24344.objecter resend_mon_ops 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff76524e640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a5ce0 0x7ff7600a35a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff760072660 tx=0x7ff75c0191e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 client.24344.objecter ms_handle_connect 0x7ff74403df40 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff764a4d640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff74403df40 0x7ff744040400 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7ff7600a5090 tx=0x7ff750007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75c0b9030 con 0x7ff7600a5ce0 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 monclient: dump: 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.946 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff75c0b1040 con 0x7ff7600a5ce0 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:34.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:34.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a5ce0 msgr2=0x7ff7600a35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a5ce0 0x7ff7600a35a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff760072660 tx=0x7ff75c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a5ce0 msgr2=0x7ff7600a35a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.924+0000 7ff755ffb640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a5ce0 0x7ff7600a35a0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff760072660 tx=0x7ff75c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff7674b2640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff7674b2640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a3ae0 0x7ff7600a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff7674b2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff76006e870 con 0x7ff7600a3ae0 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a3ae0 0x7ff7600a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient: get_auth_request con 0x7ff7600a3ae0 auth_method 0 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 10 monclient(hunting): handle_auth_done global_id 44229 payload 824 2024-09-11T02:56:34.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 1 -- 172.21.15.98:0/1679491104 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7ff7600a4400 con 0x7ff7600a3ae0 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 10 client.24344.objecter ms_handle_connect 0x7ff7600a3ae0 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 10 client.24344.objecter resend_mon_ops 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff76524e640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a3ae0 0x7ff7600a3ec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098a0 tx=0x7ff75c0a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75c0b9040 con 0x7ff7600a3ae0 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 10 monclient: dump: 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:34.951 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7ff75c0be460 con 0x7ff7600a3ae0 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:56:34.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a3ae0 msgr2=0x7ff7600a3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a3ae0 0x7ff7600a3ec0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098a0 tx=0x7ff75c0a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a3ae0 msgr2=0x7ff7600a3ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.932+0000 7ff755ffb640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff7600a3ae0 0x7ff7600a3ec0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098a0 tx=0x7ff75c0a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T02:56:34.953 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 10 librados: watch_flush enter 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 10 librados: watch_flush exit 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 20 client.24344.objecter shutdown clearing up homeless session... 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 10 client.24344.objecter successfully canceled tick 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff74403df40 msgr2=0x7ff744040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff74403df40 0x7ff744040400 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7ff7600a5090 tx=0x7ff750007b60 comp rx=0 tx=0).stop 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 10 monclient: shutdown 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:34.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff76013a380 msgr2=0x7ff7600a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 --2- 172.21.15.98:0/1679491104 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff76013a380 0x7ff7600a3060 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7ff75800b9e0 tx=0x7ff75800bea0 comp rx=0 tx=0).stop 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff765a4f640 1 -- 172.21.15.98:0/1679491104 reap_dead start 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff765a4f640 1 -- 172.21.15.98:0/1679491104 reap_dead start 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 shutdown_connections 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 >> 172.21.15.98:0/1679491104 conn(0x7ff7600ac7f0 msgr2=0x7ff7600aacd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 shutdown_connections 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 -- 172.21.15.98:0/1679491104 wait complete. 2024-09-11T02:56:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:34.936+0000 7ff7674b2640 1 librados: shutdown 2024-09-11T02:56:34.956 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-09-11T02:56:34.957 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-11T02:56:34.957 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:56:34.975 INFO:tasks.ceph.mon.g:Stopped 2024-09-11T02:56:34.976 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-09-11T02:56:34.976 INFO:tasks.ceph.mon.f:Sent signal 19 2024-09-11T02:56:34.976 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-09-11T02:56:34.976 INFO:tasks.ceph.mon.h:Sent signal 19 2024-09-11T02:56:34.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T02:56:49.977 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-09-11T02:56:49.977 INFO:tasks.ceph.mon.f:Sent signal 18 2024-09-11T02:56:49.978 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-09-11T02:56:49.978 INFO:tasks.ceph.mon.h:Sent signal 18 2024-09-11T02:56:49.978 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-09-11T02:56:49.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:56:50.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient: get_monmap_and_config 2024-09-11T02:56:50.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient: build_initial_monmap 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient: monmap: 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:50.107289+0000 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:50.107289+0000 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:50.106 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:50.107 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:50.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:50.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 1 Processor -- start 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 1 -- start start 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient: init 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient: _add_conns ranks=[1,4,0,7,9,10,8,6,2,5,3] 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8f280a48b0 addr v1:172.21.15.190:6789/0 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient(hunting): picked mon.noname-c con 0x7f8f28076b00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f280a53a0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f280a2bf0 con 0x7f8f280a53a0 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f2809cb60 con 0x7f8f280a48b0 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:50.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8f28073f80 con 0x7f8f28076b00 2024-09-11T02:56:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2d0cb640 10 monclient(hunting): authenticate will time out at 1363.987542s 2024-09-11T02:56:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:33056/0 (socket says 172.21.15.98:33056) 2024-09-11T02:56:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2759e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8f280a53a0 0x7f8f2813f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39002/0 (socket says 172.21.15.98:39002) 2024-09-11T02:56:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 learned_addr learned my addr 172.21.15.98:0/1383750440 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f28076b00 auth_method 0 2024-09-11T02:56:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:50.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.103+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:50.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:50.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f28076b00 auth_method 0 2024-09-11T02:56:50.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:50.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:50.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:50.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:50.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:50.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:50.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.307+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f28076b00 auth_method 0 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:50.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:50.711+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f28076b00 auth_method 0 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:51.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:51.515+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:53.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): tick 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 monclient(hunting): continuing hunt 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 msgr2=0x7f8f2813cac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8f28076b00 0x7f8f2813cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 >> v1:172.21.15.190:6789/0 conn(0x7f8f280a48b0 legacy=0x7f8f280a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 >> v1:172.21.15.98:6789/0 conn(0x7f8f280a53a0 legacy=0x7f8f2813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient: _add_conns ranks=[7,8,6,10,3,2,9,5] 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:53.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): picked mon.noname-i con 0x7f8f14003850 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): picked mon.noname-e con 0x7f8f14006370 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): picked mon.noname-d con 0x7f8f14008eb0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8f1400ba80 con 0x7f8f14008eb0 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f8f1400bc00 con 0x7f8f14003850 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8f1400be00 con 0x7f8f14006370 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f24d99640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:53.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 10 monclient(hunting): get_auth_request con 0x7f8f14008eb0 auth_method 0 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f14006370 auth_method 0 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request con 0x7f8f14003850 auth_method 0 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:53.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.107+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 10 monclient(hunting): get_auth_request con 0x7f8f14008eb0 auth_method 0 2024-09-11T02:56:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:53.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f14006370 auth_method 0 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request con 0x7f8f14003850 auth_method 0 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.311+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 10 monclient(hunting): get_auth_request con 0x7f8f14008eb0 auth_method 0 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f14006370 auth_method 0 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request con 0x7f8f14003850 auth_method 0 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:53.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:53.715+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:56:54.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 10 monclient(hunting): get_auth_request con 0x7f8f14008eb0 auth_method 0 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f14006370 auth_method 0 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:54.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request con 0x7f8f14003850 auth_method 0 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:56:54.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:56:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:56:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:54.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:54.519+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:56:56.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): tick 2024-09-11T02:56:56.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:56:56.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 monclient(hunting): continuing hunt 2024-09-11T02:56:56.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 msgr2=0x7f8f140087c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8f14006370 0x7f8f140087c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 msgr2=0x7f8f14005c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14003850 0x7f8f14005c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 msgr2=0x7f8f1400b340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8f14008eb0 0x7f8f1400b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 reap_dead start 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient: _add_conns ranks=[10,3,5,9,2] 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 1 -- 172.21.15.98:0/1383750440 reap_dead start 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> v2:172.21.15.190:3300/0 conn(0x7f8f14003c70 0x7f8f14006790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): picked mon.noname-f con 0x7f8f14003c70 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f14006cd0 0x7f8f14007150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): picked mon.noname-g con 0x7f8f14006cd0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8f14007690 0x7f8f14007b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): picked mon.noname-h con 0x7f8f14007690 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1400ba80 con 0x7f8f14003c70 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1400bc00 con 0x7f8f14006cd0 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 1 -- 172.21.15.98:0/1383750440 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8f1400be00 con 0x7f8f14007690 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f24d99640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2659c640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f14006cd0 msgr2=0x7f8f14007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f14006cd0 0x7f8f14007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> v2:172.21.15.190:3300/0 conn(0x7f8f14003c70 0x7f8f14006790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request con 0x7f8f14003c70 auth_method 0 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8f14007690 0x7f8f14007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 10 monclient(hunting): get_auth_request con 0x7f8f14007690 auth_method 0 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_done global_id 25228 payload 274 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8f14007690 msgr2=0x7f8f14007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8f14007690 0x7f8f14007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f14006cd0 msgr2=0x7f8f14007150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f14006cd0 0x7f8f14007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 1 monclient: found mon.noname-f 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1383750440 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f2806ff80 con 0x7f8f14003c70 2024-09-11T02:56:56.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.107+0000 7f8f2759e640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8f14007690 0x7f8f14007b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.115+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f14006790 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f8f18007c80 tx=0x7f8f1801cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48cb61d01dd90764 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.115+0000 7f8f25d9b640 1 -- 172.21.15.98:0/1383750440 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f18006d10 con 0x7f8f14003c70 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: dump: 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 1 -- 172.21.15.98:0/1383750440 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f180a9070 con 0x7f8f14003c70 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 1 -- 172.21.15.98:0/1383750440 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f180074a0 con 0x7f8f14003c70 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 5 monclient: authenticate success, global_id 25228 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 4 set_mon_vals no callback set 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f25d9b640 10 monclient: dump: 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.122 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.123 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 10 monclient: shutdown 2024-09-11T02:56:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 msgr2=0x7f8f14006790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f14006790 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f8f18007c80 tx=0x7f8f1801cbf0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1383750440 shutdown_connections 2024-09-11T02:56:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8f14007690 0x7f8f14007b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8f14006cd0 0x7f8f14007150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1383750440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f14006790 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1383750440 >> 172.21.15.98:0/1383750440 conn(0x7f8f2806ed00 msgr2=0x7f8f2806f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1383750440 shutdown_connections 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1383750440 wait complete. 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 10 monclient: build_initial_monmap 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 10 monclient: monmap: 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:56.130 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 librados: starting msgr at 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 librados: starting objecter 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 Processor -- start 2024-09-11T02:56:56.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 -- start start 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 librados: setting wanted keys 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 1 librados: calling monclient init 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.119+0000 7f8f2d0cb640 10 monclient: init 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _add_conns ranks=[1,7,2,6,3,0,8,4,5] 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f281d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): picked mon.noname-f con 0x7f8f14003c70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14006cd0 0x7f8f28136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): picked mon.noname-i con 0x7f8f14006cd0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f14007690 0x7f8f280a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): picked mon.noname-b con 0x7f8f14007690 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8f28138210 con 0x7f8f14003c70 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8f28138090 con 0x7f8f14007690 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f8f28138390 con 0x7f8f14006cd0 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient(hunting): authenticate will time out at 1370.004166s 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2759e640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f14007690 msgr2=0x7f8f280a0410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2759e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f14007690 0x7f8f280a0410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f281d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14006cd0 0x7f8f28136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f281d10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:50670/0 (socket says 172.21.15.98:50670) 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1020579761 learned_addr learned my addr 172.21.15.98:0/1020579761 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:56.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request con 0x7f8f14003c70 auth_method 0 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient(hunting): get_auth_request con 0x7f8f14006cd0 auth_method 0 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient(hunting): handle_auth_done global_id 25237 payload 824 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14006cd0 msgr2=0x7f8f28136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14006cd0 0x7f8f28136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1020579761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f14007690 msgr2=0x7f8f280a0410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f14007690 0x7f8f280a0410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 monclient: found mon.noname-f 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 -- 172.21.15.98:0/1020579761 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f281373a0 con 0x7f8f14003c70 2024-09-11T02:56:56.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f26d9d640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f281d10c0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f8f18000c10 tx=0x7f8f180199f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f8f14003c70 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 1 -- 172.21.15.98:0/1020579761 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f180b3070 con 0x7f8f14003c70 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: dump: 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.136 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.137 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.140 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 1 -- 172.21.15.98:0/1020579761 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f180b98f0 con 0x7f8f14003c70 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 1 -- 172.21.15.98:0/1020579761 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8f180b83e0 con 0x7f8f14003c70 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2559a640 4 set_mon_vals no callback set 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 monclient: dump: 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.141 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 5 monclient: authenticate success, global_id 25237 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _renew_subs 2024-09-11T02:56:56.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f280a4080 con 0x7f8f14003c70 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 client.25237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _renew_subs 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f28126030 con 0x7f8f14003c70 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 librados: init done 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f28124b10 con 0x7f8f14003c70 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 1 -- 172.21.15.98:0/1020579761 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8f180b9480 con 0x7f8f14003c70 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8f14040470 0x7f8f14042930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 1 -- 172.21.15.98:0/1020579761 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(301..301 src has 1..301) ==== 6541+0+0 (secure 0 0 0) 0x7f8f18010040 con 0x7f8f14003c70 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 10 client.25237.objecter ms_dispatch 0x7f8f28000ba0 osd_map(301..301 src has 1..301) 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 3 client.25237.objecter handle_osd_map got epochs [301,301] > 0 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 3 client.25237.objecter handle_osd_map decoding full epoch 301 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f0f7fe640 20 client.25237.objecter dump_active .. 0 homeless 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8f14040470 0x7f8f14042930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.123+0000 7f8f2659c640 10 monclient: get_auth_request con 0x7f8f14040470 auth_method 0 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.127+0000 7f8f2659c640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8f14040470 0x7f8f14042930 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f8f280a1b30 tx=0x7f8f1c01b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.127+0000 7f8f0f7fe640 10 client.25237.objecter ms_handle_connect 0x7f8f14040470 2024-09-11T02:56:56.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.135+0000 7f8f0f7fe640 1 -- 172.21.15.98:0/1020579761 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8f280a4080 con 0x7f8f14003c70 2024-09-11T02:56:56.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.135+0000 7f8f0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.135+0000 7f8f0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:56.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.383+0000 7f8f2d0cb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:56:56.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.383+0000 7f8f2d0cb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:56.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.383+0000 7f8f2d0cb640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:56:56.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.383+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8f281d1fa0 con 0x7f8f14003c70 2024-09-11T02:56:56.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.383+0000 7f8f0f7fe640 1 -- 172.21.15.98:0/1020579761 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f8f180169a0 con 0x7f8f14003c70 2024-09-11T02:56:56.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.383+0000 7f8f0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:56:56.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.383+0000 7f8f0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:56.386 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":62,"quorum":[0,1,4,5,6,7,8],"quorum_names":["a","f","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 10 librados: watch_flush enter 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 10 librados: watch_flush exit 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 20 client.25237.objecter shutdown clearing up homeless session... 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 10 client.25237.objecter successfully canceled tick 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8f14040470 msgr2=0x7f8f14042930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8f14040470 0x7f8f14042930 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f8f280a1b30 tx=0x7f8f1c01b040 comp rx=0 tx=0).stop 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 10 monclient: shutdown 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 msgr2=0x7f8f281d10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f281d10c0 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f8f18000c10 tx=0x7f8f180199f0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 shutdown_connections 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8f14040470 0x7f8f14042930 unknown :-1 s=CLOSED pgs=3520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8f14007690 0x7f8f280a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8f14006cd0 0x7f8f28136930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 --2- 172.21.15.98:0/1020579761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f14003c70 0x7f8f281d10c0 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 >> 172.21.15.98:0/1020579761 conn(0x7f8f2806ed00 msgr2=0x7f8f28072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 shutdown_connections 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 -- 172.21.15.98:0/1020579761 wait complete. 2024-09-11T02:56:56.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.387+0000 7f8f2d0cb640 1 librados: shutdown 2024-09-11T02:56:56.406 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-09-11T02:56:56.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-09-11T02:56:56.406 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:56:56.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.523+0000 7f032c078640 10 monclient: get_monmap_and_config 2024-09-11T02:56:56.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.523+0000 7f032c078640 10 monclient: build_initial_monmap 2024-09-11T02:56:56.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.523+0000 7f032c078640 10 monclient: monmap: 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:56.528962+0000 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:56.528962+0000 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:56.528 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:56.529 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:56.529 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:56.529 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:56.529 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:56.529 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 Processor -- start 2024-09-11T02:56:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 -- start start 2024-09-11T02:56:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient: init 2024-09-11T02:56:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient: _add_conns ranks=[10,4,8,2,7,6,9,0,5,1,3] 2024-09-11T02:56:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0324139b10 0x7f0324072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): picked mon.noname-f con 0x7f0324139b10 addr v2:172.21.15.190:3300/0 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0324139ef0 0x7f03240a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): picked mon.noname-c con 0x7f0324139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f032413a2d0 0x7f03240a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): picked mon.noname-e con 0x7f032413a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f03240702b0 con 0x7f0324139b10 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0324071bf0 con 0x7f0324139ef0 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0324071350 con 0x7f032413a2d0 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032c078640 10 monclient(hunting): authenticate will time out at 1370.409221s 2024-09-11T02:56:56.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f032413a2d0 0x7f03240a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0324139ef0 0x7f03240a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f032413a2d0 0x7f03240a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33898/0 (socket says 172.21.15.98:33898) 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 -- 172.21.15.98:0/1575048412 learned_addr learned my addr 172.21.15.98:0/1575048412 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): get_auth_request con 0x7f032413a2d0 auth_method 0 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 10 monclient(hunting): get_auth_request con 0x7f0324139ef0 auth_method 0 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329e14640 1 --2- 172.21.15.98:0/1575048412 >> v2:172.21.15.190:3300/0 conn(0x7f0324139b10 0x7f0324072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329e14640 10 monclient(hunting): get_auth_request con 0x7f0324139b10 auth_method 0 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329e14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329e14640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329e14640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient(hunting): handle_auth_done global_id 24353 payload 274 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 -- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0324139ef0 msgr2=0x7f03240a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:56.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 --2- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0324139ef0 0x7f03240a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 -- 172.21.15.98:0/1575048412 >> v2:172.21.15.190:3300/0 conn(0x7f0324139b10 msgr2=0x7f0324072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 --2- 172.21.15.98:0/1575048412 >> v2:172.21.15.190:3300/0 conn(0x7f0324139b10 0x7f0324072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 monclient: found mon.noname-e 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 -- 172.21.15.98:0/1575048412 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03240a9620 con 0x7f032413a2d0 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f0329613640 1 --2- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0324139ef0 0x7f03240a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.527+0000 7f032a615640 1 --2- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f032413a2d0 0x7f03240a7f50 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f031c009e50 tx=0x7f031c01c980 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5eb46071ad7f9eb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 1 -- 172.21.15.98:0/1575048412 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f031c0090d0 con 0x7f032413a2d0 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: dump: 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.538 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 5 monclient: authenticate success, global_id 24353 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 1 -- 172.21.15.98:0/1575048412 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031c0a5a30 con 0x7f032413a2d0 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 4 set_mon_vals no callback set 2024-09-11T02:56:56.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 1 -- 172.21.15.98:0/1575048412 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f031c0a5be0 con 0x7f032413a2d0 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0328e12640 10 monclient: dump: 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient: shutdown 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f032413a2d0 msgr2=0x7f03240a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 --2- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f032413a2d0 0x7f03240a7f50 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f031c009e50 tx=0x7f031c01c980 comp rx=0 tx=0).stop 2024-09-11T02:56:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- 172.21.15.98:0/1575048412 shutdown_connections 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 --2- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f032413a2d0 0x7f03240a7f50 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 --2- 172.21.15.98:0/1575048412 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0324139ef0 0x7f03240a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 --2- 172.21.15.98:0/1575048412 >> v2:172.21.15.190:3300/0 conn(0x7f0324139b10 0x7f0324072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- 172.21.15.98:0/1575048412 >> 172.21.15.98:0/1575048412 conn(0x7f03240ac790 msgr2=0x7f03240a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- 172.21.15.98:0/1575048412 shutdown_connections 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- 172.21.15.98:0/1575048412 wait complete. 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient: build_initial_monmap 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient: monmap: 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 librados: starting msgr at 2024-09-11T02:56:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 librados: starting objecter 2024-09-11T02:56:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 Processor -- start 2024-09-11T02:56:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- start start 2024-09-11T02:56:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 librados: setting wanted keys 2024-09-11T02:56:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 librados: calling monclient init 2024-09-11T02:56:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient: init 2024-09-11T02:56:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient: _add_conns ranks=[3,2,5,1,6,8,7,0,4] 2024-09-11T02:56:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0324139b10 0x7f03241d1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): picked mon.noname-g con 0x7f0324139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0324139ef0 0x7f03241d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): picked mon.noname-b con 0x7f0324139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f032413a2d0 0x7f032413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): picked mon.noname-h con 0x7f032413a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0324143460 con 0x7f0324139ef0 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f03240a88a0 con 0x7f0324139b10 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f03241435e0 con 0x7f032413a2d0 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032c078640 10 monclient(hunting): authenticate will time out at 1370.414429s 2024-09-11T02:56:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0329613640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0324139ef0 msgr2=0x7f03241d1c70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0329613640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0324139ef0 0x7f03241d1c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0329e14640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0324139b10 msgr2=0x7f03241d1730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f0329e14640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0324139b10 0x7f03241d1730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f032413a2d0 0x7f032413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f032413a2d0 0x7f032413a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:41958/0 (socket says 172.21.15.98:41958) 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 -- 172.21.15.98:0/1192324751 learned_addr learned my addr 172.21.15.98:0/1192324751 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): get_auth_request con 0x7f032413a2d0 auth_method 0 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient(hunting): handle_auth_done global_id 19985 payload 824 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0324139b10 msgr2=0x7f03241d1730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0324139b10 0x7f03241d1730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0324139ef0 msgr2=0x7f03241d1c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0324139ef0 0x7f03241d1c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 monclient: found mon.noname-h 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 -- 172.21.15.98:0/1192324751 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032413dbd0 con 0x7f032413a2d0 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f032a615640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f032413a2d0 0x7f032413a6e0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f031c009040 tx=0x7f031c019720 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f03167fc640 10 client.?.objecter ms_handle_connect 0x7f032413a2d0 2024-09-11T02:56:56.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.531+0000 7f03167fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f031c0bc020 con 0x7f032413a2d0 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: dump: 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 5 monclient: authenticate success, global_id 19985 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 10 monclient: _renew_subs 2024-09-11T02:56:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032413deb0 con 0x7f032413a2d0 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031c0039a0 con 0x7f032413a2d0 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 10 client.19985.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 10 monclient: _renew_subs 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f03240a9e00 con 0x7f032413a2d0 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 1 librados: init done 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0317fff640 4 set_mon_vals no callback set 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f031c009af0 con 0x7f032413a2d0 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: dump: 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032413c230 0x7f032413b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f032406ca10 con 0x7f032413c230 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032413c230 0x7f032413b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient: get_auth_request con 0x7f032413c230 auth_method 0 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 10 monclient(hunting): handle_auth_done global_id 44334 payload 824 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 1 -- 172.21.15.98:0/1192324751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f032406e680 con 0x7f032413c230 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329e14640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032413c230 0x7f032413b3e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0320000ee0 tx=0x7f032000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 client.19985.objecter ms_handle_connect 0x7f032413c230 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 client.19985.objecter resend_mon_ops 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0320017020 con 0x7f032413c230 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: dump: 2024-09-11T02:56:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f031c0a5890 con 0x7f032413a2d0 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f030803e210 0x7f03080406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(301..301 src has 1..301) ==== 6541+0+0 (secure 0 0 0) 0x7f031c0b3040 con 0x7f032413a2d0 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 client.19985.objecter ms_dispatch 0x7f0324000ba0 osd_map(301..301 src has 1..301) 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 3 client.19985.objecter handle_osd_map got epochs [301,301] > 0 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 3 client.19985.objecter handle_osd_map decoding full epoch 301 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 20 client.19985.objecter dump_active .. 0 homeless 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f032000ecd0 con 0x7f032413c230 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032413c230 msgr2=0x7f032413b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032413c230 0x7f032413b3e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0320000ee0 tx=0x7f032000eba0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032413c230 msgr2=0x7f032413b3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032413c230 0x7f032413b3e0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0320000ee0 tx=0x7f032000eba0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329613640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f030803e210 0x7f03080406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329613640 10 monclient: get_auth_request con 0x7f030803e210 auth_method 0 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f03167fc640 10 client.19985.objecter ms_handle_connect 0x7f030803e210 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.535+0000 7f0329613640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f030803e210 0x7f03080406d0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f0318009580 tx=0x7f0318009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032402d450 0x7f032402d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f032406e870 con 0x7f032402d450 2024-09-11T02:56:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032402d450 0x7f032402d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient: get_auth_request con 0x7f032402d450 auth_method 0 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 10 monclient(hunting): handle_auth_done global_id 44343 payload 824 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 1 -- 172.21.15.98:0/1192324751 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0324072600 con 0x7f032402d450 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 10 client.19985.objecter ms_handle_connect 0x7f032402d450 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f0329e14640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032402d450 0x7f032402d860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0320000ee0 tx=0x7f0320004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 10 client.19985.objecter resend_mon_ops 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f032000f040 con 0x7f032402d450 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 10 monclient: dump: 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9244 (secure 0 0 0) 0x7f032000cae0 con 0x7f032402d450 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032402d450 msgr2=0x7f032402d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032402d450 0x7f032402d860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0320000ee0 tx=0x7f0320004740 comp rx=0 tx=0).stop 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032402d450 msgr2=0x7f032402d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f03167fc640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f032402d450 0x7f032402d860 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0320000ee0 tx=0x7f0320004740 comp rx=0 tx=0).stop 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 62, 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 42983, 2024-09-11T02:56:56.570 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:56.571 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.572 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.573 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:56.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.575 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:56.576 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.577 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.579 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:56.582 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:56.584 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.586 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:56.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.589 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:56.591 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.592 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:56.593 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 10 librados: watch_flush enter 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 10 librados: watch_flush exit 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 20 client.19985.objecter shutdown clearing up homeless session... 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 10 client.19985.objecter successfully canceled tick 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f030803e210 msgr2=0x7f03080406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f030803e210 0x7f03080406d0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f0318009580 tx=0x7f0318009190 comp rx=0 tx=0).stop 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 10 monclient: shutdown 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f032413a2d0 msgr2=0x7f032413a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032a615640 1 -- 172.21.15.98:0/1192324751 reap_dead start 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f032413a2d0 0x7f032413a6e0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f031c009040 tx=0x7f031c019720 comp rx=0 tx=0).stop 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 shutdown_connections 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 --2- 172.21.15.98:0/1192324751 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f032413a2d0 0x7f032413a6e0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 >> 172.21.15.98:0/1192324751 conn(0x7f03240ac790 msgr2=0x7f0324138140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 shutdown_connections 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 -- 172.21.15.98:0/1192324751 wait complete. 2024-09-11T02:56:56.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.543+0000 7f032c078640 1 librados: shutdown 2024-09-11T02:56:56.596 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:56:56.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-09-11T02:56:56.597 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:56:56.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient: get_monmap_and_config 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient: build_initial_monmap 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient: monmap: 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:56.713420+0000 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:56.713420+0000 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:56.712 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:56.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 Processor -- start 2024-09-11T02:56:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 -- start start 2024-09-11T02:56:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient: init 2024-09-11T02:56:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:56.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient: _add_conns ranks=[4,5,2,1,10,0,6,8,7,9,3] 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a3de0 0x7f31dc0a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): picked mon.noname-c con 0x7f31dc0a3de0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc0a41c0 0x7f31dc0a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): picked mon.noname-h con 0x7f31dc0a41c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc09b240 0x7f31dc09b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): picked mon.noname-b con 0x7f31dc09b240 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31dc0a0170 con 0x7f31dc09b240 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc068fa0 con 0x7f31dc0a3de0 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc068b40 con 0x7f31dc0a41c0 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31e12b0640 10 monclient(hunting): authenticate will time out at 1370.593764s 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31db7fe640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc09b240 msgr2=0x7f31dc09b660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31db7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc09b240 0x7f31dc09b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a3de0 0x7f31dc0a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a3de0 0x7f31dc0a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:33090/0 (socket says 172.21.15.98:33090) 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 -- 172.21.15.98:0/2904836030 learned_addr learned my addr 172.21.15.98:0/2904836030 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): get_auth_request con 0x7f31dc0a3de0 auth_method 0 2024-09-11T02:56:56.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient(hunting): handle_auth_done global_id 23863 payload 274 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 -- 172.21.15.98:0/2904836030 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc0a41c0 msgr2=0x7f31dc0a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31da7fc640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc0a41c0 0x7f31dc0a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc0a41c0 0x7f31dc0a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 -- 172.21.15.98:0/2904836030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc09b240 msgr2=0x7f31dc09b660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc09b240 0x7f31dc09b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 monclient: found mon.noname-c 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31daffd640 1 -- 172.21.15.98:0/2904836030 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc09eaf0 con 0x7f31dc0a3de0 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31da7fc640 10 monclient: get_auth_request con 0x7f31dc0a41c0 auth_method 0 2024-09-11T02:56:56.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.711+0000 7f31da7fc640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc0a41c0 0x7f31dc0a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:56.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31daffd640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a3de0 0x7f31dc0a0570 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009ee0 tx=0x7f31d401c910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=bc55b6a6cd410fa6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 1 -- 172.21.15.98:0/2904836030 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d40090d0 con 0x7f31dc0a3de0 2024-09-11T02:56:56.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:56:56.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: dump: 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.727 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 5 monclient: authenticate success, global_id 23863 2024-09-11T02:56:56.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 1 -- 172.21.15.98:0/2904836030 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d40a5a70 con 0x7f31dc0a3de0 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 4 set_mon_vals no callback set 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 1 -- 172.21.15.98:0/2904836030 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d40a5c60 con 0x7f31dc0a3de0 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d9ffb640 10 monclient: dump: 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.729 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient: shutdown 2024-09-11T02:56:56.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- 172.21.15.98:0/2904836030 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a3de0 msgr2=0x7f31dc0a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a3de0 0x7f31dc0a0570 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009ee0 tx=0x7f31d401c910 comp rx=0 tx=0).stop 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- 172.21.15.98:0/2904836030 shutdown_connections 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31dc09b240 0x7f31dc09b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc0a41c0 0x7f31dc0a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 --2- 172.21.15.98:0/2904836030 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc0a3de0 0x7f31dc0a0570 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- 172.21.15.98:0/2904836030 >> 172.21.15.98:0/2904836030 conn(0x7f31dc0a4ed0 msgr2=0x7f31dc0a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- 172.21.15.98:0/2904836030 shutdown_connections 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- 172.21.15.98:0/2904836030 wait complete. 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient: build_initial_monmap 2024-09-11T02:56:56.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient: monmap: 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:56.732 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 librados: starting msgr at 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 librados: starting objecter 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 Processor -- start 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- start start 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 librados: setting wanted keys 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 librados: calling monclient init 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient: init 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient: _add_conns ranks=[5,7,8,2,3,6,4,1,0] 2024-09-11T02:56:56.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc09b240 0x7f31dc13c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): picked mon.noname-h con 0x7f31dc09b240 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31dc0a3de0 0x7f31dc137640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): picked mon.noname-i con 0x7f31dc0a3de0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31dc0a41c0 0x7f31dc137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): picked mon.noname-e con 0x7f31dc0a41c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc1413a0 con 0x7f31dc09b240 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f31dc141220 con 0x7f31dc0a3de0 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f31dc141520 con 0x7f31dc0a41c0 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31e12b0640 10 monclient(hunting): authenticate will time out at 1370.599154s 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31dc0a41c0 0x7f31dc137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31dc0a41c0 0x7f31dc137b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33902/0 (socket says 172.21.15.98:33902) 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 -- 172.21.15.98:0/3995781092 learned_addr learned my addr 172.21.15.98:0/3995781092 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): get_auth_request con 0x7f31dc0a41c0 auth_method 0 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31daffd640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc09b240 0x7f31dc13c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient(hunting): handle_auth_done global_id 24362 payload 824 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31da7fc640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31dc0a3de0 0x7f31dc137640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31dc0a3de0 msgr2=0x7f31dc137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31dc0a3de0 0x7f31dc137640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc09b240 msgr2=0x7f31dc13c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc09b240 0x7f31dc13c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 monclient: found mon.noname-e 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 -- 172.21.15.98:0/3995781092 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31dc13d650 con 0x7f31dc0a41c0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31daffd640 10 monclient: get_auth_request con 0x7f31dc09b240 auth_method 0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31daffd640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31dc09b240 0x7f31dc13c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31da7fc640 10 monclient: get_auth_request con 0x7f31dc0a3de0 auth_method 0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31da7fc640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31dc0a3de0 0x7f31dc137640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31db7fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31dc0a41c0 0x7f31dc137b80 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b490 tx=0x7f31cc00b950 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d37fe640 10 client.?.objecter ms_handle_connect 0x7f31dc0a41c0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d37fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.715+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31cc017070 con 0x7f31dc0a41c0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: dump: 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.737 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc003ba0 con 0x7f31dc0a41c0 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 5 monclient: authenticate success, global_id 24362 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31cc003dd0 con 0x7f31dc0a41c0 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: dump: 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.739 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d97fa640 4 set_mon_vals no callback set 2024-09-11T02:56:56.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 10 monclient: _renew_subs 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31dc09c890 con 0x7f31dc0a41c0 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 10 client.24362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 10 monclient: _renew_subs 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f31dc131df0 con 0x7f31dc0a41c0 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 1 librados: init done 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1540 0x7f31dc1d1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc06ca10 con 0x7f31dc1d1540 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1540 0x7f31dc1d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient: get_auth_request con 0x7f31dc1d1540 auth_method 0 2024-09-11T02:56:56.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f31cc0060d0 con 0x7f31dc0a41c0 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 10 monclient(hunting): handle_auth_done global_id 23872 payload 824 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 1 -- 172.21.15.98:0/3995781092 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31dc09b760 con 0x7f31dc1d1540 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f31c003df70 0x7f31c0040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(301..301 src has 1..301) ==== 6541+0+0 (secure 0 0 0) 0x7f31cc04e940 con 0x7f31dc0a41c0 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 client.24362.objecter ms_dispatch 0x7f31dc000ba0 osd_map(301..301 src has 1..301) 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 3 client.24362.objecter handle_osd_map got epochs [301,301] > 0 2024-09-11T02:56:56.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 3 client.24362.objecter handle_osd_map decoding full epoch 301 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31daffd640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1540 0x7f31dc1d1920 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009eb0 tx=0x7f31d40196b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 20 client.24362.objecter dump_active .. 0 homeless 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 client.24362.objecter ms_handle_connect 0x7f31dc1d1540 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 client.24362.objecter resend_mon_ops 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d40bb070 con 0x7f31dc1d1540 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: dump: 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.743 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31da7fc640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f31c003df70 0x7f31c0040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31da7fc640 10 monclient: get_auth_request con 0x7f31c003df70 auth_method 0 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31d4004040 con 0x7f31dc1d1540 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1540 msgr2=0x7f31dc1d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1540 0x7f31dc1d1920 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009eb0 tx=0x7f31d40196b0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1540 msgr2=0x7f31dc1d1920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1540 0x7f31dc1d1920 secure :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009eb0 tx=0x7f31d40196b0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31d37fe640 10 client.24362.objecter ms_handle_connect 0x7f31c003df70 2024-09-11T02:56:56.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.719+0000 7f31da7fc640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f31c003df70 0x7f31c0040430 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f31dc06f4e0 tx=0x7f31c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.723+0000 7f31e12b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.723+0000 7f31e12b0640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1e60 0x7f31dc1387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.723+0000 7f31e12b0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.723+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31dc06e870 con 0x7f31dc1d1e60 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.723+0000 7f31daffd640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1e60 0x7f31dc1387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient: get_auth_request con 0x7f31dc1d1e60 auth_method 0 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 10 monclient(hunting): handle_auth_done global_id 23881 payload 824 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 1 -- 172.21.15.98:0/3995781092 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31dc13db20 con 0x7f31dc1d1e60 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 10 client.24362.objecter ms_handle_connect 0x7f31dc1d1e60 2024-09-11T02:56:56.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 10 client.24362.objecter resend_mon_ops 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31daffd640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1e60 0x7f31dc1387c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f31d401c740 tx=0x7f31d40a7000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31d40a5d90 con 0x7f31dc1d1e60 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 10 monclient: dump: 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.749 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7f31d40b7420 con 0x7f31dc1d1e60 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1e60 msgr2=0x7f31dc1387c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1e60 0x7f31dc1387c0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f31d401c740 tx=0x7f31d40a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1e60 msgr2=0x7f31dc1387c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:56.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31d37fe640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31dc1d1e60 0x7f31dc1387c0 secure :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f31d401c740 tx=0x7f31d40a7000 comp rx=0 tx=0).stop 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 62, 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 432512, 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:56.751 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.752 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.753 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.754 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:56.758 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.759 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.763 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:56.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.768 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.770 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:56.773 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.775 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.777 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:56.778 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 10 librados: watch_flush enter 2024-09-11T02:56:56.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 10 librados: watch_flush exit 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 20 client.24362.objecter shutdown clearing up homeless session... 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 10 client.24362.objecter successfully canceled tick 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f31c003df70 msgr2=0x7f31c0040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f31c003df70 0x7f31c0040430 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f31dc06f4e0 tx=0x7f31c4008040 comp rx=0 tx=0).stop 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 10 monclient: shutdown 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31dc0a41c0 msgr2=0x7f31dc137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31db7fe640 1 -- 172.21.15.98:0/3995781092 reap_dead start 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31dc0a41c0 0x7f31dc137b80 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00b490 tx=0x7f31cc00b950 comp rx=0 tx=0).stop 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 shutdown_connections 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 --2- 172.21.15.98:0/3995781092 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31dc0a41c0 0x7f31dc137b80 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 >> 172.21.15.98:0/3995781092 conn(0x7f31dc0a4ed0 msgr2=0x7f31dc0a62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 shutdown_connections 2024-09-11T02:56:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 -- 172.21.15.98:0/3995781092 wait complete. 2024-09-11T02:56:56.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.727+0000 7f31e12b0640 1 librados: shutdown 2024-09-11T02:56:56.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:56:56.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-09-11T02:56:56.782 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient: get_monmap_and_config 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient: build_initial_monmap 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient: monmap: 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:56.897700+0000 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:56.897700+0000 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.896 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:56.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 Processor -- start 2024-09-11T02:56:56.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 -- start start 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient: init 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient: _add_conns ranks=[7,3,2,0,6,9,4,10,8,5,1] 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1e9c139b80 0x7f1e9c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): picked mon.noname-i con 0x7f1e9c139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e9c06fe50 0x7f1e9c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): picked mon.noname-g con 0x7f1e9c06fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e9c072640 0x7f1e9c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): picked mon.noname-b con 0x7f1e9c072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c0719c0 con 0x7f1e9c072640 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1e9c071c10 con 0x7f1e9c06fe50 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f1e9c0702d0 con 0x7f1e9c139b80 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea3b33640 10 monclient(hunting): authenticate will time out at 1370.778334s 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea20d0640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e9c072640 msgr2=0x7f1e9c0a7f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea20d0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e9c072640 0x7f1e9c0a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea10ce640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e9c06fe50 msgr2=0x7f1e9c0a7a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea10ce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e9c06fe50 0x7f1e9c0a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1e9c139b80 0x7f1e9c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1e9c139b80 0x7f1e9c139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:57508/0 (socket says 172.21.15.98:57508) 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3423831867 learned_addr learned my addr 172.21.15.98:0/3423831867 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:56.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): get_auth_request con 0x7f1e9c139b80 auth_method 0 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_done global_id 19591 payload 274 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3423831867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e9c06fe50 msgr2=0x7f1e9c0a7a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 --2- 172.21.15.98:0/3423831867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e9c06fe50 0x7f1e9c0a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3423831867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e9c072640 msgr2=0x7f1e9c0a7f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 --2- 172.21.15.98:0/3423831867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e9c072640 0x7f1e9c0a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 monclient: found mon.noname-i 2024-09-11T02:56:56.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T02:56:56.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.895+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3423831867 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c071370 con 0x7f1e9c139b80 2024-09-11T02:56:56.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea18cf640 1 --2- 172.21.15.98:0/3423831867 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1e9c139b80 0x7f1e9c139f60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f1e980098d0 tx=0x7f1e9801c480 comp rx=0 tx=0).ready entity=mon.7 client_cookie=e414b8720863630b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 1 -- 172.21.15.98:0/3423831867 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e980b0070 con 0x7f1e9c139b80 2024-09-11T02:56:56.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T02:56:56.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: dump: 2024-09-11T02:56:56.909 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.910 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 1 -- 172.21.15.98:0/3423831867 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e980aa810 con 0x7f1e9c139b80 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea3b33640 5 monclient: authenticate success, global_id 19591 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea3b33640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea3b33640 4 set_mon_vals no callback set 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 1 -- 172.21.15.98:0/3423831867 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e980aa9c0 con 0x7f1e9c139b80 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea08cd640 10 monclient: dump: 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.912 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea3b33640 10 monclient: shutdown 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea3b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3423831867 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1e9c139b80 msgr2=0x7f1e9c139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.899+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3423831867 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1e9c139b80 0x7f1e9c139f60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f1e980098d0 tx=0x7f1e9801c480 comp rx=0 tx=0).stop 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3423831867 shutdown_connections 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3423831867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1e9c072640 0x7f1e9c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3423831867 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1e9c06fe50 0x7f1e9c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3423831867 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1e9c139b80 0x7f1e9c139f60 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3423831867 >> 172.21.15.98:0/3423831867 conn(0x7f1e9c0ac730 msgr2=0x7f1e9c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3423831867 shutdown_connections 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3423831867 wait complete. 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: build_initial_monmap 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: monmap: 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:56.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 librados: starting msgr at 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 librados: starting objecter 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 Processor -- start 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- start start 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 librados: setting wanted keys 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 librados: calling monclient init 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: init 2024-09-11T02:56:56.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: _add_conns ranks=[8,1,5,4,7,2,0,3,6] 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1e9c06fe50 0x7f1e9c13a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): picked mon.noname-e con 0x7f1e9c06fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e9c072640 0x7f1e9c13abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): picked mon.noname-f con 0x7f1e9c072640 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e9c139b80 0x7f1e9c13b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): picked mon.noname-h con 0x7f1e9c139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1e9c1435c0 con 0x7f1e9c072640 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1e9c143440 con 0x7f1e9c139b80 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1e9c143740 con 0x7f1e9c06fe50 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): authenticate will time out at 1370.785510s 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1e9c06fe50 0x7f1e9c13a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1e9c06fe50 0x7f1e9c13a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:33912/0 (socket says 172.21.15.98:33912) 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3687057404 learned_addr learned my addr 172.21.15.98:0/3687057404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): get_auth_request con 0x7f1e9c06fe50 auth_method 0 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea20d0640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e9c139b80 0x7f1e9c13b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e9c072640 0x7f1e9c13abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient(hunting): handle_auth_done global_id 24371 payload 824 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e9c139b80 msgr2=0x7f1e9c13b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e9c139b80 0x7f1e9c13b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e9c072640 msgr2=0x7f1e9c13abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e9c072640 0x7f1e9c13abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient: _finish_hunting 0 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 monclient: found mon.noname-e 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 -- 172.21.15.98:0/3687057404 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e9c0aa780 con 0x7f1e9c06fe50 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient: get_auth_request con 0x7f1e9c072640 auth_method 0 2024-09-11T02:56:56.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea20d0640 10 monclient: get_auth_request con 0x7f1e9c139b80 auth_method 0 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e9c072640 0x7f1e9c13abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea20d0640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1e9c139b80 0x7f1e9c13b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea18cf640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1e9c06fe50 0x7f1e9c13a6b0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1e9801c2b0 tx=0x7f1e98019220 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 client.?.objecter ms_handle_connect 0x7f1e9c06fe50 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e980b0040 con 0x7f1e9c06fe50 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: dump: 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.919 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: _finish_auth 0 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e980b9070 con 0x7f1e9c06fe50 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 5 monclient: authenticate success, global_id 24371 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: _renew_subs 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e9c13d020 con 0x7f1e9c06fe50 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e980aae60 con 0x7f1e9c06fe50 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:56.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: dump: 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.922 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 client.24371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: _renew_subs 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e9c1359d0 con 0x7f1e9c06fe50 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8ffff640 4 set_mon_vals no callback set 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 librados: init done 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c13bb00 0x7f1e9c063640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1e9c06ca10 con 0x7f1e9c13bb00 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c13bb00 0x7f1e9c063640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient: get_auth_request con 0x7f1e9c13bb00 auth_method 0 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1e980b3070 con 0x7f1e9c06fe50 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_done global_id 44259 payload 824 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 1 -- 172.21.15.98:0/3687057404 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e9c139f60 con 0x7f1e9c13bb00 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e8003df70 0x7f1e80040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(301..301 src has 1..301) ==== 6541+0+0 (secure 0 0 0) 0x7f1e980e9440 con 0x7f1e9c06fe50 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 client.24371.objecter ms_dispatch 0x7f1e9c000ba0 osd_map(301..301 src has 1..301) 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 3 client.24371.objecter handle_osd_map got epochs [301,301] > 0 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 3 client.24371.objecter handle_osd_map decoding full epoch 301 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 20 client.24371.objecter dump_active .. 0 homeless 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 client.24371.objecter ms_handle_connect 0x7f1e9c13bb00 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 client.24371.objecter resend_mon_ops 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea10ce640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c13bb00 0x7f1e9c063640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0a4d60 tx=0x7f1e9000cb80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1ea20d0640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e8003df70 0x7f1e80040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e90003e30 con 0x7f1e9c13bb00 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:56.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.903+0000 7f1e8e7fc640 10 monclient: dump: 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.926 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1ea20d0640 10 monclient: get_auth_request con 0x7f1e8003df70 auth_method 0 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e90016070 con 0x7f1e9c13bb00 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c13bb00 msgr2=0x7f1e9c063640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c13bb00 0x7f1e9c063640 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0a4d60 tx=0x7f1e9000cb80 comp rx=0 tx=0).stop 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c13bb00 msgr2=0x7f1e9c063640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c13bb00 0x7f1e9c063640 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0a4d60 tx=0x7f1e9000cb80 comp rx=0 tx=0).stop 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1ea20d0640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e8003df70 0x7f1e80040430 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f1e94003cc0 tx=0x7f1e94007320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.907+0000 7f1e8e7fc640 10 client.24371.objecter ms_handle_connect 0x7f1e8003df70 2024-09-11T02:56:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c063b80 0x7f1e9c063f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1e9c06e870 con 0x7f1e9c063b80 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c063b80 0x7f1e9c063f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient: get_auth_request con 0x7f1e9c063b80 auth_method 0 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 10 monclient(hunting): handle_auth_done global_id 44268 payload 824 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 1 -- 172.21.15.98:0/3687057404 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e9c069090 con 0x7f1e9c063b80 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 10 client.24371.objecter ms_handle_connect 0x7f1e9c063b80 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea10ce640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c063b80 0x7f1e9c063f90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e90015000 tx=0x7f1e900059e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 10 client.24371.objecter resend_mon_ops 2024-09-11T02:56:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e9001ce00 con 0x7f1e9c063b80 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 10 monclient: dump: 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9244 (secure 0 0 0) 0x7f1e9001ce00 con 0x7f1e9c063b80 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c063b80 msgr2=0x7f1e9c063f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c063b80 0x7f1e9c063f90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e90015000 tx=0x7f1e900059e0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c063b80 msgr2=0x7f1e9c063f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1e8e7fc640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1e9c063b80 0x7f1e9c063f90 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1e90015000 tx=0x7f1e900059e0 comp rx=0 tx=0).stop 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 62, 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 43339, 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:56.933 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.934 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:56.935 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.936 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:56.937 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:56.938 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:56.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.940 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:56.943 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:56.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.945 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.951 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:56.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:56.954 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:56.955 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:56.956 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 10 librados: watch_flush enter 2024-09-11T02:56:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 10 librados: watch_flush exit 2024-09-11T02:56:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 20 client.24371.objecter shutdown clearing up homeless session... 2024-09-11T02:56:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 10 client.24371.objecter successfully canceled tick 2024-09-11T02:56:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e8003df70 msgr2=0x7f1e80040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1e8003df70 0x7f1e80040430 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f1e94003cc0 tx=0x7f1e94007320 comp rx=0 tx=0).stop 2024-09-11T02:56:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 10 monclient: shutdown 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1e9c06fe50 msgr2=0x7f1e9c13a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 --2- 172.21.15.98:0/3687057404 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1e9c06fe50 0x7f1e9c13a6b0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1e9801c2b0 tx=0x7f1e98019220 comp rx=0 tx=0).stop 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea20d0640 1 -- 172.21.15.98:0/3687057404 reap_dead start 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea20d0640 1 -- 172.21.15.98:0/3687057404 reap_dead start 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 shutdown_connections 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 >> 172.21.15.98:0/3687057404 conn(0x7f1e9c0ac730 msgr2=0x7f1e9c0a7010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 shutdown_connections 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.911+0000 7f1ea3b33640 1 -- 172.21.15.98:0/3687057404 wait complete. 2024-09-11T02:56:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:56.915+0000 7f1ea3b33640 1 librados: shutdown 2024-09-11T02:56:56.964 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:56:56.964 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-09-11T02:56:56.964 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:56:57.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient: get_monmap_and_config 2024-09-11T02:56:57.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient: monmap: 2024-09-11T02:56:57.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:57.073940+0000 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:57.073940+0000 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:57.073 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:57.074 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 1 Processor -- start 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 1 -- start start 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient: init 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient: _add_conns ranks=[2,7,0,5,10,9,3,8,1,6,4] 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f506c139b10 0x7f506c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient(hunting): picked mon.noname-b con 0x7f506c139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f506c139ef0 0x7f506c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient(hunting): picked mon.noname-i con 0x7f506c139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:57.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f506c13a2d0 addr v1:172.21.15.98:6789/0 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f506c0702b0 con 0x7f506c13a2d0 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f506c071bf0 con 0x7f506c139b10 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f506c071350 con 0x7f506c139ef0 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 10 monclient(hunting): authenticate will time out at 1370.954207s 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f506c139b10 msgr2=0x7f506c072600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f506c139b10 0x7f506c072600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5071ae5640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f506c13a2d0 0x7f506c0a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39018/0 (socket says 172.21.15.98:39018) 2024-09-11T02:56:57.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5071ae5640 1 -- 172.21.15.98:0/3726833097 learned_addr learned my addr 172.21.15.98:0/3726833097 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f506c139ef0 0x7f506c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2541133828 0 0) 0x7f50640014a0 con 0x7f506c13a2d0 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): _negotiate 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): my global_id is 44352 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5058003530 con 0x7f506c13a2d0 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): get_auth_request con 0x7f506c139ef0 auth_method 0 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4133321703 0 0) 0x7f5058003530 con 0x7f506c13a2d0 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f506c139ef0 msgr2=0x7f506c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f506c139ef0 0x7f506c0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f506c139b10 msgr2=0x7f506c072600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:57.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f506c139b10 0x7f506c072600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:56:57.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:56:57.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506c0a9620 con 0x7f506c13a2d0 2024-09-11T02:56:57.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f5064001ec0 con 0x7f506c13a2d0 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: dump: 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.093 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 >> v1:172.21.15.98:6789/0 conn(0x7f506c13a2d0 legacy=0x7f506c0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient: _add_conns ranks=[6,5,3,8,4,2,7,1,0] 2024-09-11T02:56:57.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5058004240 0x7f5058004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): picked mon.d con 0x7f5058004240 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5058004da0 0x7f5058005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): picked mon.h con 0x7f5058004da0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5058009c60 0x7f505800c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): picked mon.g con 0x7f5058009c60 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f506c071bf0 con 0x7f5058009c60 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f506c071350 con 0x7f5058004da0 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f505800ca00 con 0x7f5058004240 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5063fff640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5058004240 0x7f5058004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): get_auth_request con 0x7f5058004240 auth_method 0 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5071ae5640 1 -- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5058009c60 msgr2=0x7f505800c0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5071ae5640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5058009c60 0x7f505800c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5058004da0 0x7f5058005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): get_auth_request con 0x7f5058004da0 auth_method 0 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5070ae3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient(hunting): handle_auth_done global_id 44352 payload 293 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 -- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5058004da0 msgr2=0x7f5058005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5058004da0 0x7f5058005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 -- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5058009c60 msgr2=0x7f505800c0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5058009c60 0x7f505800c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 monclient: found mon.d 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5071ae5640 1 -- 172.21.15.98:0/3726833097 reap_dead start 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 1 -- 172.21.15.98:0/3726833097 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5058003530 con 0x7f5058004240 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f50712e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 5 monclient: authenticate success, global_id 44352 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.071+0000 7f5073548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f50712e4640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5058004240 0x7f5058004810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f505c003990 tx=0x7f505c01c420 comp rx=0 tx=0).ready entity=mon.6 client_cookie=15302ae0c5dddf6d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f505c009520 con 0x7f5058004240 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 10 monclient: dump: 2024-09-11T02:56:57.098 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.099 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f505c003e30 con 0x7f5058004240 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 4 set_mon_vals no callback set 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 1 -- 172.21.15.98:0/3726833097 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f505c0054e0 con 0x7f5058004240 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5063fff640 10 monclient: dump: 2024-09-11T02:56:57.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient: shutdown 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5058004240 msgr2=0x7f5058004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5058004240 0x7f5058004810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f505c003990 tx=0x7f505c01c420 comp rx=0 tx=0).stop 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- 172.21.15.98:0/3726833097 shutdown_connections 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 --2- 172.21.15.98:0/3726833097 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5058004240 0x7f5058004810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- 172.21.15.98:0/3726833097 >> 172.21.15.98:0/3726833097 conn(0x7f506c0ac790 msgr2=0x7f506c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- 172.21.15.98:0/3726833097 shutdown_connections 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- 172.21.15.98:0/3726833097 wait complete. 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient: monmap: 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.103 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 librados: starting msgr at 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 librados: starting objecter 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 Processor -- start 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- start start 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 librados: setting wanted keys 2024-09-11T02:56:57.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 librados: calling monclient init 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient: init 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient: _add_conns ranks=[1,0,8,3,5,7,6,4,2] 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5058004240 0x7f506c1d1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): picked mon.noname-f con 0x7f5058004240 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f506c1d1f70 0x7f506c1d2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): picked mon.noname-a con 0x7f506c1d1f70 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c1270d0 0x7f506c127530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): picked mon.noname-e con 0x7f506c1270d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f506c0a85f0 con 0x7f506c1d1f70 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f506c0a8470 con 0x7f5058004240 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f506c0a8770 con 0x7f506c1270d0 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.075+0000 7f5073548640 10 monclient(hunting): authenticate will time out at 1370.959956s 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f506c1d1f70 0x7f506c1d2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f506c1d1f70 0x7f506c1d2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46594/0 (socket says 172.21.15.98:46594) 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 -- 172.21.15.98:0/7252551 learned_addr learned my addr 172.21.15.98:0/7252551 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): get_auth_request con 0x7f506c1d1f70 auth_method 0 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c1270d0 0x7f506c127530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5058004240 0x7f506c1d1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient(hunting): get_auth_request con 0x7f506c1270d0 auth_method 0 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): get_auth_request con 0x7f5058004240 auth_method 0 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient(hunting): handle_auth_done global_id 44361 payload 824 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c1270d0 msgr2=0x7f506c127530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c1270d0 0x7f506c127530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5058004240 msgr2=0x7f506c1d1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5058004240 0x7f506c1d1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 monclient: found mon.noname-a 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 -- 172.21.15.98:0/7252551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506c12b230 con 0x7f506c1d1f70 2024-09-11T02:56:57.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5070ae3640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f506c1d1f70 0x7f506c1d2390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5068011660 tx=0x7f5068011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 client.?.objecter ms_handle_connect 0x7f506c1d1f70 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f506800fc30 con 0x7f506c1d1f70 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: dump: 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.109 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f506800fdd0 con 0x7f506c1d1f70 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 5 monclient: authenticate success, global_id 44361 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 10 monclient: _renew_subs 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506c12ad80 con 0x7f506c1d1f70 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50680158d0 con 0x7f506c1d1f70 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: dump: 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.111 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50637fe640 4 set_mon_vals no callback set 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 10 client.44361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 10 monclient: _renew_subs 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f506c12a7c0 con 0x7f506c1d1f70 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 1 librados: init done 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c127a70 0x7f506c122650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f506c0a75a0 con 0x7f506c127a70 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c127a70 0x7f506c122650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient: get_auth_request con 0x7f506c127a70 auth_method 0 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5068015460 con 0x7f506c1d1f70 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 10 monclient(hunting): handle_auth_done global_id 24389 payload 824 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 1 -- 172.21.15.98:0/7252551 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f506c0a5ca0 con 0x7f506c127a70 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f505003df70 0x7f5050040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(301..301 src has 1..301) ==== 6541+0+0 (secure 0 0 0) 0x7f5068050010 con 0x7f506c1d1f70 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 client.44361.objecter ms_dispatch 0x7f506c000ba0 osd_map(301..301 src has 1..301) 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 3 client.44361.objecter handle_osd_map got epochs [301,301] > 0 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f50712e4640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c127a70 0x7f506c122650 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c0191c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 3 client.44361.objecter handle_osd_map decoding full epoch 301 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 20 client.44361.objecter dump_active .. 0 homeless 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 client.44361.objecter ms_handle_connect 0x7f506c127a70 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 client.44361.objecter resend_mon_ops 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f505c0bc070 con 0x7f506c127a70 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: dump: 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.115 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f505003df70 0x7f5050040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 10 monclient: get_auth_request con 0x7f505003df70 auth_method 0 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f505c009500 con 0x7f506c127a70 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c127a70 msgr2=0x7f506c122650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c127a70 0x7f506c122650 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c0191c0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c127a70 msgr2=0x7f506c122650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c127a70 0x7f506c122650 secure :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f505c006fd0 tx=0x7f505c0191c0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5061ffb640 10 client.44361.objecter ms_handle_connect 0x7f505003df70 2024-09-11T02:56:57.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.079+0000 7f5071ae5640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f505003df70 0x7f5050040430 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7f506c071520 tx=0x7f5064001600 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c122b90 0x7f506c122fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f506c06ca10 con 0x7f506c122b90 2024-09-11T02:56:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c122b90 0x7f506c122fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient: get_auth_request con 0x7f506c122b90 auth_method 0 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 10 monclient(hunting): handle_auth_done global_id 24398 payload 824 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 1 -- 172.21.15.98:0/7252551 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f506c06e680 con 0x7f506c122b90 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 10 client.44361.objecter ms_handle_connect 0x7f506c122b90 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 10 client.44361.objecter resend_mon_ops 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f50712e4640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c122b90 0x7f506c122fa0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f505c01c460 tx=0x7f5058005840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f505c0bc050 con 0x7f506c122b90 2024-09-11T02:56:57.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 10 monclient: dump: 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9247 (secure 0 0 0) 0x7f505c0bd040 con 0x7f506c122b90 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c122b90 msgr2=0x7f506c122fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c122b90 0x7f506c122fa0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f505c01c460 tx=0x7f5058005840 comp rx=0 tx=0).stop 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c122b90 msgr2=0x7f506c122fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5061ffb640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f506c122b90 0x7f506c122fa0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 62, 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 432880, 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:57.123 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.124 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:57.126 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:57.127 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.128 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.132 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.137 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.139 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:57.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.144 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:57.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:57.146 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 10 librados: watch_flush enter 2024-09-11T02:56:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 10 librados: watch_flush exit 2024-09-11T02:56:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 20 client.44361.objecter shutdown clearing up homeless session... 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 10 client.44361.objecter successfully canceled tick 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f505003df70 msgr2=0x7f5050040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f505003df70 0x7f5050040430 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7f506c071520 tx=0x7f5064001600 comp rx=0 tx=0).stop 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 10 monclient: shutdown 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f506c1d1f70 msgr2=0x7f506c1d2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5073548640 1 --2- 172.21.15.98:0/7252551 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f506c1d1f70 0x7f506c1d2390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5068011660 tx=0x7f5068011b20 comp rx=0 tx=0).stop 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5071ae5640 1 -- 172.21.15.98:0/7252551 reap_dead start 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.087+0000 7f5071ae5640 1 -- 172.21.15.98:0/7252551 reap_dead start 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.091+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 shutdown_connections 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.091+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 >> 172.21.15.98:0/7252551 conn(0x7f506c0ac790 msgr2=0x7f506c0a3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.091+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 shutdown_connections 2024-09-11T02:56:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.091+0000 7f5073548640 1 -- 172.21.15.98:0/7252551 wait complete. 2024-09-11T02:56:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.091+0000 7f5073548640 1 librados: shutdown 2024-09-11T02:56:57.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:56:57.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-09-11T02:56:57.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:56:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient: get_monmap_and_config 2024-09-11T02:56:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient: monmap: 2024-09-11T02:56:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.264 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:57.265942+0000 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:57.265942+0000 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.265 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:57.266 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:57.266 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 Processor -- start 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 -- start start 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient: init 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient: _add_conns ranks=[6,3,8,4,7,5,2,1,10,0,9] 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdef8139b80 0x7fdef8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): picked mon.noname-d con 0x7fdef8139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdef806fe50 0x7fdef80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): picked mon.noname-g con 0x7fdef806fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:56:57.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8072640 0x7fdef80a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): picked mon.noname-e con 0x7fdef8072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdef80719c0 con 0x7fdef806fe50 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdef8071c10 con 0x7fdef8139b80 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdef80702d0 con 0x7fdef8072640 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdefd7da640 10 monclient(hunting): authenticate will time out at 1371.146398s 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8072640 0x7fdef80a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef67fc640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdef806fe50 msgr2=0x7fdef80a7a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8072640 0x7fdef80a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34386/0 (socket says 172.21.15.98:34386) 2024-09-11T02:56:57.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 1 -- 172.21.15.98:0/3177883304 learned_addr learned my addr 172.21.15.98:0/3177883304 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 10 monclient(hunting): get_auth_request con 0x7fdef8072640 auth_method 0 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef67fc640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdef806fe50 0x7fdef80a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdef8139b80 0x7fdef8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): get_auth_request con 0x7fdef8139b80 auth_method 0 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_done global_id 44277 payload 274 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 1 -- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8072640 msgr2=0x7fdef80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8072640 0x7fdef80a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 1 -- 172.21.15.98:0/3177883304 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdef806fe50 msgr2=0x7fdef80a7a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdef806fe50 0x7fdef80a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 1 monclient: found mon.noname-d 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef6ffd640 1 -- 172.21.15.98:0/3177883304 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef8071370 con 0x7fdef8139b80 2024-09-11T02:56:57.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.263+0000 7fdef77fe640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8072640 0x7fdef80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:57.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdef8139b80 0x7fdef8139f60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdee80097f0 tx=0x7fdee801c3c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ccf64a9947613df4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 1 -- 172.21.15.98:0/3177883304 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdee80b0070 con 0x7fdef8139b80 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: dump: 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.279 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 1 -- 172.21.15.98:0/3177883304 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdee80aa810 con 0x7fdef8139b80 2024-09-11T02:56:57.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 5 monclient: authenticate success, global_id 44277 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 1 -- 172.21.15.98:0/3177883304 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdee80a9300 con 0x7fdef8139b80 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 4 set_mon_vals no callback set 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef5ffb640 10 monclient: dump: 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient: shutdown 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdef8139b80 msgr2=0x7fdef8139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdef8139b80 0x7fdef8139f60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdee80097f0 tx=0x7fdee801c3c0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- 172.21.15.98:0/3177883304 shutdown_connections 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8072640 0x7fdef80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdef806fe50 0x7fdef80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 --2- 172.21.15.98:0/3177883304 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdef8139b80 0x7fdef8139f60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- 172.21.15.98:0/3177883304 >> 172.21.15.98:0/3177883304 conn(0x7fdef80ac730 msgr2=0x7fdef80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- 172.21.15.98:0/3177883304 shutdown_connections 2024-09-11T02:56:57.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- 172.21.15.98:0/3177883304 wait complete. 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient: monmap: 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 librados: starting msgr at 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 librados: starting objecter 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 Processor -- start 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- start start 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 librados: setting wanted keys 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 librados: calling monclient init 2024-09-11T02:56:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient: init 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient: _add_conns ranks=[1,2,8,0,4,7,5,6,3] 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef806fe50 0x7fdef8134a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): picked mon.noname-f con 0x7fdef806fe50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdef8072640 0x7fdef8134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): picked mon.noname-b con 0x7fdef8072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8139b80 0x7fdef813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): picked mon.noname-e con 0x7fdef8139b80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdef81435c0 con 0x7fdef806fe50 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fdef8143440 con 0x7fdef8072640 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdef8143740 con 0x7fdef8139b80 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdefd7da640 10 monclient(hunting): authenticate will time out at 1371.151341s 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef67fc640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdef8072640 msgr2=0x7fdef8134fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdef8072640 0x7fdef8134fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8139b80 0x7fdef813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8139b80 0x7fdef813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34390/0 (socket says 172.21.15.98:34390) 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 -- 172.21.15.98:0/1562190350 learned_addr learned my addr 172.21.15.98:0/1562190350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): get_auth_request con 0x7fdef8139b80 auth_method 0 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient(hunting): handle_auth_done global_id 24416 payload 824 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdef8072640 msgr2=0x7fdef8134fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fdef8072640 0x7fdef8134fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef806fe50 msgr2=0x7fdef8134a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef806fe50 0x7fdef8134a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 monclient: found mon.noname-e 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 -- 172.21.15.98:0/1562190350 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef813dd20 con 0x7fdef8139b80 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdef77fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8139b80 0x7fdef813a6b0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fdef000efc0 tx=0x7fdef000c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdedf7fe640 10 client.?.objecter ms_handle_connect 0x7fdef8139b80 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.267+0000 7fdedf7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdef000cbc0 con 0x7fdef8139b80 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: dump: 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.289 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef0019070 con 0x7fdef8139b80 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 5 monclient: authenticate success, global_id 24416 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 10 monclient: _renew_subs 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef8135810 con 0x7fdef8139b80 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdef000cd80 con 0x7fdef8139b80 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 10 client.24416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef57fa640 4 set_mon_vals no callback set 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: dump: 2024-09-11T02:56:57.291 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.292 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 10 monclient: _renew_subs 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef80aa350 con 0x7fdef8139b80 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 1 librados: init done 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813c660 0x7fdef813abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdef806ca10 con 0x7fdef813c660 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fdef0014480 con 0x7fdef8139b80 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813c660 0x7fdef813abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient: get_auth_request con 0x7fdef813c660 auth_method 0 2024-09-11T02:56:57.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdee403df40 0x7fdee4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(302..302 src has 1..302) ==== 6642+0+0 (secure 0 0 0) 0x7fdef0050150 con 0x7fdef8139b80 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 client.24416.objecter ms_dispatch 0x7fdef8000ba0 osd_map(302..302 src has 1..302) 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 3 client.24416.objecter handle_osd_map got epochs [302,302] > 0 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 3 client.24416.objecter handle_osd_map decoding full epoch 302 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 20 client.24416.objecter dump_active .. 0 homeless 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef67fc640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdee403df40 0x7fdee4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef67fc640 10 monclient: get_auth_request con 0x7fdee403df40 auth_method 0 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_done global_id 25264 payload 824 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 1 -- 172.21.15.98:0/1562190350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdef8139f60 con 0x7fdef813c660 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 client.24416.objecter ms_handle_connect 0x7fdef813c660 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813c660 0x7fdef813abf0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fdee80ad000 tx=0x7fdee8006ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 client.24416.objecter resend_mon_ops 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdee80b0040 con 0x7fdef813c660 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: dump: 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.295 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.296 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 client.24416.objecter ms_handle_connect 0x7fdee403df40 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdef67fc640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdee403df40 0x7fdee4040400 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7fdeec006fd0 tx=0x7fdeec006d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdee8006de0 con 0x7fdef813c660 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813c660 msgr2=0x7fdef813abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813c660 0x7fdef813abf0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fdee80ad000 tx=0x7fdee8006ca0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813c660 msgr2=0x7fdef813abf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.271+0000 7fdedf7fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813c660 0x7fdef813abf0 secure :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7fdee80ad000 tx=0x7fdee8006ca0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdefd7da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdefd7da640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813b130 0x7fdef813b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdefd7da640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdef806e870 con 0x7fdef813b130 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813b130 0x7fdef813b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient: get_auth_request con 0x7fdef813b130 auth_method 0 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 10 monclient(hunting): handle_auth_done global_id 25273 payload 824 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 1 -- 172.21.15.98:0/1562190350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdef813ba80 con 0x7fdef813b130 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdef6ffd640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813b130 0x7fdef813b540 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fdee801c400 tx=0x7fdee80a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdedf7fe640 10 client.24416.objecter ms_handle_connect 0x7fdef813b130 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdedf7fe640 10 client.24416.objecter resend_mon_ops 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdee80b0050 con 0x7fdef813b130 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdedf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdedf7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.279+0000 7fdedf7fe640 10 monclient: dump: 2024-09-11T02:56:57.300 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.301 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7fdee80bda40 con 0x7fdef813b130 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdedf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdedf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813b130 msgr2=0x7fdef813b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdedf7fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813b130 0x7fdef813b540 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7fdee801c400 tx=0x7fdee80a8000 comp rx=0 tx=0).stop 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdedf7fe640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813b130 msgr2=0x7fdef813b540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdedf7fe640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef813b130 0x7fdef813b540 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.302 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 62, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 433049, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:57.303 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.304 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.305 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.306 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.307 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.308 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:57.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.313 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:57.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.318 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:57.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.324 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.325 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.326 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.326 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:57.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.326 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:57.326 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 10 librados: watch_flush enter 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 10 librados: watch_flush exit 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 20 client.24416.objecter shutdown clearing up homeless session... 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 10 client.24416.objecter successfully canceled tick 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdee403df40 msgr2=0x7fdee4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdee403df40 0x7fdee4040400 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7fdeec006fd0 tx=0x7fdeec006d40 comp rx=0 tx=0).stop 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 10 monclient: shutdown 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8139b80 msgr2=0x7fdef813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 --2- 172.21.15.98:0/1562190350 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdef8139b80 0x7fdef813a6b0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fdef000efc0 tx=0x7fdef000c490 comp rx=0 tx=0).stop 2024-09-11T02:56:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdef77fe640 1 -- 172.21.15.98:0/1562190350 reap_dead start 2024-09-11T02:56:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdef77fe640 1 -- 172.21.15.98:0/1562190350 reap_dead start 2024-09-11T02:56:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 shutdown_connections 2024-09-11T02:56:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 >> 172.21.15.98:0/1562190350 conn(0x7fdef80ac730 msgr2=0x7fdef8135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 shutdown_connections 2024-09-11T02:56:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 -- 172.21.15.98:0/1562190350 wait complete. 2024-09-11T02:56:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.283+0000 7fdefd7da640 1 librados: shutdown 2024-09-11T02:56:57.329 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:56:57.329 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-09-11T02:56:57.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:56:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient: get_monmap_and_config 2024-09-11T02:56:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient: monmap: 2024-09-11T02:56:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:57.445625+0000 2024-09-11T02:56:57.444 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:57.445625+0000 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:57.445 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:57.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 Processor -- start 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 -- start start 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient: init 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient: _add_conns ranks=[5,8,7,9,0,4,10,6,3,1,2] 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc139b10 0x7f00dc072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): picked mon.noname-h con 0x7f00dc139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f00dc139ef0 0x7f00dc0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): picked mon.noname-e con 0x7f00dc139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f00dc13a2d0 0x7f00dc0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): picked mon.noname-i con 0x7f00dc13a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f00dc0702b0 con 0x7f00dc139b10 2024-09-11T02:56:57.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f00dc071bf0 con 0x7f00dc13a2d0 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f00dc071350 con 0x7f00dc139ef0 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e292b640 10 monclient(hunting): authenticate will time out at 1371.325920s 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f00dc139ef0 0x7f00dc0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f00dc139ef0 0x7f00dc0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34404/0 (socket says 172.21.15.98:34404) 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 -- 172.21.15.98:0/2838856121 learned_addr learned my addr 172.21.15.98:0/2838856121 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): get_auth_request con 0x7f00dc139ef0 auth_method 0 2024-09-11T02:56:57.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e0ec8640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f00dc13a2d0 0x7f00dc0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00dbfff640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc139b10 0x7f00dc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e0ec8640 10 monclient(hunting): get_auth_request con 0x7f00dc13a2d0 auth_method 0 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e0ec8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e0ec8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00e0ec8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00dbfff640 10 monclient(hunting): get_auth_request con 0x7f00dc139b10 auth_method 0 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient(hunting): handle_auth_done global_id 24425 payload 274 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 -- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f00dc13a2d0 msgr2=0x7f00dc0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f00dc13a2d0 0x7f00dc0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 -- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc139b10 msgr2=0x7f00dc072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc139b10 0x7f00dc072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 monclient: found mon.noname-e 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.443+0000 7f00db7fe640 1 -- 172.21.15.98:0/2838856121 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc0a9620 con 0x7f00dc139ef0 2024-09-11T02:56:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00db7fe640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f00dc139ef0 0x7f00dc0a7a10 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f00c40098a0 tx=0x7f00c401c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=7a7b3ad076d08d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 1 -- 172.21.15.98:0/2838856121 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00c40b0070 con 0x7f00dc139ef0 2024-09-11T02:56:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:57.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: dump: 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.460 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 1 -- 172.21.15.98:0/2838856121 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c40aa810 con 0x7f00dc139ef0 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 5 monclient: authenticate success, global_id 24425 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 1 -- 172.21.15.98:0/2838856121 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00c40a9300 con 0x7f00dc139ef0 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00daffd640 10 monclient: dump: 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 4 set_mon_vals no callback set 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient: shutdown 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- 172.21.15.98:0/2838856121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f00dc139ef0 msgr2=0x7f00dc0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f00dc139ef0 0x7f00dc0a7a10 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f00c40098a0 tx=0x7f00c401c440 comp rx=0 tx=0).stop 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- 172.21.15.98:0/2838856121 shutdown_connections 2024-09-11T02:56:57.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f00dc13a2d0 0x7f00dc0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f00dc139ef0 0x7f00dc0a7a10 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 --2- 172.21.15.98:0/2838856121 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc139b10 0x7f00dc072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- 172.21.15.98:0/2838856121 >> 172.21.15.98:0/2838856121 conn(0x7f00dc0ac790 msgr2=0x7f00dc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- 172.21.15.98:0/2838856121 shutdown_connections 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- 172.21.15.98:0/2838856121 wait complete. 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient: monmap: 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.464 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 librados: starting msgr at 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 librados: starting objecter 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 Processor -- start 2024-09-11T02:56:57.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- start start 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 librados: setting wanted keys 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 librados: calling monclient init 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient: init 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient: _add_conns ranks=[4,2,0,8,7,6,1,5,3] 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f00dc139b10 0x7f00dc13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): picked mon.noname-c con 0x7f00dc139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00dc139ef0 0x7f00dc13ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): picked mon.noname-b con 0x7f00dc139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00dc13a2d0 0x7f00dc13b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): picked mon.noname-a con 0x7f00dc13a2d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc143460 con 0x7f00dc13a2d0 2024-09-11T02:56:57.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f00dc0a88a0 con 0x7f00dc139ef0 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f00dc1435e0 con 0x7f00dc139b10 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e292b640 10 monclient(hunting): authenticate will time out at 1371.331631s 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00db7fe640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00dc139ef0 msgr2=0x7f00dc13ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00db7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00dc139ef0 0x7f00dc13ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f00dc139b10 0x7f00dc13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00dc13a2d0 0x7f00dc13b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f00dc139b10 0x7f00dc13a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:43728/0 (socket says 172.21.15.98:43728) 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 1 -- 172.21.15.98:0/4093712363 learned_addr learned my addr 172.21.15.98:0/4093712363 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:57.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 10 monclient(hunting): get_auth_request con 0x7f00dc139b10 auth_method 0 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 10 monclient(hunting): get_auth_request con 0x7f00dc13a2d0 auth_method 0 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00e0ec8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.447+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 10 monclient(hunting): handle_auth_done global_id 44370 payload 824 2024-09-11T02:56:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f00dc139b10 msgr2=0x7f00dc13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f00dc139b10 0x7f00dc13a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00dc139ef0 msgr2=0x7f00dc13ad10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:57.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f00dc139ef0 0x7f00dc13ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 1 monclient: found mon.noname-a 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 1 -- 172.21.15.98:0/4093712363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc0a59d0 con 0x7f00dc13a2d0 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f00dc139b10 0x7f00dc13a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e0ec8640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00dc13a2d0 0x7f00dc13b250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00d4b0 tx=0x7f00cc00d970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 client.?.objecter ms_handle_connect 0x7f00dc13a2d0 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00cc012a60 con 0x7f00dc13a2d0 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:56:57.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: dump: 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.471 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00cc012c60 con 0x7f00dc13a2d0 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 5 monclient: authenticate success, global_id 44370 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 10 monclient: _renew_subs 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:57.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00dc0a9f20 con 0x7f00dc13a2d0 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00cc012e10 con 0x7f00dc13a2d0 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00da7fc640 4 set_mon_vals no callback set 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 10 client.44370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: dump: 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.473 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 10 monclient: _renew_subs 2024-09-11T02:56:57.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00dc13de80 con 0x7f00dc13a2d0 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 1 librados: init done 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc1337f0 0x7f00dc133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f00dc06ca10 con 0x7f00dc1337f0 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f00cc013070 con 0x7f00dc13a2d0 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc1337f0 0x7f00dc133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f00bc03df40 0x7f00bc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient: get_auth_request con 0x7f00dc1337f0 auth_method 0 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(302..302 src has 1..302) ==== 6642+0+0 (secure 0 0 0) 0x7f00cc019020 con 0x7f00dc13a2d0 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 client.44370.objecter ms_dispatch 0x7f00dc000ba0 osd_map(302..302 src has 1..302) 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 3 client.44370.objecter handle_osd_map got epochs [302,302] > 0 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 3 client.44370.objecter handle_osd_map decoding full epoch 302 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 20 client.44370.objecter dump_active .. 0 homeless 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00db7fe640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f00bc03df40 0x7f00bc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00db7fe640 10 monclient: get_auth_request con 0x7f00bc03df40 auth_method 0 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 10 monclient(hunting): handle_auth_done global_id 20003 payload 824 2024-09-11T02:56:57.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 1 -- 172.21.15.98:0/4093712363 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00dc06e680 con 0x7f00dc1337f0 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 client.44370.objecter ms_handle_connect 0x7f00dc1337f0 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00dbfff640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc1337f0 0x7f00dc133bd0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f00d4007590 tx=0x7f00d4007a50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 client.44370.objecter resend_mon_ops 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 client.44370.objecter ms_handle_connect 0x7f00bc03df40 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00db7fe640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f00bc03df40 0x7f00bc040400 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f00c40ad000 tx=0x7f00c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00d4015a60 con 0x7f00dc1337f0 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: dump: 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.477 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f00d401b070 con 0x7f00dc1337f0 2024-09-11T02:56:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc1337f0 msgr2=0x7f00dc133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc1337f0 0x7f00dc133bd0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f00d4007590 tx=0x7f00d4007a50 comp rx=0 tx=0).stop 2024-09-11T02:56:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc1337f0 msgr2=0x7f00dc133bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.451+0000 7f00d8ff9640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc1337f0 0x7f00dc133bd0 secure :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f00d4007590 tx=0x7f00d4007a50 comp rx=0 tx=0).stop 2024-09-11T02:56:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00e292b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:57.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00e292b640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc134110 0x7f00dc137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00e292b640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f00dc06e870 con 0x7f00dc134110 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc134110 0x7f00dc137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient: get_auth_request con 0x7f00dc134110 auth_method 0 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 10 monclient(hunting): handle_auth_done global_id 20012 payload 824 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 1 -- 172.21.15.98:0/4093712363 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f00dc072600 con 0x7f00dc134110 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00dbfff640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc134110 0x7f00dc137bc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f00d401a000 tx=0x7f00d4004a50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00d8ff9640 10 client.44370.objecter ms_handle_connect 0x7f00dc134110 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00d8ff9640 10 client.44370.objecter resend_mon_ops 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00d4015e10 con 0x7f00dc134110 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00d8ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.459+0000 7f00d8ff9640 10 monclient: dump: 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.482 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7f00d4015e10 con 0x7f00dc134110 2024-09-11T02:56:57.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00d8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00d8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc134110 msgr2=0x7f00dc137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00d8ff9640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc134110 0x7f00dc137bc0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f00d401a000 tx=0x7f00d4004a50 comp rx=0 tx=0).stop 2024-09-11T02:56:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00d8ff9640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc134110 msgr2=0x7f00dc137bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00d8ff9640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f00dc134110 0x7f00dc137bc0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 62, 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 433209, 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:57.485 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:57.486 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.487 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.488 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.489 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:57.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.492 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:57.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:57.492 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:57.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:57.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:57.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:57.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:57.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:57.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.499 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.499 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.500 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:57.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:57.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:57.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:57.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.503 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.503 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.503 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:57.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:57.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:57.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:57.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.507 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.508 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.508 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.508 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:57.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:57.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:57.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:57.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.511 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.511 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.512 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.512 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.512 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:57.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:57.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:57.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:57.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.515 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.516 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.517 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:57.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:57.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:57.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:57.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.520 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.521 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:57.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:57.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:57.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:57.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.524 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:57.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:57.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.525 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.526 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:56:57.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.528 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.529 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:56:57.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.529 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:57.529 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 10 librados: watch_flush enter 2024-09-11T02:56:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 10 librados: watch_flush exit 2024-09-11T02:56:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 20 client.44370.objecter shutdown clearing up homeless session... 2024-09-11T02:56:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 10 client.44370.objecter successfully canceled tick 2024-09-11T02:56:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f00bc03df40 msgr2=0x7f00bc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f00bc03df40 0x7f00bc040400 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f00c40ad000 tx=0x7f00c4008040 comp rx=0 tx=0).stop 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 10 monclient: shutdown 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00dc13a2d0 msgr2=0x7f00dc13b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00dc13a2d0 0x7f00dc13b250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f00cc00d4b0 tx=0x7f00cc00d970 comp rx=0 tx=0).stop 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e0ec8640 1 -- 172.21.15.98:0/4093712363 reap_dead start 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 shutdown_connections 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 --2- 172.21.15.98:0/4093712363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f00dc13a2d0 0x7f00dc13b250 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 >> 172.21.15.98:0/4093712363 conn(0x7f00dc0ac790 msgr2=0x7f00dc0a78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 shutdown_connections 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 -- 172.21.15.98:0/4093712363 wait complete. 2024-09-11T02:56:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.463+0000 7f00e292b640 1 librados: shutdown 2024-09-11T02:56:57.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:56:57.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-09-11T02:56:57.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:56:57.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient: get_monmap_and_config 2024-09-11T02:56:57.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient: monmap: 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:56:57.649952+0000 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:56:57.649952+0000 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.649 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:56:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:56:57.650 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 Processor -- start 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 -- start start 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient: init 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient: _add_conns ranks=[4,8,5,3,6,2,9,0,1,10,7] 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb544139b10 0x7fb544072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): picked mon.noname-c con 0x7fb544139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139ef0 0x7fb5440a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): picked mon.noname-e con 0x7fb544139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:57.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb54413a2d0 0x7fb5440a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): picked mon.noname-h con 0x7fb54413a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb5440702b0 con 0x7fb544139b10 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb544071bf0 con 0x7fb54413a2d0 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb544071350 con 0x7fb544139ef0 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb54b4f5640 10 monclient(hunting): authenticate will time out at 1371.530383s 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139ef0 0x7fb5440a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139ef0 0x7fb5440a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34406/0 (socket says 172.21.15.98:34406) 2024-09-11T02:56:57.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 -- 172.21.15.98:0/1196225055 learned_addr learned my addr 172.21.15.98:0/1196225055 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb544139b10 0x7fb544072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): get_auth_request con 0x7fb544139ef0 auth_method 0 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 10 monclient(hunting): get_auth_request con 0x7fb544139b10 auth_method 0 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549291640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549a92640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb54413a2d0 0x7fb5440a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549a92640 10 monclient(hunting): get_auth_request con 0x7fb54413a2d0 auth_method 0 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549a92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549a92640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb549a92640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient(hunting): handle_auth_done global_id 24434 payload 274 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 -- 172.21.15.98:0/1196225055 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb54413a2d0 msgr2=0x7fb5440a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb54413a2d0 0x7fb5440a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 -- 172.21.15.98:0/1196225055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb544139b10 msgr2=0x7fb544072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb544139b10 0x7fb544072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 monclient: found mon.noname-e 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.647+0000 7fb548a90640 1 -- 172.21.15.98:0/1196225055 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5440a9620 con 0x7fb544139ef0 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb548a90640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139ef0 0x7fb5440a7a10 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fb5340098a0 tx=0x7fb53401c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=6ba4162714c612ee server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 1 -- 172.21.15.98:0/1196225055 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5340b0070 con 0x7fb544139ef0 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: dump: 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.664 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.665 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 1 -- 172.21.15.98:0/1196225055 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5340aa810 con 0x7fb544139ef0 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 5 monclient: authenticate success, global_id 24434 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 1 -- 172.21.15.98:0/1196225055 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5340a9300 con 0x7fb544139ef0 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient: get_monmap_and_config success 2024-09-11T02:56:57.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 4 set_mon_vals no callback set 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb53bfff640 10 monclient: dump: 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient: shutdown 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- 172.21.15.98:0/1196225055 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139ef0 msgr2=0x7fb5440a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139ef0 0x7fb5440a7a10 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7fb5340098a0 tx=0x7fb53401c440 comp rx=0 tx=0).stop 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- 172.21.15.98:0/1196225055 shutdown_connections 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb54413a2d0 0x7fb5440a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139ef0 0x7fb5440a7a10 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/1196225055 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb544139b10 0x7fb544072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- 172.21.15.98:0/1196225055 >> 172.21.15.98:0/1196225055 conn(0x7fb5440ac790 msgr2=0x7fb5440a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- 172.21.15.98:0/1196225055 shutdown_connections 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- 172.21.15.98:0/1196225055 wait complete. 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient: build_initial_monmap 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient: monmap: 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:56:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 librados: starting msgr at 2024-09-11T02:56:57.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 librados: starting objecter 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 Processor -- start 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- start start 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 librados: setting wanted keys 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 librados: calling monclient init 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient: init 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient: _reopen_session rank -1 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient: _add_conns ranks=[8,2,7,0,5,6,4,3,1] 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139b10 0x7fb54413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): picked mon.noname-e con 0x7fb544139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb544139ef0 0x7fb54413ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): picked mon.noname-b con 0x7fb544139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb54413a2d0 0x7fb54413b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): picked mon.noname-i con 0x7fb54413a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:56:57.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb544143460 con 0x7fb544139ef0 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb5440a88a0 con 0x7fb54413a2d0 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb5441435e0 con 0x7fb544139b10 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): _renew_subs 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb54b4f5640 10 monclient(hunting): authenticate will time out at 1371.535401s 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb548a90640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb544139ef0 msgr2=0x7fb54413ad10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb548a90640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb544139ef0 0x7fb54413ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:56:57.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139b10 0x7fb54413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139b10 0x7fb54413a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:34412/0 (socket says 172.21.15.98:34412) 2024-09-11T02:56:57.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 -- 172.21.15.98:0/2744864545 learned_addr learned my addr 172.21.15.98:0/2744864545 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:56:57.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): get_auth_request con 0x7fb544139b10 auth_method 0 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient(hunting): handle_auth_done global_id 24443 payload 824 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb54413a2d0 msgr2=0x7fb54413b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb54413a2d0 0x7fb54413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb544139ef0 msgr2=0x7fb54413ad10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb544139ef0 0x7fb54413ad10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient: _finish_hunting 0 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 monclient: found mon.noname-e 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 -- 172.21.15.98:0/2744864545 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5440a59d0 con 0x7fb544139b10 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb549291640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139b10 0x7fb54413a7d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fb54000e960 tx=0x7fb54000ee20 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb539ffb640 10 client.?.objecter ms_handle_connect 0x7fb544139b10 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.651+0000 7fb539ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb540017070 con 0x7fb544139b10 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: dump: 2024-09-11T02:56:57.675 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.676 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: _finish_auth 0 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb54000ce70 con 0x7fb544139b10 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 5 monclient: authenticate success, global_id 24443 2024-09-11T02:56:57.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 10 monclient: _renew_subs 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5440a9f20 con 0x7fb544139b10 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5400138e0 con 0x7fb544139b10 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 10 client.24443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: dump: 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.678 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb53b7fe640 4 set_mon_vals no callback set 2024-09-11T02:56:57.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 10 monclient: _renew_subs 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54413de80 con 0x7fb544139b10 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 1 librados: init done 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441337f0 0x7fb544133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb54406ca10 con 0x7fb5441337f0 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb540013470 con 0x7fb544139b10 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441337f0 0x7fb544133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient: get_auth_request con 0x7fb5441337f0 auth_method 0 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52803df40 0x7fb528040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(302..302 src has 1..302) ==== 6642+0+0 (secure 0 0 0) 0x7fb54000f040 con 0x7fb544139b10 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 client.24443.objecter ms_dispatch 0x7fb544000ba0 osd_map(302..302 src has 1..302) 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 3 client.24443.objecter handle_osd_map got epochs [302,302] > 0 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 3 client.24443.objecter handle_osd_map decoding full epoch 302 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 20 client.24443.objecter dump_active .. 0 homeless 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb549a92640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52803df40 0x7fb528040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb549a92640 10 monclient: get_auth_request con 0x7fb52803df40 auth_method 0 2024-09-11T02:56:57.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 10 monclient(hunting): handle_auth_done global_id 19618 payload 824 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 1 -- 172.21.15.98:0/2744864545 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb54406e680 con 0x7fb5441337f0 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 client.24443.objecter ms_handle_connect 0x7fb5441337f0 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 client.24443.objecter resend_mon_ops 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb548a90640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441337f0 0x7fb544133bd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb534009870 tx=0x7fb53401ac80 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5340b0040 con 0x7fb5441337f0 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: dump: 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 client.24443.objecter ms_handle_connect 0x7fb52803df40 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb549a92640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52803df40 0x7fb528040400 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fb5440a57d0 tx=0x7fb53c009450 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5340a9300 con 0x7fb5441337f0 2024-09-11T02:56:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:56:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:56:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441337f0 msgr2=0x7fb544133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441337f0 0x7fb544133bd0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb534009870 tx=0x7fb53401ac80 comp rx=0 tx=0).stop 2024-09-11T02:56:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441337f0 msgr2=0x7fb544133bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.655+0000 7fb539ffb640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441337f0 0x7fb544133bd0 secure :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb534009870 tx=0x7fb53401ac80 comp rx=0 tx=0).stop 2024-09-11T02:56:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:56:57.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544134110 0x7fb544137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb54406e870 con 0x7fb544134110 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544134110 0x7fb544137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient: get_auth_request con 0x7fb544134110 auth_method 0 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 10 monclient(hunting): handle_auth_done global_id 19627 payload 824 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 1 -- 172.21.15.98:0/2744864545 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb544072600 con 0x7fb544134110 2024-09-11T02:56:57.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb548a90640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544134110 0x7fb544137bc0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fb5340097c0 tx=0x7fb5340191e0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 10 client.24443.objecter ms_handle_connect 0x7fb544134110 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 10 client.24443.objecter resend_mon_ops 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5340b0040 con 0x7fb544134110 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 10 monclient: dump: 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:56:57.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9245 (secure 0 0 0) 0x7fb5340bf980 con 0x7fb544134110 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:56:57.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544134110 msgr2=0x7fb544137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544134110 0x7fb544137bc0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fb5340097c0 tx=0x7fb5340191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544134110 msgr2=0x7fb544137bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb539ffb640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544134110 0x7fb544137bc0 secure :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fb5340097c0 tx=0x7fb5340191e0 comp rx=0 tx=0).stop 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 62, 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 433433, 2024-09-11T02:56:57.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:56:57.691 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:56:57.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:56:57.694 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:56:57.695 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.699 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:56:57.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:56:57.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:56:57.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:56:57.713 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:56:57.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 10 librados: watch_flush enter 2024-09-11T02:56:57.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 10 librados: watch_flush exit 2024-09-11T02:56:57.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 20 client.24443.objecter shutdown clearing up homeless session... 2024-09-11T02:56:57.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 10 client.24443.objecter successfully canceled tick 2024-09-11T02:56:57.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52803df40 msgr2=0x7fb528040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52803df40 0x7fb528040400 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fb5440a57d0 tx=0x7fb53c009450 comp rx=0 tx=0).stop 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 10 monclient: shutdown 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139b10 msgr2=0x7fb54413a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 --2- 172.21.15.98:0/2744864545 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb544139b10 0x7fb54413a7d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7fb54000e960 tx=0x7fb54000ee20 comp rx=0 tx=0).stop 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb549a92640 1 -- 172.21.15.98:0/2744864545 reap_dead start 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb549a92640 1 -- 172.21.15.98:0/2744864545 reap_dead start 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 shutdown_connections 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 >> 172.21.15.98:0/2744864545 conn(0x7fb5440ac790 msgr2=0x7fb5440a78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 shutdown_connections 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.663+0000 7fb54b4f5640 1 -- 172.21.15.98:0/2744864545 wait complete. 2024-09-11T02:56:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:56:57.667+0000 7fb54b4f5640 1 librados: shutdown 2024-09-11T02:56:57.717 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:56:57.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T02:57:09.328 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ P2 2024-09-11T02:57:09.328 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2994 ms) 2024-09-11T02:57:09.328 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-11T02:57:09.328 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (9974 ms) 2024-09-11T02:57:09.328 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-11T02:57:09.328 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7080 ms) 2024-09-11T02:57:09.328 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (16020 ms) 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (25702 ms) 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7229 ms) 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6982 ms) 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (197932 ms total) 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (368331 ms total) 2024-09-11T02:57:09.329 INFO:tasks.workunit.client.0.smithi190.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-11T02:58:27.718 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T02:58:27.719 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T02:58:27.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T02:58:27.723 INFO:tasks.ceph.mon.b:Started 2024-09-11T02:58:27.723 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-09-11T02:58:27.723 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-11T02:58:27.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-11T02:58:27.726 INFO:tasks.ceph.mon.g:Started 2024-09-11T02:58:27.726 INFO:tasks.mon_thrash:Sending STOP to mon f 2024-09-11T02:58:27.726 INFO:tasks.ceph.mon.f:Sent signal 19 2024-09-11T02:58:27.726 INFO:tasks.mon_thrash:Sending STOP to mon h 2024-09-11T02:58:27.727 INFO:tasks.ceph.mon.h:Sent signal 19 2024-09-11T02:58:27.727 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T02:58:27.826 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-09-11T02:58:27.822+0000 7fde1a593e00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 292 -> 0 (sync_force? mon rebuild?) 2024-09-11T02:58:27.832 INFO:tasks.ceph.mon.g.smithi190.stderr:2024-09-11T02:58:27.832+0000 7fc24eb40e00 -1 mon.g@-1(???).log v0 log_external_backlog rewinding external_log_to from 292 -> 0 (sync_force? mon rebuild?) 2024-09-11T02:58:42.727 INFO:tasks.mon_thrash:Sending CONT to mon f 2024-09-11T02:58:42.728 INFO:tasks.ceph.mon.f:Sent signal 18 2024-09-11T02:58:42.728 INFO:tasks.mon_thrash:Sending CONT to mon h 2024-09-11T02:58:42.729 INFO:tasks.ceph.mon.h:Sent signal 18 2024-09-11T02:58:42.729 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:58:42.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:58:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.854+0000 7f1743218640 10 monclient: get_monmap_and_config 2024-09-11T02:58:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.854+0000 7f1743218640 10 monclient: build_initial_monmap 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.854+0000 7f1743218640 10 monclient: monmap: 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:42.859482+0000 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:42.859482+0000 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:42.859 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:42.859 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:42.859 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:42.859 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:42.859 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.854+0000 7f1743218640 1 Processor -- start 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.854+0000 7f1743218640 1 -- start start 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.854+0000 7f1743218640 10 monclient: init 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient: _add_conns ranks=[9,1,8,10,3,2,6,4,7,5,0] 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient(hunting): picked mon.noname-a con 0x7f173c0a5930 addr v2:172.21.15.98:3300/0 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f173c0a1870 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient(hunting): picked mon.noname-e con 0x7f173c0a6250 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f173c071d90 con 0x7f173c0a1870 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f173c0a4170 con 0x7f173c0a5930 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f173c073b50 con 0x7f173c0a6250 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1743218640 10 monclient(hunting): authenticate will time out at 1476.740462s 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:38952/0 (socket says 172.21.15.98:38952) 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 learned_addr learned my addr 172.21.15.98:0/2083138976 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 10 monclient(hunting): get_auth_request con 0x7f173c0a6250 auth_method 0 2024-09-11T02:58:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 10 monclient(hunting): get_auth_request con 0x7f173c0a5930 auth_method 0 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:58:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:42.858+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:58:43.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:43.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 10 monclient(hunting): get_auth_request con 0x7f173c0a5930 auth_method 0 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 10 monclient(hunting): get_auth_request con 0x7f173c0a6250 auth_method 0 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:43.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:58:43.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:43.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:58:43.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:43.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.062+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 10 monclient(hunting): get_auth_request con 0x7f173c0a5930 auth_method 0 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 10 monclient(hunting): get_auth_request con 0x7f173c0a6250 auth_method 0 2024-09-11T02:58:43.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:43.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:43.466+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 10 monclient(hunting): get_auth_request con 0x7f173c0a5930 auth_method 0 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 10 monclient(hunting): get_auth_request con 0x7f173c0a6250 auth_method 0 2024-09-11T02:58:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:58:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:44.270+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:58:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): tick 2024-09-11T02:58:45.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:58:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 monclient(hunting): continuing hunt 2024-09-11T02:58:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:58:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 msgr2=0x7f173c137df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:58:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f173c0a6250 0x7f173c137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 -- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 msgr2=0x7f173c0a5d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:58:45.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 --2- 172.21.15.98:0/2083138976 >> v2:172.21.15.98:3300/0 conn(0x7f173c0a5930 0x7f173c0a5d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 -- 172.21.15.98:0/2083138976 >> v1:172.21.15.190:6789/0 conn(0x7f173c0a1870 legacy=0x7f173c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient: _add_conns ranks=[5,0,3,4,7,2,10,6] 2024-09-11T02:58:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1728003820 0x7f1728005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): picked mon.noname-h con 0x7f1728003820 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17280062c0 addr v1:172.21.15.98:6789/0 2024-09-11T02:58:45.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1728009fc0 0x7f172800c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): picked mon.noname-g con 0x7f1728009fc0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 -- 172.21.15.98:0/2083138976 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f173c073870 con 0x7f17280062c0 2024-09-11T02:58:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 -- 172.21.15.98:0/2083138976 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f173c0a4170 con 0x7f1728009fc0 2024-09-11T02:58:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:45.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 1 -- 172.21.15.98:0/2083138976 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f173c073b50 con 0x7f1728003820 2024-09-11T02:58:45.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17327fc640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:45.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1728009fc0 0x7f172800c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:45.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1728003820 0x7f1728005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:45.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): get_auth_request con 0x7f1728009fc0 auth_method 0 2024-09-11T02:58:45.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:45.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 10 monclient(hunting): get_auth_request con 0x7f1728003820 auth_method 0 2024-09-11T02:58:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:45.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:45.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17337fe640 1 -- 172.21.15.98:0/2083138976 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3699909636 0 0) 0x7f173c073870 con 0x7f17280062c0 2024-09-11T02:58:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17337fe640 10 monclient(hunting): _negotiate 2024-09-11T02:58:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17337fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17337fe640 10 monclient(hunting): my global_id is 44388 2024-09-11T02:58:45.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17337fe640 1 -- 172.21.15.98:0/2083138976 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1724003530 con 0x7f17280062c0 2024-09-11T02:58:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient(hunting): handle_auth_done global_id 64101 payload 274 2024-09-11T02:58:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1728003820 msgr2=0x7f1728005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1728003820 0x7f1728005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 >> v1:172.21.15.98:6789/0 conn(0x7f17280062c0 legacy=0x7f1728008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient: _finish_hunting 0 2024-09-11T02:58:45.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 1 monclient: found mon.noname-g 2024-09-11T02:58:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T02:58:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172800dd50 con 0x7f1728009fc0 2024-09-11T02:58:45.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.858+0000 7f1740fb4640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1728003820 0x7f1728005c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17417b5640 1 -- 172.21.15.98:0/2083138976 reap_dead start 2024-09-11T02:58:45.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17417b5640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1728009fc0 0x7f172800c460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1734009c50 tx=0x7f173401d470 comp rx=0 tx=0).ready entity=mon.3 client_cookie=2cfa3ab18c5e7d3e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 1 -- 172.21.15.98:0/2083138976 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17340b3070 con 0x7f1728009fc0 2024-09-11T02:58:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T02:58:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: dump: 2024-09-11T02:58:45.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:45.884 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:45.885 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:45.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:45.886 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:45.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:45.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:45.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: _finish_auth 0 2024-09-11T02:58:45.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: _check_auth_tickets 2024-09-11T02:58:45.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:45.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 5 monclient: authenticate success, global_id 64101 2024-09-11T02:58:45.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:45.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 1 -- 172.21.15.98:0/2083138976 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17340ad7e0 con 0x7f1728009fc0 2024-09-11T02:58:45.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:45.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:45.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 4 set_mon_vals no callback set 2024-09-11T02:58:45.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 1 -- 172.21.15.98:0/2083138976 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17340ade00 con 0x7f1728009fc0 2024-09-11T02:58:45.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:45.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:58:45.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f17337fe640 10 monclient: dump: 2024-09-11T02:58:45.889 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:45.889 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:45.889 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.889 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.890 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:45.890 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:45.890 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:45.890 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:45.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:45.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 10 monclient: shutdown 2024-09-11T02:58:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 1 -- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1728009fc0 msgr2=0x7f172800c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1728009fc0 0x7f172800c460 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1734009c50 tx=0x7f173401d470 comp rx=0 tx=0).stop 2024-09-11T02:58:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 1 -- 172.21.15.98:0/2083138976 shutdown_connections 2024-09-11T02:58:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 1 --2- 172.21.15.98:0/2083138976 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1728009fc0 0x7f172800c460 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 1 -- 172.21.15.98:0/2083138976 >> 172.21.15.98:0/2083138976 conn(0x7f173c0aa520 msgr2=0x7f173c0a6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:45.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 1 -- 172.21.15.98:0/2083138976 shutdown_connections 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.862+0000 7f1743218640 1 -- 172.21.15.98:0/2083138976 wait complete. 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: build_initial_monmap 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: monmap: 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:45.893 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:45.894 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:45.894 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:45.894 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:45.894 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:45.894 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:45.894 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:45.894 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 librados: starting msgr at 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 librados: starting objecter 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 Processor -- start 2024-09-11T02:58:45.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 -- start start 2024-09-11T02:58:45.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 librados: setting wanted keys 2024-09-11T02:58:45.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 librados: calling monclient init 2024-09-11T02:58:45.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: init 2024-09-11T02:58:45.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:45.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: _add_conns ranks=[7,5,0,3,8,6,1,4,2] 2024-09-11T02:58:45.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1728009fc0 0x7f173c1d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:45.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): picked mon.noname-i con 0x7f1728009fc0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f173c13daf0 0x7f173c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): picked mon.noname-h con 0x7f173c13daf0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f173c1378d0 0x7f173c137cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): picked mon.noname-a con 0x7f173c1378d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f173c0a5d10 con 0x7f173c1378d0 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:45.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f173c070d10 con 0x7f173c13daf0 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f173c0a5e90 con 0x7f1728009fc0 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient(hunting): authenticate will time out at 1479.750231s 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f173c1378d0 0x7f173c137cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f173c1378d0 0x7f173c137cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40260/0 (socket says 172.21.15.98:40260) 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 -- 172.21.15.98:0/1200522987 learned_addr learned my addr 172.21.15.98:0/1200522987 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:45.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): get_auth_request con 0x7f173c1378d0 auth_method 0 2024-09-11T02:58:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f173c13daf0 0x7f173c13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:45.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 10 monclient(hunting): get_auth_request con 0x7f173c13daf0 auth_method 0 2024-09-11T02:58:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1740fb4640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1728009fc0 0x7f173c1d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:45.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1740fb4640 10 monclient(hunting): get_auth_request con 0x7f1728009fc0 auth_method 0 2024-09-11T02:58:45.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1740fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:45.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1740fb4640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:45.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1740fb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:45.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:45.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:45.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1733fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:45.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient(hunting): handle_auth_done global_id 44397 payload 824 2024-09-11T02:58:45.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 -- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1728009fc0 msgr2=0x7f173c1d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:45.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1728009fc0 0x7f173c1d0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:45.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 -- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f173c13daf0 msgr2=0x7f173c13df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:45.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f173c13daf0 0x7f173c13df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:45.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient: _finish_hunting 0 2024-09-11T02:58:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 monclient: found mon.noname-a 2024-09-11T02:58:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:58:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 -- 172.21.15.98:0/1200522987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173c139250 con 0x7f173c1378d0 2024-09-11T02:58:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17417b5640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f173c1378d0 0x7f173c137cf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1734009c50 tx=0x7f173401bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 client.?.objecter ms_handle_connect 0x7f173c1378d0 2024-09-11T02:58:45.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:45.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 1 -- 172.21.15.98:0/1200522987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17340b3040 con 0x7f173c1378d0 2024-09-11T02:58:45.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:45.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:58:45.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: dump: 2024-09-11T02:58:45.904 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:45.904 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:45.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:45.906 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:45.907 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: _finish_auth 0 2024-09-11T02:58:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: _check_auth_tickets 2024-09-11T02:58:45.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 1 -- 172.21.15.98:0/1200522987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17340bf070 con 0x7f173c1378d0 2024-09-11T02:58:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 1 -- 172.21.15.98:0/1200522987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17340ade00 con 0x7f173c1378d0 2024-09-11T02:58:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 5 monclient: authenticate success, global_id 44397 2024-09-11T02:58:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: _renew_subs 2024-09-11T02:58:45.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:45.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f173c13e930 con 0x7f173c1378d0 2024-09-11T02:58:45.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:45.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1732ffd640 4 set_mon_vals no callback set 2024-09-11T02:58:45.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:45.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f17317fa640 10 monclient: dump: 2024-09-11T02:58:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:45.910 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:45.911 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:45.912 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 client.44397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:45.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: _renew_subs 2024-09-11T02:58:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f173c13aff0 con 0x7f173c1378d0 2024-09-11T02:58:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.866+0000 7f1743218640 1 librados: init done 2024-09-11T02:58:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f1743218640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f1743218640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:45.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f1743218640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f173c140150 con 0x7f173c1378d0 2024-09-11T02:58:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 1 -- 172.21.15.98:0/1200522987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f17340b6030 con 0x7f173c1378d0 2024-09-11T02:58:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f172803f280 0x7f1728041740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:45.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 1 -- 172.21.15.98:0/1200522987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(357..357 src has 1..357) ==== 6202+0+0 (secure 0 0 0) 0x7f17340ab070 con 0x7f173c1378d0 2024-09-11T02:58:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 10 client.44397.objecter ms_dispatch 0x7f173c000ba0 osd_map(357..357 src has 1..357) 2024-09-11T02:58:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 3 client.44397.objecter handle_osd_map got epochs [357,357] > 0 2024-09-11T02:58:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 3 client.44397.objecter handle_osd_map decoding full epoch 357 2024-09-11T02:58:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 20 client.44397.objecter dump_active .. 0 homeless 2024-09-11T02:58:45.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f1740fb4640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f172803f280 0x7f1728041740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f1740fb4640 10 monclient: get_auth_request con 0x7f172803f280 auth_method 0 2024-09-11T02:58:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f17317fa640 10 client.44397.objecter ms_handle_connect 0x7f172803f280 2024-09-11T02:58:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.870+0000 7f1740fb4640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f172803f280 0x7f1728041740 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f1738000ee0 tx=0x7f173800c3c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.874+0000 7f17317fa640 1 -- 172.21.15.98:0/1200522987 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f17340171c0 con 0x7f173c1378d0 2024-09-11T02:58:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.874+0000 7f17317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:45.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:45.874+0000 7f17317fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:46.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.110+0000 7f1743218640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:58:46.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.110+0000 7f1743218640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:46.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.110+0000 7f1743218640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:46.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.110+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f173c139db0 con 0x7f173c1378d0 2024-09-11T02:58:46.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.110+0000 7f17317fa640 1 -- 172.21.15.98:0/1200522987 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7f17340c33c0 con 0x7f173c1378d0 2024-09-11T02:58:46.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.110+0000 7f17317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:46.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.110+0000 7f17317fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:46.113 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":66,"quorum":[0,2,3,4,5,6,7,8],"quorum_names":["a","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 10 librados: watch_flush enter 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 10 librados: watch_flush exit 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 20 client.44397.objecter shutdown clearing up homeless session... 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 10 client.44397.objecter successfully canceled tick 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f172803f280 msgr2=0x7f1728041740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f172803f280 0x7f1728041740 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f1738000ee0 tx=0x7f173800c3c0 comp rx=0 tx=0).stop 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 10 monclient: shutdown 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:46.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f173c1378d0 msgr2=0x7f173c137cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f173c1378d0 0x7f173c137cf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1734009c50 tx=0x7f173401bcb0 comp rx=0 tx=0).stop 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 shutdown_connections 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f172803f280 0x7f1728041740 unknown :-1 s=CLOSED pgs=3532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f173c1378d0 0x7f173c137cf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f173c13daf0 0x7f173c13df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 --2- 172.21.15.98:0/1200522987 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f1728009fc0 0x7f173c1d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 >> 172.21.15.98:0/1200522987 conn(0x7f173c0aa520 msgr2=0x7f173c0a2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 shutdown_connections 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 -- 172.21.15.98:0/1200522987 wait complete. 2024-09-11T02:58:46.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:46.114+0000 7f1743218640 1 librados: shutdown 2024-09-11T02:58:49.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient: get_monmap_and_config 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient: build_initial_monmap 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient: monmap: 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:49.262832+0000 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:49.262832+0000 2024-09-11T02:58:49.261 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:49.262 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:49.263 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:49.263 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:49.263 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:49.263 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:49.263 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 Processor -- start 2024-09-11T02:58:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 -- start start 2024-09-11T02:58:49.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient: init 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient: _add_conns ranks=[6,4,5,10,0,2,1,9,3,8,7] 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc0725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): picked mon.noname-d con 0x7f8fcc139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8fcc06fe30 0x7f8fcc139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): picked mon.noname-c con 0x7f8fcc06fe30 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a7a10 0x7f8fcc0a7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): picked mon.noname-h con 0x7f8fcc0a7a10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8fcc0a9650 con 0x7f8fcc06fe30 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8fcc071350 con 0x7f8fcc0a7a10 2024-09-11T02:58:49.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8fcc071630 con 0x7f8fcc139b10 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fd0ede640 10 monclient(hunting): authenticate will time out at 1483.143599s 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8fcc06fe30 0x7f8fcc139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8fcc06fe30 0x7f8fcc139ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59670/0 (socket says 172.21.15.98:59670) 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 1 -- 172.21.15.98:0/599151237 learned_addr learned my addr 172.21.15.98:0/599151237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc0725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44482/0 (socket says 172.21.15.98:44482) 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 10 monclient(hunting): get_auth_request con 0x7f8fcc06fe30 auth_method 0 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 10 monclient(hunting): get_auth_request con 0x7f8fcc139b10 auth_method 0 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.258+0000 7f8fca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fcad9d640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a7a10 0x7f8fcc0a7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_done global_id 23917 payload 274 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 1 -- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 msgr2=0x7f8fcc0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 1 -- 172.21.15.98:0/599151237 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a7a10 msgr2=0x7f8fcc0a7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a7a10 0x7f8fcc0a7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 1 monclient: found mon.noname-c 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:49.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:49.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 1 -- 172.21.15.98:0/599151237 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc0702b0 con 0x7f8fcc06fe30 2024-09-11T02:58:49.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fcad9d640 10 monclient: get_auth_request con 0x7f8fcc0a7a10 auth_method 0 2024-09-11T02:58:49.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fca59c640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc0725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:49.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fcad9d640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a7a10 0x7f8fcc0a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.262+0000 7f8fc3fff640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8fcc06fe30 0x7f8fcc139ef0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40098a0 tx=0x7f8fb401c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e626c6cadef2d43f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 1 -- 172.21.15.98:0/599151237 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8fb40b2070 con 0x7f8fcc06fe30 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: dump: 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: _finish_auth 0 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 5 monclient: authenticate success, global_id 23917 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 1 -- 172.21.15.98:0/599151237 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb40ab7f0 con 0x7f8fcc06fe30 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 4 set_mon_vals no callback set 2024-09-11T02:58:49.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 1 -- 172.21.15.98:0/599151237 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8fb40abe50 con 0x7f8fcc06fe30 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc9d9b640 10 monclient: dump: 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.282 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: shutdown 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8fcc06fe30 msgr2=0x7f8fcc139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8fcc06fe30 0x7f8fcc139ef0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40098a0 tx=0x7f8fb401c440 comp rx=0 tx=0).stop 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- 172.21.15.98:0/599151237 shutdown_connections 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a7a10 0x7f8fcc0a7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8fcc06fe30 0x7f8fcc139ef0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/599151237 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc0725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- 172.21.15.98:0/599151237 >> 172.21.15.98:0/599151237 conn(0x7f8fcc0ac740 msgr2=0x7f8fcc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:49.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- 172.21.15.98:0/599151237 shutdown_connections 2024-09-11T02:58:49.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- 172.21.15.98:0/599151237 wait complete. 2024-09-11T02:58:49.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: build_initial_monmap 2024-09-11T02:58:49.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: monmap: 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:49.285 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 librados: starting msgr at 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 librados: starting objecter 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 Processor -- start 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- start start 2024-09-11T02:58:49.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 librados: setting wanted keys 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 librados: calling monclient init 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: init 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: _add_conns ranks=[8,6,5,2,7,4,0,1,3] 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8fcc0a7a10 0x7f8fcc131b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): picked mon.noname-e con 0x7f8fcc0a7a10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc132070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): picked mon.noname-d con 0x7f8fcc139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a4dc0 0x7f8fcc0a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): picked mon.noname-h con 0x7f8fcc0a4dc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8fcc1435c0 con 0x7f8fcc0a4dc0 2024-09-11T02:58:49.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8fcc143440 con 0x7f8fcc139b10 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8fcc143740 con 0x7f8fcc0a7a10 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient(hunting): authenticate will time out at 1483.150515s 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc132070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8fcc0a7a10 0x7f8fcc131b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc132070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44486/0 (socket says 172.21.15.98:44486) 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 -- 172.21.15.98:0/2073883098 learned_addr learned my addr 172.21.15.98:0/2073883098 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): get_auth_request con 0x7f8fcc139b10 auth_method 0 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 10 monclient(hunting): get_auth_request con 0x7f8fcc0a7a10 auth_method 0 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fcad9d640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a4dc0 0x7f8fcc0a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fcad9d640 10 monclient(hunting): get_auth_request con 0x7f8fcc0a4dc0 auth_method 0 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fcad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fcad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fcad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient(hunting): handle_auth_done global_id 44295 payload 824 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 -- 172.21.15.98:0/2073883098 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8fcc0a7a10 msgr2=0x7f8fcc131b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8fcc0a7a10 0x7f8fcc131b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 -- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a4dc0 msgr2=0x7f8fcc0a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a4dc0 0x7f8fcc0a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 monclient: found mon.noname-d 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 -- 172.21.15.98:0/2073883098 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc13afa0 con 0x7f8fcc139b10 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc3fff640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc132070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40097c0 tx=0x7f8fb40191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 client.?.objecter ms_handle_connect 0x7f8fcc139b10 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 1 -- 172.21.15.98:0/2073883098 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8fb40bd070 con 0x7f8fcc139b10 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: dump: 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.291 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:49.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 1 -- 172.21.15.98:0/2073883098 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fb40abe10 con 0x7f8fcc139b10 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 5 monclient: authenticate success, global_id 44295 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 1 -- 172.21.15.98:0/2073883098 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8fb40b2070 con 0x7f8fcc139b10 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc2ffd640 10 monclient: dump: 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fc8d99640 4 set_mon_vals no callback set 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: _renew_subs 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.266+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fcc13b280 con 0x7f8fcc139b10 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 10 client.44295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 10 monclient: _renew_subs 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fcc13acc0 con 0x7f8fcc139b10 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 1 librados: init done 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 1 -- 172.21.15.98:0/2073883098 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8fb4004250 con 0x7f8fcc139b10 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fcc0a7e70 con 0x7f8fcc139b10 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fac03df40 0x7f8fac040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 1 -- 172.21.15.98:0/2073883098 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(359..359 src has 1..359) ==== 6189+0+0 (secure 0 0 0) 0x7f8fb40e8ff0 con 0x7f8fcc139b10 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 10 client.44295.objecter ms_dispatch 0x7f8fcc000ba0 osd_map(359..359 src has 1..359) 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 3 client.44295.objecter handle_osd_map got epochs [359,359] > 0 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 3 client.44295.objecter handle_osd_map decoding full epoch 359 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 20 client.44295.objecter dump_active .. 0 homeless 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fca59c640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fac03df40 0x7f8fac040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fca59c640 10 monclient: get_auth_request con 0x7f8fac03df40 auth_method 0 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fc2ffd640 10 client.44295.objecter ms_handle_connect 0x7f8fac03df40 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.270+0000 7f8fca59c640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fac03df40 0x7f8fac040400 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009580 tx=0x7f8fbc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.274+0000 7f8fc2ffd640 1 -- 172.21.15.98:0/2073883098 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8fb4016150 con 0x7f8fcc139b10 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.274+0000 7f8fc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:49.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.274+0000 7f8fc2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:49.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:58:49.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:49.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8fcc13bc40 con 0x7f8fcc139b10 2024-09-11T02:58:49.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fc2ffd640 1 -- 172.21.15.98:0/2073883098 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f8fb40ac3e0 con 0x7f8fcc139b10 2024-09-11T02:58:49.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:49.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fc2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:49.517 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":68,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:58:49.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 10 librados: watch_flush enter 2024-09-11T02:58:49.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 10 librados: watch_flush exit 2024-09-11T02:58:49.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 20 client.44295.objecter shutdown clearing up homeless session... 2024-09-11T02:58:49.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 10 client.44295.objecter successfully canceled tick 2024-09-11T02:58:49.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fac03df40 msgr2=0x7f8fac040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fac03df40 0x7f8fac040400 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009580 tx=0x7f8fbc009190 comp rx=0 tx=0).stop 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 10 monclient: shutdown 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 msgr2=0x7f8fcc132070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc132070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40097c0 tx=0x7f8fb40191e0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 shutdown_connections 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fac03df40 0x7f8fac040400 unknown :-1 s=CLOSED pgs=3533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8fcc0a4dc0 0x7f8fcc0a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8fcc139b10 0x7f8fcc132070 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 --2- 172.21.15.98:0/2073883098 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8fcc0a7a10 0x7f8fcc131b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 >> 172.21.15.98:0/2073883098 conn(0x7f8fcc0ac740 msgr2=0x7f8fcc135400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 shutdown_connections 2024-09-11T02:58:49.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 -- 172.21.15.98:0/2073883098 wait complete. 2024-09-11T02:58:49.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.514+0000 7f8fd0ede640 1 librados: shutdown 2024-09-11T02:58:49.536 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:58:49.536 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:58:49.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient: get_monmap_and_config 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient: build_initial_monmap 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient: monmap: 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:49.654864+0000 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:49.654864+0000 2024-09-11T02:58:49.653 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:49.654 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 1 Processor -- start 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 1 -- start start 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient: init 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient: _add_conns ranks=[0,7,9,10,8,4,1,6,5,2,3] 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2db4139b10 addr v1:172.21.15.98:6789/0 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2db4072600 0x7f2db413ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient(hunting): picked mon.noname-i con 0x7f2db4072600 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f2db413ef60 0x7f2db4141350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient(hunting): picked mon.noname-a con 0x7f2db413ef60 addr v2:172.21.15.98:3300/0 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db413a600 con 0x7f2db4139b10 2024-09-11T02:58:49.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f2db40702b0 con 0x7f2db413ef60 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f2db4071bf0 con 0x7f2db4072600 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db906d640 10 monclient(hunting): authenticate will time out at 1483.535287s 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db2d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2db4139b10 0x7f2db4139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50004/0 (socket says 172.21.15.98:50004) 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db2d9d640 1 -- 172.21.15.98:0/482536293 learned_addr learned my addr 172.21.15.98:0/482536293 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db359e640 1 --2- 172.21.15.98:0/482536293 >> v2:172.21.15.98:3300/0 conn(0x7f2db413ef60 0x7f2db4141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db359e640 10 monclient(hunting): get_auth_request con 0x7f2db413ef60 auth_method 0 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db359e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.650+0000 7f2db359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 1 --2- 172.21.15.98:0/482536293 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2db4072600 0x7f2db413ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 10 monclient(hunting): get_auth_request con 0x7f2db4072600 auth_method 0 2024-09-11T02:58:49.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 10 monclient(hunting): handle_auth_done global_id 44406 payload 274 2024-09-11T02:58:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 1 -- 172.21.15.98:0/482536293 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2db4072600 msgr2=0x7f2db413ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 1 --2- 172.21.15.98:0/482536293 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2db4072600 0x7f2db413ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 1 -- 172.21.15.98:0/482536293 >> v1:172.21.15.98:6789/0 conn(0x7f2db4139b10 legacy=0x7f2db4139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 10 monclient: _finish_hunting 0 2024-09-11T02:58:49.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 1 monclient: found mon.noname-a 2024-09-11T02:58:49.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:49.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:58:49.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 1 -- 172.21.15.98:0/482536293 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db4071350 con 0x7f2db413ef60 2024-09-11T02:58:49.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db359e640 1 --2- 172.21.15.98:0/482536293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2db413ef60 0x7f2db4141350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2da80097c0 tx=0x7f2da801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cd8ab18766a79d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 1 -- 172.21.15.98:0/482536293 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2da80b0070 con 0x7f2db413ef60 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: dump: 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: _finish_auth 0 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 1 -- 172.21.15.98:0/482536293 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da80aa810 con 0x7f2db413ef60 2024-09-11T02:58:49.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db906d640 5 monclient: authenticate success, global_id 44406 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db906d640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db906d640 4 set_mon_vals no callback set 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 1 -- 172.21.15.98:0/482536293 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2da80a9300 con 0x7f2db413ef60 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.654+0000 7f2db1d9b640 10 monclient: dump: 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.675 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient: shutdown 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- 172.21.15.98:0/482536293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2db413ef60 msgr2=0x7f2db4141350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 --2- 172.21.15.98:0/482536293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2db413ef60 0x7f2db4141350 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2da80097c0 tx=0x7f2da801c350 comp rx=0 tx=0).stop 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- 172.21.15.98:0/482536293 shutdown_connections 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 --2- 172.21.15.98:0/482536293 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2db413ef60 0x7f2db4141350 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 --2- 172.21.15.98:0/482536293 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2db4072600 0x7f2db413ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- 172.21.15.98:0/482536293 >> 172.21.15.98:0/482536293 conn(0x7f2db40ac790 msgr2=0x7f2db40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- 172.21.15.98:0/482536293 shutdown_connections 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- 172.21.15.98:0/482536293 wait complete. 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient: build_initial_monmap 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient: monmap: 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:49.677 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:49.678 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 librados: starting msgr at 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 librados: starting objecter 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 Processor -- start 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- start start 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 librados: setting wanted keys 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 librados: calling monclient init 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient: init 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient: _add_conns ranks=[3,6,4,5,7,0,2,8,1] 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2db4072600 0x7f2db4138770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): picked mon.noname-g con 0x7f2db4072600 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2db4139b10 0x7f2db40a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): picked mon.noname-d con 0x7f2db4139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2db413ef60 0x7f2db40a3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): picked mon.noname-c con 0x7f2db413ef60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f2db4144230 con 0x7f2db4072600 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2db41440b0 con 0x7f2db413ef60 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2db41443b0 con 0x7f2db4139b10 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db906d640 10 monclient(hunting): authenticate will time out at 1483.542820s 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2db4139b10 0x7f2db40a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2db4139b10 0x7f2db40a2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44496/0 (socket says 172.21.15.98:44496) 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 -- 172.21.15.98:0/3359925966 learned_addr learned my addr 172.21.15.98:0/3359925966 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): get_auth_request con 0x7f2db4139b10 auth_method 0 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient(hunting): handle_auth_done global_id 44304 payload 824 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2db413ef60 msgr2=0x7f2db40a3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2db413ef60 0x7f2db40a3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2db4072600 msgr2=0x7f2db4138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f2db4072600 0x7f2db4138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient: _finish_hunting 0 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 monclient: found mon.noname-d 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.658+0000 7f2db259c640 1 -- 172.21.15.98:0/3359925966 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db40a5660 con 0x7f2db4139b10 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db259c640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2db4139b10 0x7f2db40a2fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2da400e930 tx=0x7f2da400edf0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 client.?.objecter ms_handle_connect 0x7f2db4139b10 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2da4017070 con 0x7f2db4139b10 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.682 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: _finish_auth 0 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da4013470 con 0x7f2db4139b10 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2da4013a90 con 0x7f2db4139b10 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2d9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 5 monclient: authenticate success, global_id 44304 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 10 monclient: _renew_subs 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db413ec20 con 0x7f2db4139b10 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 10 client.44304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 10 monclient: _renew_subs 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db4122e40 con 0x7f2db4139b10 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 1 librados: init done 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db159a640 4 set_mon_vals no callback set 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:49.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94002890 0x7f2d94004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.662+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94007cc0 con 0x7f2d94002890 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2da4013c30 con 0x7f2db4139b10 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2da003df40 0x7f2da0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(360..360 src has 1..360) ==== 6202+0+0 (secure 0 0 0) 0x7f2da400f040 con 0x7f2db4139b10 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 client.44304.objecter ms_dispatch 0x7f2db4000ba0 osd_map(360..360 src has 1..360) 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 3 client.44304.objecter handle_osd_map got epochs [360,360] > 0 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 3 client.44304.objecter handle_osd_map decoding full epoch 360 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 20 client.44304.objecter dump_active .. 0 homeless 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db359e640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2da003df40 0x7f2da0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94002890 0x7f2d94004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient: get_auth_request con 0x7f2d94002890 auth_method 0 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db359e640 10 monclient: get_auth_request con 0x7f2da003df40 auth_method 0 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_done global_id 44415 payload 824 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 1 -- 172.21.15.98:0/3359925966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d94007ec0 con 0x7f2d94002890 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db2d9d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94002890 0x7f2d94004d30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00ca90 tx=0x7f2dac00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 client.44304.objecter ms_handle_connect 0x7f2d94002890 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 client.44304.objecter resend_mon_ops 2024-09-11T02:58:49.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2dac017070 con 0x7f2d94002890 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2dac00ae20 con 0x7f2d94002890 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94002890 msgr2=0x7f2d94004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94002890 0x7f2d94004d30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00ca90 tx=0x7f2dac00cf50 comp rx=0 tx=0).stop 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94002890 msgr2=0x7f2d94004d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:49.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94002890 0x7f2d94004d30 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00ca90 tx=0x7f2dac00cf50 comp rx=0 tx=0).stop 2024-09-11T02:58:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2db359e640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2da003df40 0x7f2da0040400 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7f2da80ad000 tx=0x7f2da8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.666+0000 7f2d9b7fe640 10 client.44304.objecter ms_handle_connect 0x7f2da003df40 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db906d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db906d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94006d30 0x7f2d940071c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db906d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9400a1e0 con 0x7f2d94006d30 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94006d30 0x7f2d940071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient: get_auth_request con 0x7f2d94006d30 auth_method 0 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 10 monclient(hunting): handle_auth_done global_id 44424 payload 824 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 1 -- 172.21.15.98:0/3359925966 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d94007700 con 0x7f2d94006d30 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2db2d9d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94006d30 0x7f2d940071c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00ca60 tx=0x7f2dac001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2d9b7fe640 10 client.44304.objecter ms_handle_connect 0x7f2d94006d30 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2d9b7fe640 10 client.44304.objecter resend_mon_ops 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2dac017040 con 0x7f2d94006d30 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2d9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2d9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.674+0000 7f2d9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.694 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.695 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f2dac023420 con 0x7f2d94006d30 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2d9b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2d9b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94006d30 msgr2=0x7f2d940071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2d9b7fe640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94006d30 0x7f2d940071c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00ca60 tx=0x7f2dac001260 comp rx=0 tx=0).stop 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2d9b7fe640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94006d30 msgr2=0x7f2d940071c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2d9b7fe640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d94006d30 0x7f2d940071c0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2dac00ca60 tx=0x7f2dac001260 comp rx=0 tx=0).stop 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:49.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 156114, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:49.697 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:49.698 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:49.699 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:49.700 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:49.701 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:49.702 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:49.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:49.708 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:49.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:49.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:49.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.719 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.720 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:49.721 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 10 librados: watch_flush enter 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 10 librados: watch_flush exit 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 20 client.44304.objecter shutdown clearing up homeless session... 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 10 client.44304.objecter successfully canceled tick 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2da003df40 msgr2=0x7f2da0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2da003df40 0x7f2da0040400 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7f2da80ad000 tx=0x7f2da8004740 comp rx=0 tx=0).stop 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 10 monclient: shutdown 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:49.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2db4139b10 msgr2=0x7f2db40a2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 --2- 172.21.15.98:0/3359925966 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2db4139b10 0x7f2db40a2fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2da400e930 tx=0x7f2da400edf0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db359e640 1 -- 172.21.15.98:0/3359925966 reap_dead start 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db359e640 1 -- 172.21.15.98:0/3359925966 reap_dead start 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 shutdown_connections 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 >> 172.21.15.98:0/3359925966 conn(0x7f2db40ac790 msgr2=0x7f2db40a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 shutdown_connections 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 -- 172.21.15.98:0/3359925966 wait complete. 2024-09-11T02:58:49.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.678+0000 7f2db906d640 1 librados: shutdown 2024-09-11T02:58:49.724 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:49.725 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:58:49.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:58:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient: get_monmap_and_config 2024-09-11T02:58:49.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient: build_initial_monmap 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient: monmap: 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:49.837582+0000 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:49.837582+0000 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:49.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:49.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:49.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:49.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:49.837 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:49.837 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:49.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 Processor -- start 2024-09-11T02:58:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 -- start start 2024-09-11T02:58:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient: init 2024-09-11T02:58:49.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient: _add_conns ranks=[3,9,8,10,7,5,0,6,1,2,4] 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eb40a3de0 0x7f9eb40a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): picked mon.noname-g con 0x7f9eb40a3de0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9eb40a41c0 0x7f9eb40a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): picked mon.noname-a con 0x7f9eb40a41c0 addr v2:172.21.15.98:3300/0 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9eb409b240 0x7f9eb409b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): picked mon.noname-e con 0x7f9eb409b240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f9eb40a0170 con 0x7f9eb40a41c0 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb4068fa0 con 0x7f9eb40a3de0 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9eb4068b40 con 0x7f9eb409b240 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb9664640 10 monclient(hunting): authenticate will time out at 1483.718593s 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9eb40a41c0 0x7f9eb40a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9eb40a41c0 0x7f9eb40a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38608/0 (socket says 172.21.15.98:38608) 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 -- 172.21.15.98:0/2943410877 learned_addr learned my addr 172.21.15.98:0/2943410877 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): get_auth_request con 0x7f9eb40a41c0 auth_method 0 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9eb409b240 0x7f9eb409b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 10 monclient(hunting): get_auth_request con 0x7f9eb409b240 auth_method 0 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb2ffd640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eb40a3de0 0x7f9eb40a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb2ffd640 10 monclient(hunting): get_auth_request con 0x7f9eb40a3de0 auth_method 0 2024-09-11T02:58:49.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient(hunting): handle_auth_done global_id 44433 payload 274 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 -- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9eb409b240 msgr2=0x7f9eb409b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9eb409b240 0x7f9eb409b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 -- 172.21.15.98:0/2943410877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eb40a3de0 msgr2=0x7f9eb40a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eb40a3de0 0x7f9eb40a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient: _finish_hunting 0 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 monclient: found mon.noname-a 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:58:49.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.834+0000 7f9eb27fc640 1 -- 172.21.15.98:0/2943410877 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb409eaf0 con 0x7f9eb40a41c0 2024-09-11T02:58:49.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb40a41c0 0x7f9eb40a45c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80098a0 tx=0x7f9ea801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e50831ff669a85 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 1 -- 172.21.15.98:0/2943410877 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea80b0070 con 0x7f9eb40a41c0 2024-09-11T02:58:49.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: dump: 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.853 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: _finish_auth 0 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 1 -- 172.21.15.98:0/2943410877 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea80aa810 con 0x7f9eb40a41c0 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 5 monclient: authenticate success, global_id 44433 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:49.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 4 set_mon_vals no callback set 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 1 -- 172.21.15.98:0/2943410877 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea80a9300 con 0x7f9eb40a41c0 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb1ffb640 10 monclient: dump: 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.855 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient: shutdown 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb40a41c0 msgr2=0x7f9eb40a45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb40a41c0 0x7f9eb40a45c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80098a0 tx=0x7f9ea801c400 comp rx=0 tx=0).stop 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- 172.21.15.98:0/2943410877 shutdown_connections 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9eb409b240 0x7f9eb409b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9eb40a41c0 0x7f9eb40a45c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 --2- 172.21.15.98:0/2943410877 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9eb40a3de0 0x7f9eb40a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- 172.21.15.98:0/2943410877 >> 172.21.15.98:0/2943410877 conn(0x7f9eb40a4ed0 msgr2=0x7f9eb40a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- 172.21.15.98:0/2943410877 shutdown_connections 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- 172.21.15.98:0/2943410877 wait complete. 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient: build_initial_monmap 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient: monmap: 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:49.857 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 librados: starting msgr at 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 librados: starting objecter 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 Processor -- start 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- start start 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 librados: setting wanted keys 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 librados: calling monclient init 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient: init 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient: _add_conns ranks=[7,2,4,1,6,3,8,5,0] 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eb409b240 0x7f9eb413ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): picked mon.noname-i con 0x7f9eb409b240 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40a3de0 0x7f9eb413b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): picked mon.noname-b con 0x7f9eb40a3de0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9eb40a41c0 0x7f9eb413b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): picked mon.noname-c con 0x7f9eb40a41c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb41413a0 con 0x7f9eb40a3de0 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9eb4141220 con 0x7f9eb40a41c0 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f9eb4141520 con 0x7f9eb409b240 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb9664640 10 monclient(hunting): authenticate will time out at 1483.723515s 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9eb40a41c0 0x7f9eb413b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40a3de0 0x7f9eb413b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9eb40a41c0 0x7f9eb413b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59678/0 (socket says 172.21.15.98:59678) 2024-09-11T02:58:49.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 -- 172.21.15.98:0/1624124214 learned_addr learned my addr 172.21.15.98:0/1624124214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40a3de0 0x7f9eb413b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45256/0 (socket says 172.21.15.98:45256) 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): get_auth_request con 0x7f9eb40a41c0 auth_method 0 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 10 monclient(hunting): get_auth_request con 0x7f9eb40a3de0 auth_method 0 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb2ffd640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eb409b240 0x7f9eb413ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient(hunting): handle_auth_done global_id 23926 payload 824 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eb409b240 msgr2=0x7f9eb413ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9eb409b240 0x7f9eb413ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40a3de0 msgr2=0x7f9eb413b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40a3de0 0x7f9eb413b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient: _finish_hunting 0 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 monclient: found mon.noname-c 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb37fe640 1 -- 172.21.15.98:0/1624124214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eb412e260 con 0x7f9eb40a41c0 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.838+0000 7f9eb27fc640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb40a3de0 0x7f9eb413b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb37fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9eb40a41c0 0x7f9eb413b6e0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00e930 tx=0x7f9eac00edf0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 client.?.objecter ms_handle_connect 0x7f9eb40a41c0 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9eac017070 con 0x7f9eb40a41c0 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: _finish_auth 0 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:58:49.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 5 monclient: authenticate success, global_id 23926 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eac00cd50 con 0x7f9eb40a41c0 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 10 monclient: _renew_subs 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eb40a4610 con 0x7f9eb40a41c0 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9eac0138e0 con 0x7f9eb40a41c0 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 10 client.23926.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 10 monclient: _renew_subs 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb17fa640 4 set_mon_vals no callback set 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eb413c500 con 0x7f9eb40a41c0 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 1 librados: init done 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41cf730 0x7f9eb41cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb413d060 con 0x7f9eb41cf730 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9eac024060 con 0x7f9eb40a41c0 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41cf730 0x7f9eb41cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient: get_auth_request con 0x7f9eb41cf730 auth_method 0 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9ea003df40 0x7f9ea0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_done global_id 64109 payload 824 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(360..360 src has 1..360) ==== 6202+0+0 (secure 0 0 0) 0x7f9eac00f040 con 0x7f9eb40a41c0 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 client.23926.objecter ms_dispatch 0x7f9eb4000ba0 osd_map(360..360 src has 1..360) 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 3 client.23926.objecter handle_osd_map got epochs [360,360] > 0 2024-09-11T02:58:49.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 3 client.23926.objecter handle_osd_map decoding full epoch 360 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 1 -- 172.21.15.98:0/1624124214 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9eb40a0570 con 0x7f9eb41cf730 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 20 client.23926.objecter dump_active .. 0 homeless 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 client.23926.objecter ms_handle_connect 0x7f9eb41cf730 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 client.23926.objecter resend_mon_ops 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb2ffd640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41cf730 0x7f9eb41cfb10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4007d90 tx=0x7f9ea400f520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb27fc640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9ea003df40 0x7f9ea0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea4018020 con 0x7f9eb41cf730 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb27fc640 10 monclient: get_auth_request con 0x7f9ea003df40 auth_method 0 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9ea40143f0 con 0x7f9eb41cf730 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41cf730 msgr2=0x7f9eb41cfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41cf730 0x7f9eb41cfb10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4007d90 tx=0x7f9ea400f520 comp rx=0 tx=0).stop 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41cf730 msgr2=0x7f9eb41cfb10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41cf730 0x7f9eb41cfb10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9e9b7fe640 10 client.23926.objecter ms_handle_connect 0x7f9ea003df40 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.842+0000 7f9eb27fc640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9ea003df40 0x7f9ea0040400 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80ad000 tx=0x7f9ea80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41d0050 0x7f9eb40992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:49.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9eb406ca10 con 0x7f9eb41d0050 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41d0050 0x7f9eb40992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient: get_auth_request con 0x7f9eb41d0050 auth_method 0 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 10 monclient(hunting): handle_auth_done global_id 64118 payload 824 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 1 -- 172.21.15.98:0/1624124214 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9eb412db80 con 0x7f9eb41d0050 2024-09-11T02:58:49.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 10 client.23926.objecter ms_handle_connect 0x7f9eb41d0050 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb2ffd640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41d0050 0x7f9eb40992a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4007d90 tx=0x7f9ea4004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 10 client.23926.objecter resend_mon_ops 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9ea4018040 con 0x7f9eb41d0050 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 10 monclient: dump: 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:49.873 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f9ea4029400 con 0x7f9eb41d0050 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41d0050 msgr2=0x7f9eb40992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41d0050 0x7f9eb40992a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4007d90 tx=0x7f9ea4004a50 comp rx=0 tx=0).stop 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41d0050 msgr2=0x7f9eb40992a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9e9b7fe640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9eb41d0050 0x7f9eb40992a0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4007d90 tx=0x7f9ea4004a50 comp rx=0 tx=0).stop 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:49.875 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 22043, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:49.876 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:49.877 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:49.878 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:49.879 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:49.880 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.883 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:49.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:49.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:49.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:49.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:49.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:49.895 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:49.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:49.897 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:49.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:49.899 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 10 librados: watch_flush enter 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 10 librados: watch_flush exit 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 20 client.23926.objecter shutdown clearing up homeless session... 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 10 client.23926.objecter successfully canceled tick 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9ea003df40 msgr2=0x7f9ea0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9ea003df40 0x7f9ea0040400 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7f9ea80ad000 tx=0x7f9ea80a6000 comp rx=0 tx=0).stop 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 10 monclient: shutdown 2024-09-11T02:58:49.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9eb40a41c0 msgr2=0x7f9eb413b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 --2- 172.21.15.98:0/1624124214 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9eb40a41c0 0x7f9eb413b6e0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f9eac00e930 tx=0x7f9eac00edf0 comp rx=0 tx=0).stop 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb37fe640 1 -- 172.21.15.98:0/1624124214 reap_dead start 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb37fe640 1 -- 172.21.15.98:0/1624124214 reap_dead start 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 shutdown_connections 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 >> 172.21.15.98:0/1624124214 conn(0x7f9eb40a4ed0 msgr2=0x7f9eb40a62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 shutdown_connections 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 -- 172.21.15.98:0/1624124214 wait complete. 2024-09-11T02:58:49.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:49.850+0000 7f9eb9664640 1 librados: shutdown 2024-09-11T02:58:49.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:49.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:58:49.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:58:50.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient: get_monmap_and_config 2024-09-11T02:58:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient: monmap: 2024-09-11T02:58:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:50.013757+0000 2024-09-11T02:58:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:50.013757+0000 2024-09-11T02:58:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.014 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.014 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.014 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:50.014 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:50.014 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 Processor -- start 2024-09-11T02:58:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 -- start start 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient: init 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient: _add_conns ranks=[8,9,3,5,0,4,6,7,1,10,2] 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): picked mon.noname-e con 0x7f1f8c0a7030 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f1f8c06f720 0x7f1f8c13a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c06f720 addr v2:172.21.15.98:3300/0 2024-09-11T02:58:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c0a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): picked mon.noname-g con 0x7f1f8c0a5830 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f1f8c13a620 con 0x7f1f8c06f720 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8c0a7410 con 0x7f1f8c0a5830 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1f8c06ee20 con 0x7f1f8c0a7030 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f92035640 10 monclient(hunting): authenticate will time out at 1483.893981s 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c139ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39618/0 (socket says 172.21.15.98:39618) 2024-09-11T02:58:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/56275569 learned_addr learned my addr 172.21.15.98:0/56275569 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): get_auth_request con 0x7f1f8c0a7030 auth_method 0 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8affd640 1 --2- 172.21.15.98:0/56275569 >> v2:172.21.15.98:3300/0 conn(0x7f1f8c06f720 0x7f1f8c13a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8affd640 10 monclient(hunting): get_auth_request con 0x7f1f8c06f720 auth_method 0 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8bfff640 1 --2- 172.21.15.98:0/56275569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c0a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8bfff640 10 monclient(hunting): get_auth_request con 0x7f1f8c0a5830 auth_method 0 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_done global_id 24470 payload 274 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/56275569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 msgr2=0x7f1f8c0a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 --2- 172.21.15.98:0/56275569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c0a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/56275569 >> v2:172.21.15.98:3300/0 conn(0x7f1f8c06f720 msgr2=0x7f1f8c13a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 --2- 172.21.15.98:0/56275569 >> v2:172.21.15.98:3300/0 conn(0x7f1f8c06f720 0x7f1f8c13a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 monclient: found mon.noname-e 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.010+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/56275569 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c06f4e0 con 0x7f1f8c0a7030 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8b7fe640 1 --2- 172.21.15.98:0/56275569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c139ba0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0098a0 tx=0x7f1f7c01c3d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a665701e01b47a8b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 1 -- 172.21.15.98:0/56275569 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f7c0b0070 con 0x7f1f8c0a7030 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: dump: 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 1 -- 172.21.15.98:0/56275569 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0044a0 con 0x7f1f8c0a7030 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 1 -- 172.21.15.98:0/56275569 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f7c003a90 con 0x7f1f8c0a7030 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f8a7fc640 10 monclient: dump: 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 5 monclient: authenticate success, global_id 24470 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 4 set_mon_vals no callback set 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 10 monclient: shutdown 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 -- 172.21.15.98:0/56275569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 msgr2=0x7f1f8c139ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 --2- 172.21.15.98:0/56275569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c139ba0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0098a0 tx=0x7f1f7c01c3d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 -- 172.21.15.98:0/56275569 shutdown_connections 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 --2- 172.21.15.98:0/56275569 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c0a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 --2- 172.21.15.98:0/56275569 >> v2:172.21.15.98:3300/0 conn(0x7f1f8c06f720 0x7f1f8c13a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 --2- 172.21.15.98:0/56275569 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c139ba0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 -- 172.21.15.98:0/56275569 >> 172.21.15.98:0/56275569 conn(0x7f1f8c0aa420 msgr2=0x7f1f8c0a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 -- 172.21.15.98:0/56275569 shutdown_connections 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 -- 172.21.15.98:0/56275569 wait complete. 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 10 monclient: monmap: 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.037 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 librados: starting msgr at 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 librados: starting objecter 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 Processor -- start 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 -- start start 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 librados: setting wanted keys 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 1 librados: calling monclient init 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.014+0000 7f1f92035640 10 monclient: init 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient: _add_conns ranks=[6,3,8,7,0,1,2,5,4] 2024-09-11T02:58:50.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1f8c06f720 0x7f1f8c1340b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): picked mon.noname-d con 0x7f1f8c06f720 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c1345f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): picked mon.noname-g con 0x7f1f8c0a5830 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c13c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): picked mon.noname-e con 0x7f1f8c0a7030 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8c143230 con 0x7f1f8c0a5830 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1f8c06fce0 con 0x7f1f8c06f720 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1f8c1433b0 con 0x7f1f8c0a7030 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): authenticate will time out at 1483.900536s 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1f8c06f720 0x7f1f8c1340b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1f8c06f720 0x7f1f8c1340b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44500/0 (socket says 172.21.15.98:44500) 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3581846417 learned_addr learned my addr 172.21.15.98:0/3581846417 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): get_auth_request con 0x7f1f8c06f720 auth_method 0 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c13c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient(hunting): get_auth_request con 0x7f1f8c0a7030 auth_method 0 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c1345f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): get_auth_request con 0x7f1f8c0a5830 auth_method 0 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient(hunting): handle_auth_done global_id 44313 payload 824 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 msgr2=0x7f1f8c13c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c13c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 msgr2=0x7f1f8c1345f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c1345f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 monclient: found mon.noname-d 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 -- 172.21.15.98:0/3581846417 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c1356e0 con 0x7f1f8c06f720 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1f8c0a7030 0x7f1f8c13c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1f8c0a5830 0x7f1f8c1345f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8b7fe640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1f8c06f720 0x7f1f8c1340b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0ae040 tx=0x7f1f7c019170 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 client.?.objecter ms_handle_connect 0x7f1f8c06f720 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f7c0b9020 con 0x7f1f8c06f720 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: dump: 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.043 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0b1040 con 0x7f1f8c06f720 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f7c0b0070 con 0x7f1f8c06f720 2024-09-11T02:58:50.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 5 monclient: authenticate success, global_id 44313 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient: _renew_subs 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c13ff90 con 0x7f1f8c06f720 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f89ffb640 4 set_mon_vals no callback set 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: dump: 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.045 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 client.44313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient: _renew_subs 2024-09-11T02:58:50.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8c0a7db0 con 0x7f1f8c06f720 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 librados: init done 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13c990 0x7f1f8c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1f8c06ca10 con 0x7f1f8c13c990 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13c990 0x7f1f8c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient: get_auth_request con 0x7f1f8c13c990 auth_method 0 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 10 monclient(hunting): handle_auth_done global_id 23935 payload 824 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1f7c003a70 con 0x7f1f8c06f720 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 1 -- 172.21.15.98:0/3581846417 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f8c0a5cb0 con 0x7f1f8c13c990 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8affd640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13c990 0x7f1f8c13cd70 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0a0ef0 tx=0x7f1f7400f9e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1f6c03df40 0x7f1f6c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 client.44313.objecter ms_handle_connect 0x7f1f8c13c990 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 client.44313.objecter resend_mon_ops 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(360..360 src has 1..360) ==== 6202+0+0 (secure 0 0 0) 0x7f1f7c0e8cf0 con 0x7f1f8c06f720 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 client.44313.objecter ms_dispatch 0x7f1f8c000ba0 osd_map(360..360 src has 1..360) 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 3 client.44313.objecter handle_osd_map got epochs [360,360] > 0 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 3 client.44313.objecter handle_osd_map decoding full epoch 360 2024-09-11T02:58:50.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 20 client.44313.objecter dump_active .. 0 homeless 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f740044a0 con 0x7f1f8c13c990 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: dump: 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.049 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1f74014670 con 0x7f1f8c13c990 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13c990 msgr2=0x7f1f8c13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13c990 0x7f1f8c13cd70 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0a0ef0 tx=0x7f1f7400f9e0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13c990 msgr2=0x7f1f8c13cd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f83fff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13c990 0x7f1f8c13cd70 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1f6c03df40 0x7f1f6c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.018+0000 7f1f8bfff640 10 monclient: get_auth_request con 0x7f1f6c03df40 auth_method 0 2024-09-11T02:58:50.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.022+0000 7f1f8bfff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1f6c03df40 0x7f1f6c040400 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c134f60 tx=0x7f1f84009820 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.022+0000 7f1f83fff640 10 client.44313.objecter ms_handle_connect 0x7f1f6c03df40 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13d2b0 0x7f1f8c1358f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1f8c06e870 con 0x7f1f8c13d2b0 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13d2b0 0x7f1f8c1358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient: get_auth_request con 0x7f1f8c13d2b0 auth_method 0 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 10 monclient(hunting): handle_auth_done global_id 23944 payload 824 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 1 -- 172.21.15.98:0/3581846417 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f8c069090 con 0x7f1f8c13d2b0 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 10 client.44313.objecter ms_handle_connect 0x7f1f8c13d2b0 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 10 client.44313.objecter resend_mon_ops 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8affd640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13d2b0 0x7f1f8c1358f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0a0ef0 tx=0x7f1f740046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f74014e20 con 0x7f1f8c13d2b0 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 10 monclient: dump: 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f1f74014e20 con 0x7f1f8c13d2b0 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13d2b0 msgr2=0x7f1f8c1358f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13d2b0 0x7f1f8c1358f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0a0ef0 tx=0x7f1f740046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13d2b0 msgr2=0x7f1f8c1358f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f83fff640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1f8c13d2b0 0x7f1f8c1358f0 secure :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0a0ef0 tx=0x7f1f740046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:50.056 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 545811, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:50.057 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.058 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.059 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:50.060 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.061 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.062 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.064 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.065 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.066 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.067 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:50.070 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.074 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.076 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:50.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:50.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:50.080 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 10 librados: watch_flush enter 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 10 librados: watch_flush exit 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 20 client.44313.objecter shutdown clearing up homeless session... 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 10 client.44313.objecter successfully canceled tick 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1f6c03df40 msgr2=0x7f1f6c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1f6c03df40 0x7f1f6c040400 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c134f60 tx=0x7f1f84009820 comp rx=0 tx=0).stop 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 10 monclient: shutdown 2024-09-11T02:58:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1f8c06f720 msgr2=0x7f1f8c1340b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f8bfff640 1 -- 172.21.15.98:0/3581846417 reap_dead start 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1f8c06f720 0x7f1f8c1340b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0ae040 tx=0x7f1f7c019170 comp rx=0 tx=0).stop 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 shutdown_connections 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 --2- 172.21.15.98:0/3581846417 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1f8c06f720 0x7f1f8c1340b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 >> 172.21.15.98:0/3581846417 conn(0x7f1f8c0aa420 msgr2=0x7f1f8c0a86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 shutdown_connections 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 -- 172.21.15.98:0/3581846417 wait complete. 2024-09-11T02:58:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.026+0000 7f1f92035640 1 librados: shutdown 2024-09-11T02:58:50.083 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:50.084 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:58:50.084 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:58:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient: get_monmap_and_config 2024-09-11T02:58:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient: monmap: 2024-09-11T02:58:50.199 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:50.201435+0000 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:50.201435+0000 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.200 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.201 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.201 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.201 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.201 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:50.201 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:50.201 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 Processor -- start 2024-09-11T02:58:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 -- start start 2024-09-11T02:58:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient: init 2024-09-11T02:58:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient: _add_conns ranks=[8,3,2,9,0,1,5,7,6,10,4] 2024-09-11T02:58:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f88380a46a0 0x7f883809d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): picked mon.noname-e con 0x7f88380a46a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a4a80 0x7f88380a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): picked mon.noname-g con 0x7f88380a4a80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8838137460 0x7f88381378c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): picked mon.noname-b con 0x7f8838137460 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88380770a0 con 0x7f8838137460 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f88380a3160 con 0x7f88380a4a80 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f883809caf0 con 0x7f88380a46a0 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f883e228640 10 monclient(hunting): authenticate will time out at 1484.081771s 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f88380a46a0 0x7f883809d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f88380a46a0 0x7f883809d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39628/0 (socket says 172.21.15.98:39628) 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 -- 172.21.15.98:0/1327164955 learned_addr learned my addr 172.21.15.98:0/1327164955 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): get_auth_request con 0x7f88380a46a0 auth_method 0 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8836ffd640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a4a80 0x7f88380a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8836ffd640 10 monclient(hunting): get_auth_request con 0x7f88380a4a80 auth_method 0 2024-09-11T02:58:50.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8837fff640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8838137460 0x7f88381378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8836ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8836ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8837fff640 10 monclient(hunting): get_auth_request con 0x7f8838137460 auth_method 0 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8837fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8837fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f8837fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient(hunting): handle_auth_done global_id 24488 payload 274 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 -- 172.21.15.98:0/1327164955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a4a80 msgr2=0x7f88380a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a4a80 0x7f88380a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 -- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8838137460 msgr2=0x7f88381378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8838137460 0x7f88381378c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 monclient: found mon.noname-e 2024-09-11T02:58:50.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:50.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.198+0000 7f88377fe640 1 -- 172.21.15.98:0/1327164955 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883809cdd0 con 0x7f88380a46a0 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88377fe640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f88380a46a0 0x7f883809d4d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e80 tx=0x7f883001c8d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=52b65bbbb9a913c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 1 -- 172.21.15.98:0/1327164955 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88300090d0 con 0x7f88380a46a0 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: dump: 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.215 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 5 monclient: authenticate success, global_id 24488 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 1 -- 172.21.15.98:0/1327164955 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88300a5b80 con 0x7f88380a46a0 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 4 set_mon_vals no callback set 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 1 -- 172.21.15.98:0/1327164955 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88300a5d70 con 0x7f88380a46a0 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88367fc640 10 monclient: dump: 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.217 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient: shutdown 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f88380a46a0 msgr2=0x7f883809d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f88380a46a0 0x7f883809d4d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e80 tx=0x7f883001c8d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- 172.21.15.98:0/1327164955 shutdown_connections 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8838137460 0x7f88381378c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a4a80 0x7f88380a4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 --2- 172.21.15.98:0/1327164955 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f88380a46a0 0x7f883809d4d0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- 172.21.15.98:0/1327164955 >> 172.21.15.98:0/1327164955 conn(0x7f883806ed00 msgr2=0x7f883806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- 172.21.15.98:0/1327164955 shutdown_connections 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- 172.21.15.98:0/1327164955 wait complete. 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient: monmap: 2024-09-11T02:58:50.219 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.220 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 librados: starting msgr at 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 librados: starting objecter 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 Processor -- start 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- start start 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 librados: setting wanted keys 2024-09-11T02:58:50.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 librados: calling monclient init 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient: init 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient: _add_conns ranks=[3,2,6,8,4,5,7,0,1] 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a46a0 0x7f88381d1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): picked mon.noname-g con 0x7f88380a46a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88380a4a80 0x7f88381d1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): picked mon.noname-b con 0x7f88380a4a80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838137460 0x7f88381d1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): picked mon.noname-d con 0x7f8838137460 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:50.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f88380a53a0 con 0x7f88380a4a80 2024-09-11T02:58:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8838141150 con 0x7f88380a46a0 2024-09-11T02:58:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f883813d160 con 0x7f8838137460 2024-09-11T02:58:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f883e228640 10 monclient(hunting): authenticate will time out at 1484.087030s 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88380a4a80 0x7f88381d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838137460 0x7f88381d1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88380a4a80 0x7f88381d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45292/0 (socket says 172.21.15.98:45292) 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): get_auth_request con 0x7f8838137460 auth_method 0 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 1 -- 172.21.15.98:0/2350747154 learned_addr learned my addr 172.21.15.98:0/2350747154 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 10 monclient(hunting): get_auth_request con 0x7f88380a4a80 auth_method 0 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88377fe640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a46a0 0x7f88381d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88377fe640 10 monclient(hunting): get_auth_request con 0x7f88380a46a0 auth_method 0 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient(hunting): handle_auth_done global_id 44322 payload 824 2024-09-11T02:58:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a46a0 msgr2=0x7f88381d1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a46a0 0x7f88381d1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88380a4a80 msgr2=0x7f88381d1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88380a4a80 0x7f88381d1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 monclient: found mon.noname-d 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 -- 172.21.15.98:0/2350747154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8838123b00 con 0x7f8838137460 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8836ffd640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f88380a4a80 0x7f88381d1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:50.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88377fe640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f88380a46a0 0x7f88381d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f8837fff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838137460 0x7f88381d1d90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f882800d480 tx=0x7f882800d940 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 client.?.objecter ms_handle_connect 0x7f8838137460 2024-09-11T02:58:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8828012a60 con 0x7f8838137460 2024-09-11T02:58:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 monclient: dump: 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.228 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8828012c40 con 0x7f8838137460 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8828012df0 con 0x7f8838137460 2024-09-11T02:58:50.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f882ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 monclient: dump: 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.230 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.202+0000 7f88357fa640 4 set_mon_vals no callback set 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 5 monclient: authenticate success, global_id 44322 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 10 monclient: _renew_subs 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88380a15e0 con 0x7f8838137460 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 10 client.44322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 10 monclient: _renew_subs 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f883809d640 con 0x7f8838137460 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 1 librados: init done 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838135eb0 0x7f8838136290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f88380a5520 con 0x7f8838135eb0 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838135eb0 0x7f8838136290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient: get_auth_request con 0x7f8838135eb0 auth_method 0 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8828013070 con 0x7f8838137460 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 10 monclient(hunting): handle_auth_done global_id 44331 payload 824 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 1 -- 172.21.15.98:0/2350747154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8838123e40 con 0x7f8838135eb0 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f881803df40 0x7f8818040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(360..360 src has 1..360) ==== 6202+0+0 (secure 0 0 0) 0x7f8828019020 con 0x7f8838137460 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 client.44322.objecter ms_dispatch 0x7f8838000ba0 osd_map(360..360 src has 1..360) 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 3 client.44322.objecter handle_osd_map got epochs [360,360] > 0 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 3 client.44322.objecter handle_osd_map decoding full epoch 360 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f88377fe640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838135eb0 0x7f8838136290 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f8830019670 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 20 client.44322.objecter dump_active .. 0 homeless 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 client.44322.objecter ms_handle_connect 0x7f8838135eb0 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 client.44322.objecter resend_mon_ops 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88300bd030 con 0x7f8838135eb0 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 monclient: dump: 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.234 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f8836ffd640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f881803df40 0x7f8818040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f8836ffd640 10 monclient: get_auth_request con 0x7f881803df40 auth_method 0 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8830004040 con 0x7f8838135eb0 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838135eb0 msgr2=0x7f8838136290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838135eb0 0x7f8838136290 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f8830019670 comp rx=0 tx=0).stop 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838135eb0 msgr2=0x7f8838136290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838135eb0 0x7f8838136290 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f8830019670 comp rx=0 tx=0).stop 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f8836ffd640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f881803df40 0x7f8818040400 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f883813cdd0 tx=0x7f8820008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.206+0000 7f882ffff640 10 client.44322.objecter ms_handle_connect 0x7f881803df40 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f88381367d0 0x7f88381386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f883806ca10 con 0x7f88381367d0 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f88381367d0 0x7f88381386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient: get_auth_request con 0x7f88381367d0 auth_method 0 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 10 monclient(hunting): handle_auth_done global_id 44340 payload 824 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 1 -- 172.21.15.98:0/2350747154 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88381241e0 con 0x7f88381367d0 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 10 client.44322.objecter ms_handle_connect 0x7f88381367d0 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 10 client.44322.objecter resend_mon_ops 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f88377fe640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f88381367d0 0x7f88381386f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e80 tx=0x7f88300a7000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88300bd040 con 0x7f88381367d0 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 10 monclient: dump: 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.241 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f8830004850 con 0x7f88381367d0 2024-09-11T02:58:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f88381367d0 msgr2=0x7f88381386f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f88381367d0 0x7f88381386f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e80 tx=0x7f88300a7000 comp rx=0 tx=0).stop 2024-09-11T02:58:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f88381367d0 msgr2=0x7f88381386f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f882ffff640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f88381367d0 0x7f88381386f0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e80 tx=0x7f88300a7000 comp rx=0 tx=0).stop 2024-09-11T02:58:50.243 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:50.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:50.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:50.243 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:50.243 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:50.243 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 156642, 2024-09-11T02:58:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:50.244 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:50.245 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:50.245 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:50.245 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:50.245 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.245 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:58:50.245 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.246 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:50.246 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:50.246 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.246 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.246 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.246 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:50.247 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:50.248 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.248 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.248 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.248 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.248 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.248 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:50.249 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.250 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.251 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:50.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:50.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:50.256 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:50.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:50.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.259 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:50.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:50.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.265 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.267 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.269 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.270 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.270 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:50.270 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:50.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.271 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.271 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.271 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:50.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:50.272 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 10 librados: watch_flush enter 2024-09-11T02:58:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 10 librados: watch_flush exit 2024-09-11T02:58:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 20 client.44322.objecter shutdown clearing up homeless session... 2024-09-11T02:58:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 10 client.44322.objecter successfully canceled tick 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f881803df40 msgr2=0x7f8818040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f881803df40 0x7f8818040400 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f883813cdd0 tx=0x7f8820008040 comp rx=0 tx=0).stop 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 10 monclient: shutdown 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838137460 msgr2=0x7f88381d1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 --2- 172.21.15.98:0/2350747154 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8838137460 0x7f88381d1d90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f882800d480 tx=0x7f882800d940 comp rx=0 tx=0).stop 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f8837fff640 1 -- 172.21.15.98:0/2350747154 reap_dead start 2024-09-11T02:58:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f8837fff640 1 -- 172.21.15.98:0/2350747154 reap_dead start 2024-09-11T02:58:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 shutdown_connections 2024-09-11T02:58:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 >> 172.21.15.98:0/2350747154 conn(0x7f883806ed00 msgr2=0x7f88380a75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 shutdown_connections 2024-09-11T02:58:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 -- 172.21.15.98:0/2350747154 wait complete. 2024-09-11T02:58:50.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.214+0000 7f883e228640 1 librados: shutdown 2024-09-11T02:58:50.277 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:50.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:58:50.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:58:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient: get_monmap_and_config 2024-09-11T02:58:50.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient: monmap: 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:50.389374+0000 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:50.389374+0000 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.388 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:50.389 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:50.389 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 1 Processor -- start 2024-09-11T02:58:50.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 1 -- start start 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient: init 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient: _add_conns ranks=[3,5,0,1,10,4,8,7,2,6,9] 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 0x7fc3e009d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient(hunting): picked mon.noname-g con 0x7fc3e00a46a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 0x7fc3e00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient(hunting): picked mon.noname-h con 0x7fc3e00a4a80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3e0137460 addr v1:172.21.15.98:6789/0 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e00779a0 con 0x7fc3e0137460 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e0070130 con 0x7fc3e00a46a0 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc3e0076fc0 con 0x7fc3e00a4a80 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 10 monclient(hunting): authenticate will time out at 1484.269701s 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e491b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc3e0137460 0x7fc3e0137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50008/0 (socket says 172.21.15.98:50008) 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e491b640 1 -- 172.21.15.98:0/4038315418 learned_addr learned my addr 172.21.15.98:0/4038315418 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 0x7fc3e009d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 0x7fc3e00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 722247697 0 0) 0x7fc3e00779a0 con 0x7fc3e0137460 2024-09-11T02:58:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): _negotiate 2024-09-11T02:58:50.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): my global_id is 44451 2024-09-11T02:58:50.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3c8003530 con 0x7fc3e0137460 2024-09-11T02:58:50.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 10 monclient(hunting): get_auth_request con 0x7fc3e00a4a80 auth_method 0 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): get_auth_request con 0x7fc3e00a46a0 auth_method 0 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 960767618 0 0) 0x7fc3c8003530 con 0x7fc3e0137460 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 msgr2=0x7fc3e00a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 0x7fc3e00a4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 msgr2=0x7fc3e009d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 0x7fc3e009d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:58:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e00a53a0 con 0x7fc3e0137460 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fc3d8001cf0 con 0x7fc3e0137460 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: dump: 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 >> v1:172.21.15.98:6789/0 conn(0x7fc3e0137460 legacy=0x7fc3e0137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient: _add_conns ranks=[2,5,3,8,0,7,6,1,4] 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3c8004210 0x7fc3c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): picked mon.b con 0x7fc3c8004210 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 0x7fc3c8004d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): picked mon.h con 0x7fc3e00a4a80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 0x7fc3c800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): picked mon.g con 0x7fc3e00a46a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e0070130 con 0x7fc3c8004210 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e0076fc0 con 0x7fc3e00a46a0 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc3c8005530 con 0x7fc3e00a4a80 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3deffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:58:50.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3c8004210 0x7fc3c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): get_auth_request con 0x7fc3c8004210 auth_method 0 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:58:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient(hunting): handle_auth_done global_id 44451 payload 293 2024-09-11T02:58:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 -- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 msgr2=0x7fc3c8004d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e491b640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 0x7fc3c800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 0x7fc3c8004d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 -- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 msgr2=0x7fc3c800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 0x7fc3c800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 monclient: found mon.b 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 1 -- 172.21.15.98:0/4038315418 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c8003530 con 0x7fc3c8004210 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 5 monclient: authenticate success, global_id 44451 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.386+0000 7fc3e637e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3c8004210 0x7fc3c8004810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009ae0 tx=0x7fc3d001ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c126611678ed0c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3d00096b0 con 0x7fc3c8004210 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 10 monclient: dump: 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.417 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d00a68a0 con 0x7fc3c8004210 2024-09-11T02:58:50.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 1 -- 172.21.15.98:0/4038315418 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3d00a7410 con 0x7fc3c8004210 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3deffd640 10 monclient: dump: 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.420 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.421 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.421 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:50.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 4 set_mon_vals no callback set 2024-09-11T02:58:50.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient: shutdown 2024-09-11T02:58:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- 172.21.15.98:0/4038315418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3c8004210 msgr2=0x7fc3c8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3c8004210 0x7fc3c8004810 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009ae0 tx=0x7fc3d001ca50 comp rx=0 tx=0).stop 2024-09-11T02:58:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- 172.21.15.98:0/4038315418 shutdown_connections 2024-09-11T02:58:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc3e00a46a0 0x7fc3c800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc3e00a4a80 0x7fc3c8004d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 --2- 172.21.15.98:0/4038315418 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc3c8004210 0x7fc3c8004810 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- 172.21.15.98:0/4038315418 >> 172.21.15.98:0/4038315418 conn(0x7fc3e006ed00 msgr2=0x7fc3e006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- 172.21.15.98:0/4038315418 shutdown_connections 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- 172.21.15.98:0/4038315418 wait complete. 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient: monmap: 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 librados: starting msgr at 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 librados: starting objecter 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 Processor -- start 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- start start 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 librados: setting wanted keys 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 librados: calling monclient init 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient: init 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient: _add_conns ranks=[0,1,4,7,6,3,2,8,5] 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c8004210 0x7fc3e01cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c8004210 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3e00a46a0 0x7fc3e0136530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): picked mon.noname-f con 0x7fc3e00a46a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc3e00a4a80 0x7fc3e0136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e00a4a80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e009d800 con 0x7fc3c8004210 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e00517e0 con 0x7fc3e00a46a0 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc3e0123ef0 con 0x7fc3e00a4a80 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e637e640 10 monclient(hunting): authenticate will time out at 1484.275338s 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c8004210 0x7fc3e01cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc3e00a4a80 0x7fc3e0136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c8004210 0x7fc3e01cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38616/0 (socket says 172.21.15.98:38616) 2024-09-11T02:58:50.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 -- 172.21.15.98:0/1069914389 learned_addr learned my addr 172.21.15.98:0/1069914389 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): get_auth_request con 0x7fc3c8004210 auth_method 0 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 10 monclient(hunting): get_auth_request con 0x7fc3e00a4a80 auth_method 0 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3e491b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3df7fe640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3e00a46a0 0x7fc3e0136530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient(hunting): handle_auth_done global_id 44460 payload 824 2024-09-11T02:58:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc3e00a4a80 msgr2=0x7fc3e0136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc3e00a4a80 0x7fc3e0136a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3e00a46a0 msgr2=0x7fc3e0136530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3e00a46a0 0x7fc3e0136530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 monclient: found mon.noname-a 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 -- 172.21.15.98:0/1069914389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e00a40a0 con 0x7fc3c8004210 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3df7fe640 10 monclient: get_auth_request con 0x7fc3e00a46a0 auth_method 0 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3df7fe640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3e00a46a0 0x7fc3e0136530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dffff640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c8004210 0x7fc3e01cf210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc3d001c880 tx=0x7fc3c8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dcff9640 10 client.?.objecter ms_handle_connect 0x7fc3c8004210 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.390+0000 7fc3dcff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3d00044b0 con 0x7fc3c8004210 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: dump: 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.431 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d00a6da0 con 0x7fc3c8004210 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 5 monclient: authenticate success, global_id 44460 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 10 monclient: _renew_subs 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e0123930 con 0x7fc3c8004210 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3d0003a90 con 0x7fc3c8004210 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:50.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3de7fc640 4 set_mon_vals no callback set 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 10 client.44460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: dump: 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.434 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 10 monclient: _renew_subs 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e01200f0 con 0x7fc3c8004210 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 1 librados: init done 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0137460 0x7fc3e0063e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc3e006ca10 con 0x7fc3e0137460 2024-09-11T02:58:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0137460 0x7fc3e0063e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient: get_auth_request con 0x7fc3e0137460 auth_method 0 2024-09-11T02:58:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc3d00c1030 con 0x7fc3c8004210 2024-09-11T02:58:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_done global_id 24497 payload 824 2024-09-11T02:58:50.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 1 -- 172.21.15.98:0/1069914389 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3e0074030 con 0x7fc3e0137460 2024-09-11T02:58:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc3cc03df40 0x7fc3cc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(360..360 src has 1..360) ==== 6202+0+0 (secure 0 0 0) 0x7fc3d000f040 con 0x7fc3c8004210 2024-09-11T02:58:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 client.44460.objecter ms_dispatch 0x7fc3e0000ba0 osd_map(360..360 src has 1..360) 2024-09-11T02:58:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 3 client.44460.objecter handle_osd_map got epochs [360,360] > 0 2024-09-11T02:58:50.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 3 client.44460.objecter handle_osd_map decoding full epoch 360 2024-09-11T02:58:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3df7fe640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0137460 0x7fc3e0063e00 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0122200 tx=0x7fc3d4010c90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 20 client.44460.objecter dump_active .. 0 homeless 2024-09-11T02:58:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 client.44460.objecter ms_handle_connect 0x7fc3e0137460 2024-09-11T02:58:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 client.44460.objecter resend_mon_ops 2024-09-11T02:58:50.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3d400f040 con 0x7fc3e0137460 2024-09-11T02:58:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: dump: 2024-09-11T02:58:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e491b640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc3cc03df40 0x7fc3cc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e491b640 10 monclient: get_auth_request con 0x7fc3cc03df40 auth_method 0 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc3d401c400 con 0x7fc3e0137460 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0137460 msgr2=0x7fc3e0063e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0137460 0x7fc3e0063e00 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0122200 tx=0x7fc3d4010c90 comp rx=0 tx=0).stop 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0137460 msgr2=0x7fc3e0063e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0137460 0x7fc3e0063e00 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3dcff9640 10 client.44460.objecter ms_handle_connect 0x7fc3cc03df40 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.394+0000 7fc3e491b640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc3cc03df40 0x7fc3cc040400 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00a7f00 tx=0x7fc3d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3e637e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3e637e640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0064340 0x7fc3e002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3e637e640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc3e006e870 con 0x7fc3e0064340 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0064340 0x7fc3e002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 10 monclient: get_auth_request con 0x7fc3e0064340 auth_method 0 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.398+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3df7fe640 10 monclient(hunting): handle_auth_done global_id 24506 payload 824 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3df7fe640 1 -- 172.21.15.98:0/1069914389 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3e0123b70 con 0x7fc3e0064340 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 10 client.44460.objecter ms_handle_connect 0x7fc3e0064340 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 10 client.44460.objecter resend_mon_ops 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3df7fe640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0064340 0x7fc3e002d450 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0122200 tx=0x7fc3d4009190 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3d400ff80 con 0x7fc3e0064340 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 10 monclient: dump: 2024-09-11T02:58:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9269 (secure 0 0 0) 0x7fc3d400ff80 con 0x7fc3e0064340 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0064340 msgr2=0x7fc3e002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0064340 0x7fc3e002d450 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0122200 tx=0x7fc3d4009190 comp rx=0 tx=0).stop 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0064340 msgr2=0x7fc3e002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3dcff9640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc3e0064340 0x7fc3e002d450 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.447 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:58:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 546195, 2024-09-11T02:58:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:50.448 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:58:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.449 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.450 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.451 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.452 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.453 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.460 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.463 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.464 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.465 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:50.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.468 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.469 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:50.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.473 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:50.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.476 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.477 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:50.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:50.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.483 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:50.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:50.485 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 10 librados: watch_flush enter 2024-09-11T02:58:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 10 librados: watch_flush exit 2024-09-11T02:58:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 20 client.44460.objecter shutdown clearing up homeless session... 2024-09-11T02:58:50.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 10 client.44460.objecter successfully canceled tick 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc3cc03df40 msgr2=0x7fc3cc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc3cc03df40 0x7fc3cc040400 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7fc3e00a7f00 tx=0x7fc3d8009040 comp rx=0 tx=0).stop 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 10 monclient: shutdown 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c8004210 msgr2=0x7fc3e01cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e491b640 1 -- 172.21.15.98:0/1069914389 reap_dead start 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c8004210 0x7fc3e01cf210 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc3d001c880 tx=0x7fc3c8005840 comp rx=0 tx=0).stop 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 shutdown_connections 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 --2- 172.21.15.98:0/1069914389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3c8004210 0x7fc3e01cf210 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 >> 172.21.15.98:0/1069914389 conn(0x7fc3e006ed00 msgr2=0x7fc3e00726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 shutdown_connections 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 -- 172.21.15.98:0/1069914389 wait complete. 2024-09-11T02:58:50.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.402+0000 7fc3e637e640 1 librados: shutdown 2024-09-11T02:58:50.489 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:50.489 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:58:50.489 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:58:50.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient: get_monmap_and_config 2024-09-11T02:58:50.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient: monmap: 2024-09-11T02:58:50.612 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.612 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.612 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:50.613740+0000 2024-09-11T02:58:50.612 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:50.613740+0000 2024-09-11T02:58:50.612 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:50.613 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 1 Processor -- start 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 1 -- start start 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient: init 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient: _add_conns ranks=[4,1,6,2,8,9,3,10,5,0,7] 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f32d4139b10 0x7f32d4072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient(hunting): picked mon.noname-c con 0x7f32d4139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f32d4139ef0 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32d413a2d0 0x7f32d40a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient(hunting): picked mon.noname-d con 0x7f32d413a2d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d40702b0 con 0x7f32d4139ef0 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f32d4071bf0 con 0x7f32d4139b10 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f32d4071350 con 0x7f32d413a2d0 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32dc3f7640 10 monclient(hunting): authenticate will time out at 1484.494083s 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32d413a2d0 0x7f32d40a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f32d4139b10 0x7f32d4072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32d413a2d0 0x7f32d40a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44544/0 (socket says 172.21.15.98:44544) 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f32d4139b10 0x7f32d4072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59698/0 (socket says 172.21.15.98:59698) 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 -- 172.21.15.98:0/271704320 learned_addr learned my addr 172.21.15.98:0/271704320 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): get_auth_request con 0x7f32d413a2d0 auth_method 0 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 10 monclient(hunting): get_auth_request con 0x7f32d4139b10 auth_method 0 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da193640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient(hunting): handle_auth_done global_id 44349 payload 274 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 -- 172.21.15.98:0/271704320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f32d4139b10 msgr2=0x7f32d4072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 --2- 172.21.15.98:0/271704320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f32d4139b10 0x7f32d4072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 -- 172.21.15.98:0/271704320 >> v1:172.21.15.190:6789/0 conn(0x7f32d4139ef0 legacy=0x7f32d40a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 monclient: found mon.noname-d 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.610+0000 7f32da994640 1 -- 172.21.15.98:0/271704320 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d40a9620 con 0x7f32d413a2d0 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da994640 1 --2- 172.21.15.98:0/271704320 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32d413a2d0 0x7f32d40a8120 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32c40098d0 tx=0x7f32c401c4d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6a15e384ddc28882 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 1 -- 172.21.15.98:0/271704320 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32c40b0070 con 0x7f32d413a2d0 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:50.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: dump: 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.622 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 1 -- 172.21.15.98:0/271704320 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c40aa810 con 0x7f32d413a2d0 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 1 -- 172.21.15.98:0/271704320 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32c40a9300 con 0x7f32d413a2d0 2024-09-11T02:58:50.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 5 monclient: authenticate success, global_id 44349 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 4 set_mon_vals no callback set 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9191640 10 monclient: dump: 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.624 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.625 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient: shutdown 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- 172.21.15.98:0/271704320 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32d413a2d0 msgr2=0x7f32d40a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 --2- 172.21.15.98:0/271704320 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32d413a2d0 0x7f32d40a8120 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f32c40098d0 tx=0x7f32c401c4d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- 172.21.15.98:0/271704320 shutdown_connections 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 --2- 172.21.15.98:0/271704320 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f32d413a2d0 0x7f32d40a8120 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 --2- 172.21.15.98:0/271704320 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f32d4139b10 0x7f32d4072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- 172.21.15.98:0/271704320 >> 172.21.15.98:0/271704320 conn(0x7f32d40ac790 msgr2=0x7f32d40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- 172.21.15.98:0/271704320 shutdown_connections 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- 172.21.15.98:0/271704320 wait complete. 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient: monmap: 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.627 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 librados: starting msgr at 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 librados: starting objecter 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 Processor -- start 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- start start 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 librados: setting wanted keys 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 librados: calling monclient init 2024-09-11T02:58:50.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient: init 2024-09-11T02:58:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient: _add_conns ranks=[8,3,1,6,4,0,2,7,5] 2024-09-11T02:58:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32d4139b10 0x7f32d40a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): picked mon.noname-e con 0x7f32d4139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d4139ef0 0x7f32d413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): picked mon.noname-g con 0x7f32d4139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413a2d0 0x7f32d413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): picked mon.noname-f con 0x7f32d413a2d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f32d4144470 con 0x7f32d413a2d0 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f32d40a8870 con 0x7f32d4139ef0 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f32d41445f0 con 0x7f32d4139b10 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32dc3f7640 10 monclient(hunting): authenticate will time out at 1484.499233s 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32d4139b10 0x7f32d40a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32d4139b10 0x7f32d40a4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39668/0 (socket says 172.21.15.98:39668) 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 -- 172.21.15.98:0/3063567446 learned_addr learned my addr 172.21.15.98:0/3063567446 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): get_auth_request con 0x7f32d4139b10 auth_method 0 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9992640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d4139ef0 0x7f32d413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9992640 10 monclient(hunting): get_auth_request con 0x7f32d4139ef0 auth_method 0 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9992640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9992640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient(hunting): handle_auth_done global_id 24515 payload 824 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d4139ef0 msgr2=0x7f32d413a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d4139ef0 0x7f32d413a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da994640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413a2d0 0x7f32d413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413a2d0 msgr2=0x7f32d413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413a2d0 0x7f32d413ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 monclient: found mon.noname-e 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da193640 1 -- 172.21.15.98:0/3063567446 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d4134890 con 0x7f32d4139b10 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da994640 10 monclient: get_auth_request con 0x7f32d413a2d0 auth_method 0 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32d9992640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f32d4139ef0 0x7f32d413a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.614+0000 7f32da994640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413a2d0 0x7f32d413ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32da193640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32d4139b10 0x7f32d40a4f80 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f32cc00e930 tx=0x7f32cc00edf0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 client.?.objecter ms_handle_connect 0x7f32d4139b10 2024-09-11T02:58:50.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32cc017070 con 0x7f32d4139b10 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: dump: 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.640 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32cc013470 con 0x7f32d4139b10 2024-09-11T02:58:50.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32cc013a90 con 0x7f32d4139b10 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 5 monclient: authenticate success, global_id 24515 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d8990640 4 set_mon_vals no callback set 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: dump: 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.642 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 10 monclient: _renew_subs 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d413d2c0 con 0x7f32d4139b10 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 10 client.24515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 10 monclient: _renew_subs 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d413b7d0 con 0x7f32d4139b10 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 1 librados: init done 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413b160 0x7f32d4133f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f32d413c7a0 con 0x7f32d413b160 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f32cc013c30 con 0x7f32d4139b10 2024-09-11T02:58:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f32c003df40 0x7f32c0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413b160 0x7f32d4133f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(360..360 src has 1..360) ==== 6202+0+0 (secure 0 0 0) 0x7f32cc00f040 con 0x7f32d4139b10 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 client.24515.objecter ms_dispatch 0x7f32d4000ba0 osd_map(360..360 src has 1..360) 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 3 client.24515.objecter handle_osd_map got epochs [360,360] > 0 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 3 client.24515.objecter handle_osd_map decoding full epoch 360 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient: get_auth_request con 0x7f32d413b160 auth_method 0 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 20 client.24515.objecter dump_active .. 0 homeless 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32da994640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f32c003df40 0x7f32c0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32da994640 10 monclient: get_auth_request con 0x7f32c003df40 auth_method 0 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 10 monclient(hunting): handle_auth_done global_id 25282 payload 824 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 1 -- 172.21.15.98:0/3063567446 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32d40aa310 con 0x7f32d413b160 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 client.24515.objecter ms_handle_connect 0x7f32d413b160 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 client.24515.objecter resend_mon_ops 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d9992640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413b160 0x7f32d4133f20 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f32c800ea70 tx=0x7f32c800ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 client.24515.objecter ms_handle_connect 0x7f32c003df40 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32da994640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f32c003df40 0x7f32c0040400 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7f32c40ad000 tx=0x7f32c40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32c80050d0 con 0x7f32d413b160 2024-09-11T02:58:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: dump: 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f32c8018070 con 0x7f32d413b160 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413b160 msgr2=0x7f32d4133f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413b160 0x7f32d4133f20 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f32c800ea70 tx=0x7f32c800ef30 comp rx=0 tx=0).stop 2024-09-11T02:58:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413b160 msgr2=0x7f32d4133f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.618+0000 7f32d2ffd640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d413b160 0x7f32d4133f20 secure :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f32c800ea70 tx=0x7f32c800ef30 comp rx=0 tx=0).stop 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32dc3f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32dc3f7640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d402d450 0x7f32d402d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32dc3f7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f32d406ca10 con 0x7f32d402d450 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d402d450 0x7f32d402d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient: get_auth_request con 0x7f32d402d450 auth_method 0 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 10 monclient(hunting): handle_auth_done global_id 25291 payload 824 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 1 -- 172.21.15.98:0/3063567446 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f32d413b540 con 0x7f32d402d450 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d9992640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d402d450 0x7f32d402d860 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009010 tx=0x7f32c8001260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 10 client.24515.objecter ms_handle_connect 0x7f32d402d450 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 10 client.24515.objecter resend_mon_ops 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32c8006e60 con 0x7f32d402d450 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 10 monclient: dump: 2024-09-11T02:58:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.653 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f32c801edb0 con 0x7f32d402d450 2024-09-11T02:58:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d402d450 msgr2=0x7f32d402d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d402d450 0x7f32d402d860 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009010 tx=0x7f32c8001260 comp rx=0 tx=0).stop 2024-09-11T02:58:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d402d450 msgr2=0x7f32d402d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.626+0000 7f32d2ffd640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32d402d450 0x7f32d402d860 secure :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009010 tx=0x7f32c8001260 comp rx=0 tx=0).stop 2024-09-11T02:58:50.655 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 546400, 2024-09-11T02:58:50.655 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:50.656 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:58:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.657 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.659 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.660 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.662 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:50.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:50.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:50.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:50.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:50.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.671 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.675 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:50.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.678 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.679 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.680 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.681 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:50.682 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 10 librados: watch_flush enter 2024-09-11T02:58:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 10 librados: watch_flush exit 2024-09-11T02:58:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 20 client.24515.objecter shutdown clearing up homeless session... 2024-09-11T02:58:50.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 10 client.24515.objecter successfully canceled tick 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f32c003df40 msgr2=0x7f32c0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f32c003df40 0x7f32c0040400 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7f32c40ad000 tx=0x7f32c40a6000 comp rx=0 tx=0).stop 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 10 monclient: shutdown 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32d4139b10 msgr2=0x7f32d40a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 --2- 172.21.15.98:0/3063567446 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f32d4139b10 0x7f32d40a4f80 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f32cc00e930 tx=0x7f32cc00edf0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32da994640 1 -- 172.21.15.98:0/3063567446 reap_dead start 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32da994640 1 -- 172.21.15.98:0/3063567446 reap_dead start 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 shutdown_connections 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 >> 172.21.15.98:0/3063567446 conn(0x7f32d40ac790 msgr2=0x7f32d4134aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 shutdown_connections 2024-09-11T02:58:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 -- 172.21.15.98:0/3063567446 wait complete. 2024-09-11T02:58:50.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.630+0000 7f32dc3f7640 1 librados: shutdown 2024-09-11T02:58:50.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:50.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:58:50.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:58:50.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient: get_monmap_and_config 2024-09-11T02:58:50.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient: monmap: 2024-09-11T02:58:50.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:50.809746+0000 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:50.809746+0000 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:50.809 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.810 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.810 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.810 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.810 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.810 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.811 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:50.811 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:50.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 Processor -- start 2024-09-11T02:58:50.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 -- start start 2024-09-11T02:58:50.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient: init 2024-09-11T02:58:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient: _add_conns ranks=[7,5,6,8,0,1,4,3,2,9,10] 2024-09-11T02:58:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0528139b80 0x7f0528139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): picked mon.noname-i con 0x7f0528139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f052806fe50 0x7f05280a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): picked mon.noname-h con 0x7f052806fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0528072640 0x7f05280a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): picked mon.noname-d con 0x7f0528072640 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f05280719c0 con 0x7f052806fe50 2024-09-11T02:58:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0528071c10 con 0x7f0528072640 2024-09-11T02:58:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f05280702d0 con 0x7f0528139b80 2024-09-11T02:58:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052f6ea640 10 monclient(hunting): authenticate will time out at 1484.689976s 2024-09-11T02:58:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0528072640 0x7f05280a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0528072640 0x7f05280a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44552/0 (socket says 172.21.15.98:44552) 2024-09-11T02:58:50.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 -- 172.21.15.98:0/740923672 learned_addr learned my addr 172.21.15.98:0/740923672 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): get_auth_request con 0x7f0528072640 auth_method 0 2024-09-11T02:58:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052cc85640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f052806fe50 0x7f05280a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052d486640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0528139b80 0x7f0528139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052cc85640 10 monclient(hunting): get_auth_request con 0x7f052806fe50 auth_method 0 2024-09-11T02:58:50.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052cc85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052cc85640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052cc85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052d486640 10 monclient(hunting): get_auth_request con 0x7f0528139b80 auth_method 0 2024-09-11T02:58:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052d486640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052d486640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052d486640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient(hunting): handle_auth_done global_id 44358 payload 274 2024-09-11T02:58:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 -- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0528139b80 msgr2=0x7f0528139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0528139b80 0x7f0528139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 -- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f052806fe50 msgr2=0x7f05280a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f052806fe50 0x7f05280a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 monclient: found mon.noname-d 2024-09-11T02:58:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.806+0000 7f052dc87640 1 -- 172.21.15.98:0/740923672 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0528071370 con 0x7f0528072640 2024-09-11T02:58:50.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052dc87640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0528072640 0x7f05280a7f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0520009e80 tx=0x7f052001c910 comp rx=0 tx=0).ready entity=mon.6 client_cookie=606ae9bc9484b122 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 1 -- 172.21.15.98:0/740923672 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05200090d0 con 0x7f0528072640 2024-09-11T02:58:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: dump: 2024-09-11T02:58:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.841 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.842 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.843 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.844 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 1 -- 172.21.15.98:0/740923672 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05200a5a70 con 0x7f0528072640 2024-09-11T02:58:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 1 -- 172.21.15.98:0/740923672 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05200a5c20 con 0x7f0528072640 2024-09-11T02:58:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 5 monclient: authenticate success, global_id 44358 2024-09-11T02:58:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 4 set_mon_vals no callback set 2024-09-11T02:58:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f051bfff640 10 monclient: dump: 2024-09-11T02:58:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient: shutdown 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- 172.21.15.98:0/740923672 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0528072640 msgr2=0x7f05280a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0528072640 0x7f05280a7f50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0520009e80 tx=0x7f052001c910 comp rx=0 tx=0).stop 2024-09-11T02:58:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- 172.21.15.98:0/740923672 shutdown_connections 2024-09-11T02:58:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f052806fe50 0x7f05280a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0528072640 0x7f05280a7f50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 --2- 172.21.15.98:0/740923672 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0528139b80 0x7f0528139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- 172.21.15.98:0/740923672 >> 172.21.15.98:0/740923672 conn(0x7f05280ac730 msgr2=0x7f05280a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- 172.21.15.98:0/740923672 shutdown_connections 2024-09-11T02:58:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- 172.21.15.98:0/740923672 wait complete. 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient: build_initial_monmap 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient: monmap: 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 librados: starting msgr at 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 librados: starting objecter 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 Processor -- start 2024-09-11T02:58:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- start start 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 librados: setting wanted keys 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 librados: calling monclient init 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient: init 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient: _add_conns ranks=[0,2,1,8,6,7,3,4,5] 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052806fe50 0x7f05281d19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): picked mon.noname-a con 0x7f052806fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0528072640 0x7f052813f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): picked mon.noname-b con 0x7f0528072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0528139b80 0x7f052813fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): picked mon.noname-f con 0x7f0528139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05281435c0 con 0x7f052806fe50 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0528143440 con 0x7f0528139b80 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0528143740 con 0x7f0528072640 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052f6ea640 10 monclient(hunting): authenticate will time out at 1484.695184s 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052806fe50 0x7f05281d19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0528072640 0x7f052813f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052806fe50 0x7f05281d19a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38632/0 (socket says 172.21.15.98:38632) 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 -- 172.21.15.98:0/3466575187 learned_addr learned my addr 172.21.15.98:0/3466575187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): get_auth_request con 0x7f052806fe50 auth_method 0 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 10 monclient(hunting): get_auth_request con 0x7f0528072640 auth_method 0 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052dc87640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0528139b80 0x7f052813fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052dc87640 10 monclient(hunting): get_auth_request con 0x7f0528139b80 auth_method 0 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052dc87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052dc87640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052dc87640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient(hunting): handle_auth_done global_id 44469 payload 824 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0528072640 msgr2=0x7f052813f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0528072640 0x7f052813f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0528139b80 msgr2=0x7f052813fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0528139b80 0x7f052813fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient: _finish_hunting 0 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 monclient: found mon.noname-a 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 -- 172.21.15.98:0/3466575187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052813bfc0 con 0x7f052806fe50 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052cc85640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0528072640 0x7f052813f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052dc87640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0528139b80 0x7f052813fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f052d486640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052806fe50 0x7f05281d19a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f051c00d5f0 tx=0x7f051c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f0519ffb640 10 client.?.objecter ms_handle_connect 0x7f052806fe50 2024-09-11T02:58:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.810+0000 7f0519ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f051c007380 con 0x7f052806fe50 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: dump: 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: _finish_auth 0 2024-09-11T02:58:50.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f051c019070 con 0x7f052806fe50 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 5 monclient: authenticate success, global_id 44469 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 10 monclient: _renew_subs 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05280aa200 con 0x7f052806fe50 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f051c014430 con 0x7f052806fe50 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:50.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f051b7fe640 4 set_mon_vals no callback set 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: dump: 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.860 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 10 client.44469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 10 monclient: _renew_subs 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f052813e320 con 0x7f052806fe50 2024-09-11T02:58:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 1 librados: init done 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813a6b0 0x7f052813aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f052806ca10 con 0x7f052813a6b0 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f051c013070 con 0x7f052806fe50 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813a6b0 0x7f052813aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient: get_auth_request con 0x7f052813a6b0 auth_method 0 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df70 0x7f050c040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(361..361 src has 1..361) ==== 6202+0+0 (secure 0 0 0) 0x7f051c00f040 con 0x7f052806fe50 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 client.44469.objecter ms_dispatch 0x7f0528000ba0 osd_map(361..361 src has 1..361) 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 3 client.44469.objecter handle_osd_map got epochs [361,361] > 0 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 3 client.44469.objecter handle_osd_map decoding full epoch 361 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 20 client.44469.objecter dump_active .. 0 homeless 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052dc87640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df70 0x7f050c040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052dc87640 10 monclient: get_auth_request con 0x7f050c03df70 auth_method 0 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 10 monclient(hunting): handle_auth_done global_id 64173 payload 824 2024-09-11T02:58:50.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 1 -- 172.21.15.98:0/3466575187 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0528139f60 con 0x7f052813a6b0 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 client.44469.objecter ms_handle_connect 0x7f052813a6b0 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052cc85640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813a6b0 0x7f052813aa90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f052813c630 tx=0x7f052400e1d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 client.44469.objecter resend_mon_ops 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0524018020 con 0x7f052813a6b0 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: dump: 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 client.44469.objecter ms_handle_connect 0x7f050c03df70 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f052dc87640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df70 0x7f050c040430 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7f05280704e0 tx=0x7f05200a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f052400ed10 con 0x7f052813a6b0 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813a6b0 msgr2=0x7f052813aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813a6b0 0x7f052813aa90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f052813c630 tx=0x7f052400e1d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813a6b0 msgr2=0x7f052813aa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.814+0000 7f0519ffb640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813a6b0 0x7f052813aa90 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f052813c630 tx=0x7f052400e1d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052f6ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052f6ea640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813afd0 0x7f05280a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052f6ea640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f052806e870 con 0x7f052813afd0 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813afd0 0x7f05280a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient: get_auth_request con 0x7f052813afd0 auth_method 0 2024-09-11T02:58:50.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 10 monclient(hunting): handle_auth_done global_id 64182 payload 824 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 1 -- 172.21.15.98:0/3466575187 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05280a8270 con 0x7f052813afd0 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 10 client.44469.objecter ms_handle_connect 0x7f052813afd0 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f052cc85640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813afd0 0x7f05280a0e60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f052813c630 tx=0x7f05240046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 10 client.44469.objecter resend_mon_ops 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0524018040 con 0x7f052813afd0 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:50.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 10 monclient: dump: 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:50.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f0524003b10 con 0x7f052813afd0 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813afd0 msgr2=0x7f05280a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813afd0 0x7f05280a0e60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f052813c630 tx=0x7f05240046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813afd0 msgr2=0x7f05280a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:50.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.822+0000 7f0519ffb640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f052813afd0 0x7f05280a0e60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 23011, 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:50.870 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:50.871 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:50.871 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:50.871 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:50.871 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:50.871 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:50.871 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:50.872 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:50.872 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.872 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:58:50.872 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.872 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:50.872 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:50.873 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.873 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.873 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.873 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.873 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.873 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.874 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.874 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.874 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.874 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.874 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.874 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.875 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:50.875 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:50.875 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.875 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.875 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.876 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.876 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.876 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.876 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.876 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.877 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.877 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.877 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.877 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:50.878 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:50.878 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:50.878 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:50.878 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:50.878 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:50.878 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.879 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:50.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:50.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:50.879 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:50.879 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:50.879 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:50.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:50.880 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:50.880 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:50.880 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:50.880 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:50.881 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:50.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:50.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:50.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:50.881 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:50.882 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:50.882 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:50.882 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:50.882 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:50.882 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:50.883 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.883 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:50.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.883 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:50.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.883 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:50.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:50.884 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:50.885 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:50.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.886 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:50.887 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.887 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.887 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.887 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:50.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:50.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:50.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:50.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.890 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:50.891 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.891 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.891 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.892 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:50.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:50.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:50.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:50.895 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.895 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.895 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.896 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:50.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:50.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:50.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:50.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:50.899 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.899 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.900 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.900 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:50.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:50.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:50.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:50.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.903 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:50.903 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.903 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.904 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:50.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:50.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:50.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:50.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:50.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.908 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.909 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:50.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:50.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:50.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:50.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.912 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:50.913 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.913 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.913 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.914 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:50.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:50.914 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:50.915 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:50.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.917 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:50.917 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.917 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.917 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.917 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:50.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:50.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:50.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:50.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:50.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:50.919 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:50.920 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:50.921 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 10 librados: watch_flush enter 2024-09-11T02:58:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 10 librados: watch_flush exit 2024-09-11T02:58:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 20 client.44469.objecter shutdown clearing up homeless session... 2024-09-11T02:58:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 10 client.44469.objecter successfully canceled tick 2024-09-11T02:58:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df70 msgr2=0x7f050c040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f050c03df70 0x7f050c040430 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7f05280704e0 tx=0x7f05200a7070 comp rx=0 tx=0).stop 2024-09-11T02:58:50.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 10 monclient: shutdown 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052806fe50 msgr2=0x7f05281d19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 --2- 172.21.15.98:0/3466575187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f052806fe50 0x7f05281d19a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f051c00d5f0 tx=0x7f051c00dab0 comp rx=0 tx=0).stop 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052dc87640 1 -- 172.21.15.98:0/3466575187 reap_dead start 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052dc87640 1 -- 172.21.15.98:0/3466575187 reap_dead start 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 shutdown_connections 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 >> 172.21.15.98:0/3466575187 conn(0x7f05280ac730 msgr2=0x7f05280ab470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 shutdown_connections 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 -- 172.21.15.98:0/3466575187 wait complete. 2024-09-11T02:58:50.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:50.826+0000 7f052f6ea640 1 librados: shutdown 2024-09-11T02:58:50.925 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:50.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:58:50.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.050+0000 7fc08a988640 10 monclient: get_monmap_and_config 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.050+0000 7fc08a988640 10 monclient: build_initial_monmap 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.050+0000 7fc08a988640 10 monclient: monmap: 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:51.057017+0000 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:51.057017+0000 2024-09-11T02:58:51.055 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:51.056 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:51.057 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 Processor -- start 2024-09-11T02:58:51.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 -- start start 2024-09-11T02:58:51.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient: init 2024-09-11T02:58:51.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:51.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient: _add_conns ranks=[4,10,2,7,3,5,1,6,9,8,0] 2024-09-11T02:58:51.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc084139b10 0x7fc084072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): picked mon.noname-c con 0x7fc084139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc084139ef0 0x7fc0840a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): picked mon.noname-f con 0x7fc084139ef0 addr v2:172.21.15.190:3300/0 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08413a2d0 0x7fc0840a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): picked mon.noname-b con 0x7fc08413a2d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc0840702b0 con 0x7fc084139ef0 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc084071bf0 con 0x7fc08413a2d0 2024-09-11T02:58:51.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc084071350 con 0x7fc084139b10 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc08a988640 10 monclient(hunting): authenticate will time out at 1484.937378s 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08413a2d0 0x7fc0840a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc084139b10 0x7fc084072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08413a2d0 0x7fc0840a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45326/0 (socket says 172.21.15.98:45326) 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 -- 172.21.15.98:0/2280506907 learned_addr learned my addr 172.21.15.98:0/2280506907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): get_auth_request con 0x7fc08413a2d0 auth_method 0 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 10 monclient(hunting): get_auth_request con 0x7fc084139b10 auth_method 0 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc083fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc0837fe640 1 --2- 172.21.15.98:0/2280506907 >> v2:172.21.15.190:3300/0 conn(0x7fc084139ef0 0x7fc0840a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc0837fe640 10 monclient(hunting): get_auth_request con 0x7fc084139ef0 auth_method 0 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc0837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc0837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc0837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient(hunting): handle_auth_done global_id 64154 payload 274 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 -- 172.21.15.98:0/2280506907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc084139b10 msgr2=0x7fc084072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 --2- 172.21.15.98:0/2280506907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc084139b10 0x7fc084072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 -- 172.21.15.98:0/2280506907 >> v2:172.21.15.190:3300/0 conn(0x7fc084139ef0 msgr2=0x7fc0840a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 --2- 172.21.15.98:0/2280506907 >> v2:172.21.15.190:3300/0 conn(0x7fc084139ef0 0x7fc0840a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient: _finish_hunting 0 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 monclient: found mon.noname-b 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.054+0000 7fc088f25640 1 -- 172.21.15.98:0/2280506907 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0840a9620 con 0x7fc08413a2d0 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 1 --2- 172.21.15.98:0/2280506907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08413a2d0 0x7fc0840a7f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009e80 tx=0x7fc07c01c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7883e972574cdc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 1 -- 172.21.15.98:0/2280506907 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc07c0090d0 con 0x7fc08413a2d0 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: dump: 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.066 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 1 -- 172.21.15.98:0/2280506907 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc07c0a5a10 con 0x7fc08413a2d0 2024-09-11T02:58:51.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 5 monclient: authenticate success, global_id 64154 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 1 -- 172.21.15.98:0/2280506907 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc07c0a5c00 con 0x7fc08413a2d0 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 4 set_mon_vals no callback set 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc082ffd640 10 monclient: dump: 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.068 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: shutdown 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- 172.21.15.98:0/2280506907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08413a2d0 msgr2=0x7fc0840a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 --2- 172.21.15.98:0/2280506907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08413a2d0 0x7fc0840a7f50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009e80 tx=0x7fc07c01c960 comp rx=0 tx=0).stop 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- 172.21.15.98:0/2280506907 shutdown_connections 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 --2- 172.21.15.98:0/2280506907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08413a2d0 0x7fc0840a7f50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 --2- 172.21.15.98:0/2280506907 >> v2:172.21.15.190:3300/0 conn(0x7fc084139ef0 0x7fc0840a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 --2- 172.21.15.98:0/2280506907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc084139b10 0x7fc084072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- 172.21.15.98:0/2280506907 >> 172.21.15.98:0/2280506907 conn(0x7fc0840ac790 msgr2=0x7fc0840a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:51.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- 172.21.15.98:0/2280506907 shutdown_connections 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- 172.21.15.98:0/2280506907 wait complete. 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: build_initial_monmap 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: monmap: 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:51.071 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 librados: starting msgr at 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 librados: starting objecter 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 Processor -- start 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- start start 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 librados: setting wanted keys 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 librados: calling monclient init 2024-09-11T02:58:51.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: init 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: _add_conns ranks=[2,1,4,7,0,3,5,6,8] 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc084139b10 0x7fc084134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): picked mon.noname-b con 0x7fc084139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc084139ef0 0x7fc084135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): picked mon.noname-f con 0x7fc084139ef0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc08413a2d0 0x7fc08413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): picked mon.noname-c con 0x7fc08413a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc084143460 con 0x7fc084139ef0 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0840a88a0 con 0x7fc084139b10 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc0841435e0 con 0x7fc08413a2d0 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient(hunting): authenticate will time out at 1484.942318s 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc084139b10 0x7fc084134d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc084139b10 0x7fc084134d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45338/0 (socket says 172.21.15.98:45338) 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 -- 172.21.15.98:0/3550347027 learned_addr learned my addr 172.21.15.98:0/3550347027 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:51.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): get_auth_request con 0x7fc084139b10 auth_method 0 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc08413a2d0 0x7fc08413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 10 monclient(hunting): get_auth_request con 0x7fc08413a2d0 auth_method 0 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc0837fe640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc084139ef0 0x7fc084135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc0837fe640 10 monclient(hunting): get_auth_request con 0x7fc084139ef0 auth_method 0 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc0837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc0837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc0837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient(hunting): handle_auth_done global_id 64163 payload 824 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc08413a2d0 msgr2=0x7fc08413a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc08413a2d0 0x7fc08413a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc084139ef0 msgr2=0x7fc084135260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc084139ef0 0x7fc084135260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 monclient: found mon.noname-b 2024-09-11T02:58:51.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 -- 172.21.15.98:0/3550347027 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc084135810 con 0x7fc084139b10 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc088f25640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc08413a2d0 0x7fc08413a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc083fff640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc084139b10 0x7fc084134d20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc07400e930 tx=0x7fc07400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 client.?.objecter ms_handle_connect 0x7fc084139b10 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc074017070 con 0x7fc084139b10 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: dump: 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.083 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.084 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: _finish_auth 0 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 5 monclient: authenticate success, global_id 64163 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074013470 con 0x7fc084139b10 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: _renew_subs 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0840aa210 con 0x7fc084139b10 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc074013a90 con 0x7fc084139b10 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc080ff9640 10 monclient: dump: 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.086 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 client.64163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc0827fc640 4 set_mon_vals no callback set 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: _renew_subs 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc08413e0a0 con 0x7fc084139b10 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 1 librados: init done 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.058+0000 7fc08a988640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc08a988640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413ac20 0x7fc08413b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc08413d800 con 0x7fc08413ac20 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc074013c30 con 0x7fc084139b10 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413ac20 0x7fc08413b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07003df40 0x7fc070040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient: get_auth_request con 0x7fc08413ac20 auth_method 0 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(361..361 src has 1..361) ==== 6202+0+0 (secure 0 0 0) 0x7fc07400f040 con 0x7fc084139b10 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 client.64163.objecter ms_dispatch 0x7fc084000ba0 osd_map(361..361 src has 1..361) 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 3 client.64163.objecter handle_osd_map got epochs [361,361] > 0 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 3 client.64163.objecter handle_osd_map decoding full epoch 361 2024-09-11T02:58:51.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 20 client.64163.objecter dump_active .. 0 homeless 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc088f25640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07003df40 0x7fc070040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc088f25640 10 monclient: get_auth_request con 0x7fc07003df40 auth_method 0 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 10 monclient(hunting): handle_auth_done global_id 20075 payload 824 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 1 -- 172.21.15.98:0/3550347027 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc08406e680 con 0x7fc08413ac20 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 client.64163.objecter ms_handle_connect 0x7fc08413ac20 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 client.64163.objecter resend_mon_ops 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc0837fe640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413ac20 0x7fc08413b000 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fc0840a5e80 tx=0x7fc07800f8a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc078019030 con 0x7fc08413ac20 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 monclient: dump: 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.090 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc088f25640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07003df40 0x7fc070040400 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0ae000 tx=0x7fc07c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 client.64163.objecter ms_handle_connect 0x7fc07003df40 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc078014670 con 0x7fc08413ac20 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413ac20 msgr2=0x7fc08413b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413ac20 0x7fc08413b000 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fc0840a5e80 tx=0x7fc07800f8a0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413ac20 msgr2=0x7fc08413b000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:51.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.062+0000 7fc080ff9640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413ac20 0x7fc08413b000 secure :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fc0840a5e80 tx=0x7fc07800f8a0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413b540 0x7fc0840a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc08406ca10 con 0x7fc08413b540 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413b540 0x7fc0840a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient: get_auth_request con 0x7fc08413b540 auth_method 0 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 10 monclient(hunting): handle_auth_done global_id 20084 payload 824 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 1 -- 172.21.15.98:0/3550347027 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc084072600 con 0x7fc08413b540 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 10 client.64163.objecter ms_handle_connect 0x7fc08413b540 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 10 client.64163.objecter resend_mon_ops 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc0837fe640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413b540 0x7fc0840a0e60 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc078000c10 tx=0x7fc0780046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc078003620 con 0x7fc08413b540 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 10 monclient: dump: 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.093 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.094 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fc07801fa20 con 0x7fc08413b540 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:51.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413b540 msgr2=0x7fc0840a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413b540 0x7fc0840a0e60 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc078000c10 tx=0x7fc0780046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413b540 msgr2=0x7fc0840a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc080ff9640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08413b540 0x7fc0840a0e60 secure :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc078000c10 tx=0x7fc0780046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:51.096 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 546820, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:51.097 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:51.098 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:51.099 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:51.100 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:51.101 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:51.102 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:51.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:51.104 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:51.107 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:51.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:51.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.114 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:51.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.117 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:51.120 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 10 librados: watch_flush enter 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 10 librados: watch_flush exit 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 20 client.64163.objecter shutdown clearing up homeless session... 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 10 client.64163.objecter successfully canceled tick 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07003df40 msgr2=0x7fc070040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07003df40 0x7fc070040400 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0ae000 tx=0x7fc07c0a7070 comp rx=0 tx=0).stop 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 10 monclient: shutdown 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc084139b10 msgr2=0x7fc084134d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 --2- 172.21.15.98:0/3550347027 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc084139b10 0x7fc084134d20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc07400e930 tx=0x7fc07400edf0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc088f25640 1 -- 172.21.15.98:0/3550347027 reap_dead start 2024-09-11T02:58:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc088f25640 1 -- 172.21.15.98:0/3550347027 reap_dead start 2024-09-11T02:58:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 shutdown_connections 2024-09-11T02:58:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 >> 172.21.15.98:0/3550347027 conn(0x7fc0840ac790 msgr2=0x7fc084135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 shutdown_connections 2024-09-11T02:58:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 -- 172.21.15.98:0/3550347027 wait complete. 2024-09-11T02:58:51.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.070+0000 7fc08a988640 1 librados: shutdown 2024-09-11T02:58:51.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:51.123 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:58:51.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient: get_monmap_and_config 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient: build_initial_monmap 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient: monmap: 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:51.241858+0000 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:51.241858+0000 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:51.241 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 Processor -- start 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 -- start start 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient: init 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient: _add_conns ranks=[9,10,8,6,7,4,0,1,5,3,2] 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2f4139b10 0x7ff2f4072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): picked mon.noname-a con 0x7ff2f4139b10 addr v2:172.21.15.98:3300/0 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7ff2f4139ef0 0x7ff2f40a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): picked mon.noname-f con 0x7ff2f4139ef0 addr v2:172.21.15.190:3300/0 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f413a2d0 0x7ff2f40a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): picked mon.noname-e con 0x7ff2f413a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff2f40702b0 con 0x7ff2f4139b10 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7ff2f4071bf0 con 0x7ff2f4139ef0 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff2f4071350 con 0x7ff2f413a2d0 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fc288640 10 monclient(hunting): authenticate will time out at 1485.122746s 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f413a2d0 0x7ff2f40a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f413a2d0 0x7ff2f40a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39684/0 (socket says 172.21.15.98:39684) 2024-09-11T02:58:51.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 -- 172.21.15.98:0/1729618585 learned_addr learned my addr 172.21.15.98:0/1729618585 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): get_auth_request con 0x7ff2f413a2d0 auth_method 0 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff2f4139b10 0x7ff2f4072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 10 monclient(hunting): get_auth_request con 0x7ff2f4139b10 auth_method 0 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient(hunting): handle_auth_done global_id 24524 payload 274 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 -- 172.21.15.98:0/1729618585 >> v2:172.21.15.190:3300/0 conn(0x7ff2f4139ef0 msgr2=0x7ff2f40a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2f9823640 1 --2- 172.21.15.98:0/1729618585 >> v2:172.21.15.190:3300/0 conn(0x7ff2f4139ef0 0x7ff2f40a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 --2- 172.21.15.98:0/1729618585 >> v2:172.21.15.190:3300/0 conn(0x7ff2f4139ef0 0x7ff2f40a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 -- 172.21.15.98:0/1729618585 >> v2:172.21.15.98:3300/0 conn(0x7ff2f4139b10 msgr2=0x7ff2f4072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 --2- 172.21.15.98:0/1729618585 >> v2:172.21.15.98:3300/0 conn(0x7ff2f4139b10 0x7ff2f4072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient: _finish_hunting 0 2024-09-11T02:58:51.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 monclient: found mon.noname-e 2024-09-11T02:58:51.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:51.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:51.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa825640 1 -- 172.21.15.98:0/1729618585 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f40a9620 con 0x7ff2f413a2d0 2024-09-11T02:58:51.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.238+0000 7ff2fa024640 1 --2- 172.21.15.98:0/1729618585 >> v2:172.21.15.98:3300/0 conn(0x7ff2f4139b10 0x7ff2f4072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:51.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fa825640 1 --2- 172.21.15.98:0/1729618585 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f413a2d0 0x7ff2f40a7f50 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40a9a70 tx=0x7ff2ec01c470 comp rx=0 tx=0).ready entity=mon.8 client_cookie=42b73b7c0c68f0e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 1 -- 172.21.15.98:0/1729618585 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff2ec0b0070 con 0x7ff2f413a2d0 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: dump: 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.256 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: _finish_auth 0 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: _check_auth_tickets 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 1 -- 172.21.15.98:0/1729618585 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2ec0aa810 con 0x7ff2f413a2d0 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:51.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 1 -- 172.21.15.98:0/1729618585 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff2ec0a9300 con 0x7ff2f413a2d0 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2f9022640 10 monclient: dump: 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 5 monclient: authenticate success, global_id 24524 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 4 set_mon_vals no callback set 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 10 monclient: shutdown 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 -- 172.21.15.98:0/1729618585 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f413a2d0 msgr2=0x7ff2f40a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 --2- 172.21.15.98:0/1729618585 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f413a2d0 0x7ff2f40a7f50 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40a9a70 tx=0x7ff2ec01c470 comp rx=0 tx=0).stop 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 -- 172.21.15.98:0/1729618585 shutdown_connections 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 --2- 172.21.15.98:0/1729618585 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f413a2d0 0x7ff2f40a7f50 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 --2- 172.21.15.98:0/1729618585 >> v2:172.21.15.190:3300/0 conn(0x7ff2f4139ef0 0x7ff2f40a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 --2- 172.21.15.98:0/1729618585 >> v2:172.21.15.98:3300/0 conn(0x7ff2f4139b10 0x7ff2f4072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 -- 172.21.15.98:0/1729618585 >> 172.21.15.98:0/1729618585 conn(0x7ff2f40ac790 msgr2=0x7ff2f40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 -- 172.21.15.98:0/1729618585 shutdown_connections 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 -- 172.21.15.98:0/1729618585 wait complete. 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 10 monclient: build_initial_monmap 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 10 monclient: monmap: 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:51.260 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 librados: starting msgr at 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 librados: starting objecter 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 Processor -- start 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 -- start start 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 librados: setting wanted keys 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 1 librados: calling monclient init 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.242+0000 7ff2fc288640 10 monclient: init 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient: _add_conns ranks=[8,5,3,6,0,1,2,4,7] 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f4139b10 0x7ff2f413a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): picked mon.noname-e con 0x7ff2f4139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff2f4139ef0 0x7ff2f413acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): picked mon.noname-h con 0x7ff2f4139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:51.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2f413a2d0 0x7ff2f413b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): picked mon.noname-g con 0x7ff2f413a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff2f4143460 con 0x7ff2f413a2d0 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7ff2f40a88a0 con 0x7ff2f4139ef0 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff2f41435e0 con 0x7ff2f4139b10 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): authenticate will time out at 1485.128268s 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f4139b10 0x7ff2f413a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f4139b10 0x7ff2f413a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39686/0 (socket says 172.21.15.98:39686) 2024-09-11T02:58:51.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 -- 172.21.15.98:0/24216692 learned_addr learned my addr 172.21.15.98:0/24216692 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): get_auth_request con 0x7ff2f4139b10 auth_method 0 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient(hunting): handle_auth_done global_id 24533 payload 824 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa825640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2f413a2d0 0x7ff2f413b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff2f4139ef0 0x7ff2f413acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff2f4139ef0 msgr2=0x7ff2f413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff2f4139ef0 0x7ff2f413acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2f413a2d0 msgr2=0x7ff2f413b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff2f413a2d0 0x7ff2f413b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient: _finish_hunting 0 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 monclient: found mon.noname-e 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 -- 172.21.15.98:0/24216692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f413cee0 con 0x7ff2f4139b10 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa024640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f4139b10 0x7ff2f413a7b0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000efc0 tx=0x7ff2f000c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 client.?.objecter ms_handle_connect 0x7ff2f4139b10 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff2f0018020 con 0x7ff2f4139b10 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:51.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: dump: 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.266 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 5 monclient: authenticate success, global_id 24533 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f00148f0 con 0x7ff2f4139b10 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient: _renew_subs 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f413d1c0 con 0x7ff2f4139b10 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:51.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff2f00133e0 con 0x7ff2f4139b10 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 client.24533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2ebfff640 4 set_mon_vals no callback set 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: dump: 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.268 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient: _renew_subs 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f40a9f00 con 0x7ff2f4139b10 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 librados: init done 2024-09-11T02:58:51.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f413cab0 0x7ff2f4133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7ff2f406ca10 con 0x7ff2f413cab0 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ff2f0014480 con 0x7ff2f4139b10 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f413cab0 0x7ff2f4133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient: get_auth_request con 0x7ff2f413cab0 auth_method 0 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff2d803df70 0x7ff2d8040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(361..361 src has 1..361) ==== 6202+0+0 (secure 0 0 0) 0x7ff2f00503d0 con 0x7ff2f4139b10 2024-09-11T02:58:51.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 client.24533.objecter ms_dispatch 0x7ff2f4000ba0 osd_map(361..361 src has 1..361) 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 3 client.24533.objecter handle_osd_map got epochs [361,361] > 0 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 3 client.24533.objecter handle_osd_map decoding full epoch 361 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 20 client.24533.objecter dump_active .. 0 homeless 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa825640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff2d803df70 0x7ff2d8040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa825640 10 monclient: get_auth_request con 0x7ff2d803df70 auth_method 0 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 10 monclient(hunting): handle_auth_done global_id 19672 payload 824 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 1 -- 172.21.15.98:0/24216692 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2f406e680 con 0x7ff2f413cab0 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 client.24533.objecter ms_handle_connect 0x7ff2f413cab0 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2f9823640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f413cab0 0x7ff2f4133760 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40a5470 tx=0x7ff2e400d950 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 client.24533.objecter resend_mon_ops 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff2e4016070 con 0x7ff2f413cab0 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 monclient: dump: 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.272 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2eaffd640 10 client.24533.objecter ms_handle_connect 0x7ff2d803df70 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.246+0000 7ff2fa825640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff2d803df70 0x7ff2d8040430 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40a9a70 tx=0x7ff2ec0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.250+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff2e4012670 con 0x7ff2f413cab0 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.250+0000 7ff2eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.250+0000 7ff2eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.250+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f413cab0 msgr2=0x7ff2f4133760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.250+0000 7ff2eaffd640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f413cab0 0x7ff2f4133760 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40a5470 tx=0x7ff2e400d950 comp rx=0 tx=0).stop 2024-09-11T02:58:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.250+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f413cab0 msgr2=0x7ff2f4133760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:51.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.250+0000 7ff2eaffd640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f413cab0 0x7ff2f4133760 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2fc288640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:51.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2fc288640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f4133ca0 0x7ff2f41340b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2fc288640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7ff2f406e870 con 0x7ff2f4133ca0 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f4133ca0 0x7ff2f41340b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient: get_auth_request con 0x7ff2f4133ca0 auth_method 0 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 10 monclient(hunting): handle_auth_done global_id 19681 payload 824 2024-09-11T02:58:51.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.254+0000 7ff2f9823640 1 -- 172.21.15.98:0/24216692 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff2f41345f0 con 0x7ff2f4133ca0 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2f9823640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f4133ca0 0x7ff2f41340b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4000c10 tx=0x7ff2e4004740 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 10 client.24533.objecter ms_handle_connect 0x7ff2f4133ca0 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 10 client.24533.objecter resend_mon_ops 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff2e401be20 con 0x7ff2f4133ca0 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 10 monclient: dump: 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.277 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7ff2e40115d0 con 0x7ff2f4133ca0 2024-09-11T02:58:51.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f4133ca0 msgr2=0x7ff2f41340b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f4133ca0 0x7ff2f41340b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4000c10 tx=0x7ff2e4004740 comp rx=0 tx=0).stop 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f4133ca0 msgr2=0x7ff2f41340b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2eaffd640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff2f4133ca0 0x7ff2f41340b0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:51.279 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 547028, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:51.280 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:51.281 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:51.282 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:51.283 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:51.284 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.290 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.292 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:51.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.294 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:51.296 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:51.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.298 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.300 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:58:51.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:51.304 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 10 librados: watch_flush enter 2024-09-11T02:58:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 10 librados: watch_flush exit 2024-09-11T02:58:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 20 client.24533.objecter shutdown clearing up homeless session... 2024-09-11T02:58:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 10 client.24533.objecter successfully canceled tick 2024-09-11T02:58:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff2d803df70 msgr2=0x7ff2d8040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff2d803df70 0x7ff2d8040430 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7ff2f40a9a70 tx=0x7ff2ec0a6000 comp rx=0 tx=0).stop 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 10 monclient: shutdown 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f4139b10 msgr2=0x7ff2f413a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 --2- 172.21.15.98:0/24216692 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff2f4139b10 0x7ff2f413a7b0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000efc0 tx=0x7ff2f000c490 comp rx=0 tx=0).stop 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fa825640 1 -- 172.21.15.98:0/24216692 reap_dead start 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fa825640 1 -- 172.21.15.98:0/24216692 reap_dead start 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 shutdown_connections 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 >> 172.21.15.98:0/24216692 conn(0x7ff2f40ac790 msgr2=0x7ff2f40aa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 shutdown_connections 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 -- 172.21.15.98:0/24216692 wait complete. 2024-09-11T02:58:51.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.258+0000 7ff2fc288640 1 librados: shutdown 2024-09-11T02:58:51.308 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:51.308 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T02:58:51.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T02:58:51.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient: get_monmap_and_config 2024-09-11T02:58:51.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient: build_initial_monmap 2024-09-11T02:58:51.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient: monmap: 2024-09-11T02:58:51.422 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:51.422 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.422 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:51.423726+0000 2024-09-11T02:58:51.422 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:51.423726+0000 2024-09-11T02:58:51.422 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:51.423 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 1 Processor -- start 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 1 -- start start 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient: init 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient: _add_conns ranks=[3,4,1,0,7,2,10,6,5,9,8] 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81b80a37a0 0x7f81b809e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient(hunting): picked mon.noname-g con 0x7f81b80a37a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f81b80a3b80 0x7f81b80a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient(hunting): picked mon.noname-c con 0x7f81b80a3b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f81b8137570 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b809d440 con 0x7f81b8137570 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f81b80a1b90 con 0x7f81b80a37a0 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f81b809d720 con 0x7f81b80a3b80 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.418+0000 7f81bd1a6640 10 monclient(hunting): authenticate will time out at 1485.303903s 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f81b80a3b80 0x7f81b80a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f81b80a3b80 0x7f81b80a3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59722/0 (socket says 172.21.15.98:59722) 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 -- 172.21.15.98:0/2718063649 learned_addr learned my addr 172.21.15.98:0/2718063649 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): get_auth_request con 0x7f81b80a3b80 auth_method 0 2024-09-11T02:58:51.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b6d9d640 1 --2- 172.21.15.98:0/2718063649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81b80a37a0 0x7f81b809e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b6d9d640 10 monclient(hunting): get_auth_request con 0x7f81b80a37a0 auth_method 0 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient(hunting): handle_auth_done global_id 23989 payload 274 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 -- 172.21.15.98:0/2718063649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81b80a37a0 msgr2=0x7f81b809e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:51.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 --2- 172.21.15.98:0/2718063649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81b80a37a0 0x7f81b809e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 -- 172.21.15.98:0/2718063649 >> v1:172.21.15.190:6789/0 conn(0x7f81b8137570 legacy=0x7f81b81379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient: _finish_hunting 0 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 monclient: found mon.noname-c 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 -- 172.21.15.98:0/2718063649 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b809da00 con 0x7f81b80a3b80 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b659c640 1 --2- 172.21.15.98:0/2718063649 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f81b80a3b80 0x7f81b80a3f80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f81a00098a0 tx=0x7f81a001cea0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ed920d82e925b6b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 1 -- 172.21.15.98:0/2718063649 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a00b1070 con 0x7f81b80a3b80 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: dump: 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.430 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.431 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: _finish_auth 0 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81bd1a6640 5 monclient: authenticate success, global_id 23989 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 1 -- 172.21.15.98:0/2718063649 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a00aa7b0 con 0x7f81b80a3b80 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81bd1a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81bd1a6640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81bd1a6640 4 set_mon_vals no callback set 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 1 -- 172.21.15.98:0/2718063649 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a00aae50 con 0x7f81b80a3b80 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81b5d9b640 10 monclient: dump: 2024-09-11T02:58:51.432 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.433 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81bd1a6640 10 monclient: shutdown 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81bd1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.422+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2718063649 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f81b80a3b80 msgr2=0x7f81b80a3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2718063649 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f81b80a3b80 0x7f81b80a3f80 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f81a00098a0 tx=0x7f81a001cea0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2718063649 shutdown_connections 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2718063649 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f81b80a3b80 0x7f81b80a3f80 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2718063649 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f81b80a37a0 0x7f81b809e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2718063649 >> 172.21.15.98:0/2718063649 conn(0x7f81b80a4890 msgr2=0x7f81b80a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2718063649 shutdown_connections 2024-09-11T02:58:51.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2718063649 wait complete. 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: build_initial_monmap 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: monmap: 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:51.439 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 librados: starting msgr at 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 librados: starting objecter 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 Processor -- start 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- start start 2024-09-11T02:58:51.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 librados: setting wanted keys 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 librados: calling monclient init 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: init 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _add_conns ranks=[2,1,7,4,5,3,0,6,8] 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b80a37a0 0x7f81b812c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): picked mon.noname-b con 0x7f81b80a37a0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81b80a3b80 0x7f81b812ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): picked mon.noname-f con 0x7f81b80a3b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f81b8137570 0x7f81b812cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): picked mon.noname-i con 0x7f81b8137570 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f81b8142380 con 0x7f81b80a3b80 2024-09-11T02:58:51.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f81b8142200 con 0x7f81b80a37a0 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f81b8142500 con 0x7f81b8137570 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient(hunting): authenticate will time out at 1485.309603s 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b80a37a0 0x7f81b812c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b80a37a0 0x7f81b812c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45352/0 (socket says 172.21.15.98:45352) 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 -- 172.21.15.98:0/2013468138 learned_addr learned my addr 172.21.15.98:0/2013468138 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): get_auth_request con 0x7f81b80a37a0 auth_method 0 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b659c640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81b80a3b80 0x7f81b812ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b759e640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f81b8137570 0x7f81b812cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b759e640 10 monclient(hunting): get_auth_request con 0x7f81b8137570 auth_method 0 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b759e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b659c640 10 monclient(hunting): get_auth_request con 0x7f81b80a3b80 auth_method 0 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:51.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient(hunting): handle_auth_done global_id 64172 payload 824 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 -- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f81b8137570 msgr2=0x7f81b812cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f81b8137570 0x7f81b812cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 -- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81b80a3b80 msgr2=0x7f81b812ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81b80a3b80 0x7f81b812ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient: _finish_hunting 0 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 monclient: found mon.noname-b 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 -- 172.21.15.98:0/2013468138 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b8135e40 con 0x7f81b80a37a0 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b6d9d640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b80a37a0 0x7f81b812c4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f81a800d800 tx=0x7f81a800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 client.?.objecter ms_handle_connect 0x7f81b80a37a0 2024-09-11T02:58:51.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 1 -- 172.21.15.98:0/2013468138 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a800bb30 con 0x7f81b80a37a0 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: dump: 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.445 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: _finish_auth 0 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 1 -- 172.21.15.98:0/2013468138 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a8019070 con 0x7f81b80a37a0 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:51.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 1 -- 172.21.15.98:0/2013468138 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a800bd10 con 0x7f81b80a37a0 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 5 monclient: authenticate success, global_id 64172 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _renew_subs 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b8136120 con 0x7f81b80a37a0 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b559a640 4 set_mon_vals no callback set 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 monclient: dump: 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:51.447 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 client.64172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:51.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _renew_subs 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b8135b60 con 0x7f81b80a37a0 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 librados: init done 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b809e100 con 0x7f81b80a37a0 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 1 -- 172.21.15.98:0/2013468138 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f81a8013070 con 0x7f81b80a37a0 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f819803df40 0x7f8198040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 1 -- 172.21.15.98:0/2013468138 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(361..361 src has 1..361) ==== 6202+0+0 (secure 0 0 0) 0x7f81a800f040 con 0x7f81b80a37a0 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 10 client.64172.objecter ms_dispatch 0x7f81b8000ba0 osd_map(361..361 src has 1..361) 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 3 client.64172.objecter handle_osd_map got epochs [361,361] > 0 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 3 client.64172.objecter handle_osd_map decoding full epoch 361 2024-09-11T02:58:51.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81af7fe640 20 client.64172.objecter dump_active .. 0 homeless 2024-09-11T02:58:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.426+0000 7f81b659c640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f819803df40 0x7f8198040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.430+0000 7f81b659c640 10 monclient: get_auth_request con 0x7f819803df40 auth_method 0 2024-09-11T02:58:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.430+0000 7f81af7fe640 10 client.64172.objecter ms_handle_connect 0x7f819803df40 2024-09-11T02:58:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.430+0000 7f81b659c640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f819803df40 0x7f8198040400 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7f81a00ae000 tx=0x7f81a0004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.434+0000 7f81af7fe640 1 -- 172.21.15.98:0/2013468138 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f81a801edc0 con 0x7f81b80a37a0 2024-09-11T02:58:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.434+0000 7f81af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:51.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.434+0000 7f81af7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.666+0000 7f81bd1a6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T02:58:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.666+0000 7f81bd1a6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T02:58:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.666+0000 7f81bd1a6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:58:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.666+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f81b80690b0 con 0x7f81b80a37a0 2024-09-11T02:58:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.670+0000 7f81af7fe640 1 -- 172.21.15.98:0/2013468138 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f81a801e980 con 0x7f81b80a37a0 2024-09-11T02:58:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.670+0000 7f81af7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T02:58:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.670+0000 7f81af7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:51.675 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.675 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.675 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=709187625,config=692619927,health=1423591699,kv=2823699850,logm=2071103505}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=9,logm=64} crc {auth=709187625,config=692619927,health=1983350863,kv=894428524,logm=3675110557}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=1,health=11,kv=10,logm=66} crc {auth=2164995329,config=1228700967,health=136006561,kv=3075885491,logm=330464251}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=4164564380,config=692619927,health=1423591699,kv=3763817534,logm=1320761124}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.676 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=13,config=2,health=11,kv=10,logm=64} crc {auth=4064585297,config=692619927,health=2621999868,kv=3991532845,logm=1994610194}) 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=13,config=2,health=11,kv=10,logm=64} crc {auth=3928914093,config=692619927,health=3649997111,kv=3991532846,logm=3710943467}) 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=709187625,config=692619927,health=4017629869,kv=3991532845,logm=1883282340}) 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3098957462,config=692619927,health=2340146688,kv=3991532845,logm=3820639929}) 2024-09-11T02:58:51.677 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.670+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=14,config=1,health=11,kv=10,logm=64} crc {auth=709187625,config=3678701417,health=4114116883,kv=3991532845,logm=1127469323}) 2024-09-11T02:58:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 10 librados: watch_flush enter 2024-09-11T02:58:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 10 librados: watch_flush exit 2024-09-11T02:58:51.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 20 client.64172.objecter shutdown clearing up homeless session... 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 10 client.64172.objecter successfully canceled tick 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f819803df40 msgr2=0x7f8198040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f819803df40 0x7f8198040400 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7f81a00ae000 tx=0x7f81a0004740 comp rx=0 tx=0).stop 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 10 monclient: shutdown 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b80a37a0 msgr2=0x7f81b812c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b80a37a0 0x7f81b812c4c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f81a800d800 tx=0x7f81a800dcc0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 shutdown_connections 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f81b8137570 0x7f81b812cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f81b80a3b80 0x7f81b812ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f819803df40 0x7f8198040400 unknown :-1 s=CLOSED pgs=3543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 --2- 172.21.15.98:0/2013468138 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f81b80a37a0 0x7f81b812c4c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:51.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 >> 172.21.15.98:0/2013468138 conn(0x7f81b80a4890 msgr2=0x7f81b80a6b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 shutdown_connections 2024-09-11T02:58:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 -- 172.21.15.98:0/2013468138 wait complete. 2024-09-11T02:58:51.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:51.674+0000 7f81bd1a6640 1 librados: shutdown 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744579237}) 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3210412558}) 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333081176}) 2024-09-11T02:58:51.682 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3360336387}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=253058148}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1665630124}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1691380662}) 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.683 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176255737}) 2024-09-11T02:58:51.684 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.678+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3149631437}) 2024-09-11T02:58:51.686 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.686 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=675285079}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2087838032}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3895537647}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=462813358}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.687 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1141817216}) 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=610795687}) 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3061138213}) 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2582816165}) 2024-09-11T02:58:51.688 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.682+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=106697074}) 2024-09-11T02:58:51.698 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.698 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.698 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=648751001}) 2024-09-11T02:58:51.698 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.698 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.698 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147091626}) 2024-09-11T02:58:51.698 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T02:58:51.700 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1634804471}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2670581946}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1576074153}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2193842211}) 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.701 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.702 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3792507472}) 2024-09-11T02:58:51.702 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.702 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2534982589}) 2024-09-11T02:58:51.702 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.694+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3965826763}) 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177833065}) 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270757216}) 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.711 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=203542292}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2524878421}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2742118937}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2987263236}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.712 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2597002575}) 2024-09-11T02:58:51.713 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.713 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2967419701}) 2024-09-11T02:58:51.713 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.706+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1070342436}) 2024-09-11T02:58:51.716 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.716 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.716 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965540858}) 2024-09-11T02:58:51.716 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190629634}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282946764}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1908679233}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=83583412}) 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.717 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.718 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=685864039}) 2024-09-11T02:58:51.718 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.718 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.718 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1187527023}) 2024-09-11T02:58:51.718 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.718 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3658442890}) 2024-09-11T02:58:51.718 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.714+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1409695859}) 2024-09-11T02:58:51.721 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.721 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.721 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363926720}) 2024-09-11T02:58:51.721 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2859268849}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2447262319}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758635793}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2767209454}) 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.722 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.723 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3896793486}) 2024-09-11T02:58:51.723 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.723 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.723 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=570081482}) 2024-09-11T02:58:51.723 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.723 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642476588}) 2024-09-11T02:58:51.723 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.718+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=50671028}) 2024-09-11T02:58:51.725 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.725 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.725 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2609022994}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245359038}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059560332}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2128196674}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2784985466}) 2024-09-11T02:58:51.726 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1277040903}) 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=614609535}) 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=528785005}) 2024-09-11T02:58:51.727 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.722+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3624036500}) 2024-09-11T02:58:51.730 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.730 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.730 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885806192}) 2024-09-11T02:58:51.730 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.730 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.730 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=65919258}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1069125420}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2292616596}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=312665615}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2990725634}) 2024-09-11T02:58:51.731 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.732 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.732 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=523919241}) 2024-09-11T02:58:51.732 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.732 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1069435038}) 2024-09-11T02:58:51.732 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.726+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2345165981}) 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755319584}) 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921202740}) 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.734 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829412596}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2993103146}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3775550826}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1555195684}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2357871905}) 2024-09-11T02:58:51.735 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.736 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=15110596}) 2024-09-11T02:58:51.736 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.730+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2046895010}) 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570850867}) 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680620461}) 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.737 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832898619}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1973697522}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2442075080}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2925133186}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2088336428}) 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.738 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3215181717}) 2024-09-11T02:58:51.739 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2383588337}) 2024-09-11T02:58:51.740 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.740 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224469982}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1004050812}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.734+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351195157}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2818341490}) 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.741 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4057598986}) 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2373109748}) 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=201299559}) 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3300439592}) 2024-09-11T02:58:51.742 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4135931396}) 2024-09-11T02:58:51.743 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.743 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705436427}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527247773}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=127652166}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1233835452}) 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.744 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2927772402}) 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=390869856}) 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3458640123}) 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1580491849}) 2024-09-11T02:58:51.745 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.738+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3452026316}) 2024-09-11T02:58:51.748 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.748 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060397829}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4175874988}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680432287}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=608853798}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.749 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1277521825}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3877459660}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2226042468}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=55680585}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4023263733}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.750 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924906015}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864867758}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104499356}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2057927169}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1457205912}) 2024-09-11T02:58:51.751 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3836481829}) 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3724288650}) 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.742+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2395585423}) 2024-09-11T02:58:51.752 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=993140587}) 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016150064}) 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1468074624}) 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.762 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248145688}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1195598936}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2057889328}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=668662147}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3647471827}) 2024-09-11T02:58:51.763 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=732322909}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.746+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1025067255}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=471365546}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1560837769}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713677846}) 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.764 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2111843461}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4232526080}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3606460856}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1734959797}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125611440}) 2024-09-11T02:58:51.765 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.750+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4244673561}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397079930}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2789135124}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=420986320}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3625692749}) 2024-09-11T02:58:51.766 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3066160985}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=435075618}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1482228523}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1535953755}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2391676200}) 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.767 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965167662}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=650916104}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747918844}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3585004541}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.768 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=384349388}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3250211976}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2662873735}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=496994832}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.754+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2075733057}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152288377}) 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.769 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091440922}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2661555865}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1641745624}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1070780872}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.770 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3156613339}) 2024-09-11T02:58:51.771 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.771 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.771 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3668094675}) 2024-09-11T02:58:51.771 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.771 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292141069}) 2024-09-11T02:58:51.771 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3713580801}) 2024-09-11T02:58:51.810 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.810 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.810 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=519239336}) 2024-09-11T02:58:51.810 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.810 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.810 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3688769286}) 2024-09-11T02:58:51.810 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1222836511}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.758+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763169920}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3352566164}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2046232732}) 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.811 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3151618544}) 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3806515741}) 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2670207773}) 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478062431}) 2024-09-11T02:58:51.812 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080786333}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4221985895}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1855646917}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2093206257}) 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.813 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2554514128}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3169011070}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2529697524}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.762+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4062388181}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2592612369}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.814 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3202497576}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616400103}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687435532}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4068894098}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3062297501}) 2024-09-11T02:58:51.815 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3647129088}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3006593169}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2524902325}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780192585}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072818677}) 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.816 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2702422883}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3509713205}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=48402550}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3927568715}) 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.817 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2242285785}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883440752}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.766+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4126106605}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232376532}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3308700043}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.818 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205043159}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1669632979}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=996271094}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2063538035}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3821050903}) 2024-09-11T02:58:51.819 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1848027107}) 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4153932296}) 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273687841}) 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902510570}) 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.820 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2920426643}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2185335585}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2455008166}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3511713688}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2932381443}) 2024-09-11T02:58:51.821 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333084018}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.770+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922885434}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132848561}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3070016738}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304387285}) 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.822 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2960496789}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2671232045}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1166118892}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3286504698}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3014715846}) 2024-09-11T02:58:51.823 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3649815830}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221680602}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1917611283}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=406923092}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.824 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=144033514}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4059210146}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.774+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=267236406}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3802596350}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1745053987}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2556269247}) 2024-09-11T02:58:51.825 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878362590}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160200981}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3982589356}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=119648768}) 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.826 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=148432049}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1736829319}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4218096879}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3529620656}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.778+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1370601112}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.827 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506967839}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3948724350}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2830652572}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1259047141}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2752011583}) 2024-09-11T02:58:51.828 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=861922576}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=12943083}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228635923}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1859497388}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2951194282}) 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.829 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561604740}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136155226}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137995308}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=12711185}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.830 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2337998840}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1006842338}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=713567771}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.782+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=339031009}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545639653}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140938386}) 2024-09-11T02:58:51.831 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140210578}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=309838111}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3357422936}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=790811800}) 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.832 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=114766543}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3164816124}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=755982969}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688578327}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1029873456}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.833 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147316275}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3465334698}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4107088220}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1198002491}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.834 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2862370129}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3090207161}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.786+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4105418898}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76258449}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=606836337}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905109200}) 2024-09-11T02:58:51.835 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3208946468}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1717242569}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2929065684}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3486993026}) 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.836 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2545932842}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4215329150}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726040690}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4164687207}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199270521}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.837 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2465132542}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2054757832}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=45109029}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2872609547}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3860267183}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.790+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=825718311}) 2024-09-11T02:58:51.838 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534265921}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640794853}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1758753226}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=662617573}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.839 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=611690537}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1365178865}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3822701806}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3572938984}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1866208992}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.840 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849628246}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456841410}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1628039900}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410686896}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=316204252}) 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.841 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3365443874}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=762930804}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3818917549}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.794+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3372380222}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226101347}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.842 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421025839}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=345900252}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=585917337}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3819522616}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3422366090}) 2024-09-11T02:58:51.843 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3316869854}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497517121}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2956133583}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498455837}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=584520187}) 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.844 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4241908960}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1498717980}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3713867719}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1802675579}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.798+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.845 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3660975067}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2857779006}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1850249248}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977870803}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3555299725}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1314706945}) 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.846 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3496230203}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2408311133}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4134466562}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=722255273}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2464429628}) 2024-09-11T02:58:51.847 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2022674097}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349919721}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3201326763}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3408792305}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2315579610}) 2024-09-11T02:58:51.848 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.802+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2910085}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1633274674}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3066444797}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4070250900}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=336721405}) 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.849 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771348905}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244241040}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907846989}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2993995400}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.850 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4227046644}) 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3134476951}) 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1965202305}) 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2470590932}) 2024-09-11T02:58:51.851 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.806+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1902049435}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338241100}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1573727305}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3105327461}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.905 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2649605916}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2331062813}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2516334934}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=844207125}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=426029042}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=888705505}) 2024-09-11T02:58:51.906 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800481482}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1819562432}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646613922}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3611776795}) 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.907 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3447762446}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=247500350}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=920448822}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468615644}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.810+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3396270679}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095131331}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=210545640}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=631384306}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4136934074}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=966494566}) 2024-09-11T02:58:51.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3967485843}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3954770949}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4257744002}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1820825635}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1662290211}) 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1864275611}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2388905444}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2394354240}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1478391101}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1951085465}) 2024-09-11T02:58:51.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2985686519}) 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2824328309}) 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.814+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3656443410}) 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387456892}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073159103}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415443662}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3170422069}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1747218631}) 2024-09-11T02:58:51.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=632524187}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3044328112}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1381103547}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.818+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3552247899}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4084957262}) 2024-09-11T02:58:51.914 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=195307971}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653407504}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=113241295}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2618730432}) 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.915 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1677163533}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2916796188}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=437411263}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1219555374}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608516558}) 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.916 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2323745547}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367209124}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2144495630}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=481648469}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.917 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=644182213}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2929011523}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=788613026}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.822+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1309563171}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2814121269}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792891708}) 2024-09-11T02:58:51.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082544902}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3221212202}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4120110283}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=296276311}) 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3649893950}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021692901}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3701327962}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851384311}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992562898}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046480235}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=282002248}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3719434071}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1094192462}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2678968436}) 2024-09-11T02:58:51.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3180343347}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.826+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2003123865}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426716110}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3111614555}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515025}) 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2063235749}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1529932412}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=130675986}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2304337988}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2903421697}) 2024-09-11T02:58:51.923 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2816081385}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726395140}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3617832649}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3892157991}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3591515546}) 2024-09-11T02:58:51.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1784615399}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1941398907}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3218685951}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=668258273}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.830+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2913135732}) 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=174716483}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322756945}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046515216}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3419074163}) 2024-09-11T02:58:51.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2954171757}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3732225139}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1479137476}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2264720050}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3537005146}) 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550603144}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=785594515}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2408927364}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2007034545}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3443603307}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3332285232}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=690890795}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091854607}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.834+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=821483581}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952000809}) 2024-09-11T02:58:51.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3771970492}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1076975494}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=105427301}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=221551105}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3366571524}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3865527274}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273008808}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1163001360}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632463763}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3260245228}) 2024-09-11T02:58:51.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2366266779}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1091677733}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=849323627}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2359571397}) 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1306724005}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1508399069}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.838+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=281517141}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=602247766}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3313956194}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176962082}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523673092}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2405114451}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1223196535}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2151574153}) 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.934 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079689423}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1339956420}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114348986}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3647043338}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975006741}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=246452969}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1866829535}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2470146359}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.842+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2599700267}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3811620609}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1367295626}) 2024-09-11T02:58:51.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938717784}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2924324418}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513568897}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=691010031}) 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.937 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4268055668}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3331542556}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3473872570}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2275784536}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=425716943}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.938 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230485700}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1941292118}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1116442821}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=627338526}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2295173020}) 2024-09-11T02:58:51.939 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1835960974}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3892397985}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=841987746}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.846+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2671469374}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695926770}) 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=644143411}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619933557}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1680617957}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3444985403}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4272102189}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=768875346}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052274075}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.850+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1876035866}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=23,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=31} crc {logm=1102053975,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2867275173,mon_config_key=484053719,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=317851720,osd_snap=3117249633}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=26} crc {logm=1003280229,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2082086268,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=1596038039,osd_metadata=1477139452,osd_pg_creating=317851719,osd_snap=2759177049}) 2024-09-11T02:58:51.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=925336626,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2082086268,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=317851719,osd_snap=1396962549}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=20,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33,osdmap=1} crc {logm=3943790383,mdsmap=92030302,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2867275173,mon_config_key=2106052021,monmap=416611756,nvmeofgw=598337686,osd_metadata=1810866017,osd_pg_creating=317851719,osd_snap=1803073604,osdmap=1586882630}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=24,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=2791912376,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2082086268,mon_config_key=3492741104,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=317851719,osd_snap=3875409742}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30} crc {logm=1940592206,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1737091844,mon_config_key=2106052021,monmap=416611756,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=317851719,osd_snap=1102485605}) 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26} crc {logm=2391339291,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=317851719,osd_snap=328602650}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=24,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=1895297296,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2901436849,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2334082449,osd_pg_creating=317851719,osd_snap=378807334}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=2023710765,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1654261804,mon_config_key=2204150802,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139453,osd_pg_creating=317851719,osd_snap=3734035330}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1515653928}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=3099437117}) 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.944 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=618802987}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=2508062136}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=730081477}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1084638916}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.945 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1440617210}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1616625474}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.854+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=1858615167,osdmap=1468541397}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2882986478}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1950850082}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.946 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1679288293}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=545283454}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3434138439}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4236041554}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3024391974}) 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.947 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2601606380}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=678683585}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1993887450}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1732665776}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=724769085}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.948 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=707140953}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=503887319}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2115849408}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3069177012}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4009208260}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.858+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=795045094}) 2024-09-11T02:58:51.949 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2769595194}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2619865924}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1212130788}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=181848727}) 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.950 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=737310354}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4123669452}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3219766616}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=777100484}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.862+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=518457387}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1194555492}) 2024-09-11T02:58:51.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1595127523}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1547838226}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3051019093}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2642397258}) 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1143892111}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=724519963}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3273521192}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.870+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1640440289}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=4113099578}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=937144138}) 2024-09-11T02:58:51.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=35} crc {osdmap=3407140415}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=34} crc {osdmap=2011633867}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=35} crc {osdmap=2547112264}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=35} crc {osdmap=395400359}) 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=33} crc {osdmap=3842495255}) 2024-09-11T02:58:51.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T02:58:51.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=35} crc {osdmap=4111798882}) 2024-09-11T02:58:51.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T02:58:51.882+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=36} crc {osdmap=2405404622}) 2024-09-11T02:58:52.700 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T02:58:52.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:58:52.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient: get_monmap_and_config 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient: build_initial_monmap 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient: monmap: 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:52.847721+0000 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:52.847721+0000 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:52.846 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:52.847 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 1 Processor -- start 2024-09-11T02:58:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 1 -- start start 2024-09-11T02:58:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient: init 2024-09-11T02:58:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient: _add_conns ranks=[2,4,0,9,10,7,3,1,5,8,6] 2024-09-11T02:58:52.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac06f6c0 0x7f97ac072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient(hunting): picked mon.noname-b con 0x7f97ac06f6c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f97ac06faa0 0x7f97ac06fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient(hunting): picked mon.noname-c con 0x7f97ac06faa0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97ac0703c0 addr v1:172.21.15.98:6789/0 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97ac0a8750 con 0x7f97ac0703c0 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f97ac071ff0 con 0x7f97ac06f6c0 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f97ac0716c0 con 0x7f97ac06faa0 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b35d2640 10 monclient(hunting): authenticate will time out at 1486.727725s 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b136e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac06f6c0 0x7f97ac072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:52.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b1b6f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f97ac0703c0 0x7f97ac1397c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50024/0 (socket says 172.21.15.98:50024) 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b1b6f640 1 -- 172.21.15.98:0/354601305 learned_addr learned my addr 172.21.15.98:0/354601305 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b0b6d640 1 --2- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f97ac06faa0 0x7f97ac06fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b136e640 10 monclient(hunting): get_auth_request con 0x7f97ac06f6c0 auth_method 0 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b136e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b136e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b136e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b0b6d640 10 monclient(hunting): get_auth_request con 0x7f97ac06faa0 auth_method 0 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b0b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b0b6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.842+0000 7f97b0b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:52.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b0b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b0b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b0b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 10 monclient(hunting): handle_auth_done global_id 64181 payload 274 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 1 -- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f97ac06faa0 msgr2=0x7f97ac06fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 1 --2- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f97ac06faa0 0x7f97ac06fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 1 -- 172.21.15.98:0/354601305 >> v1:172.21.15.98:6789/0 conn(0x7f97ac0703c0 legacy=0x7f97ac1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 10 monclient: _finish_hunting 0 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 1 monclient: found mon.noname-b 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 1 -- 172.21.15.98:0/354601305 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 848646525 0 0) 0x7f97ac0a8750 con 0x7f97ac0703c0 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 1 -- 172.21.15.98:0/354601305 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac0a64a0 con 0x7f97ac06f6c0 2024-09-11T02:58:52.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b136e640 1 --2- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac06f6c0 0x7f97ac072940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009dc0 tx=0x7f97a801c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdee3a3a6eac2822 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 1 -- 172.21.15.98:0/354601305 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f97a80a6980 con 0x7f97ac06f6c0 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: dump: 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:52.860 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: _finish_auth 0 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: _check_auth_tickets 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:52.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 1 -- 172.21.15.98:0/354601305 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a80a6b40 con 0x7f97ac06f6c0 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 5 monclient: authenticate success, global_id 64181 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 1 -- 172.21.15.98:0/354601305 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f97a80a6cf0 con 0x7f97ac06f6c0 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 4 set_mon_vals no callback set 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97a3fff640 10 monclient: dump: 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.862 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 10 monclient: shutdown 2024-09-11T02:58:52.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 -- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac06f6c0 msgr2=0x7f97ac072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 --2- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac06f6c0 0x7f97ac072940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009dc0 tx=0x7f97a801c850 comp rx=0 tx=0).stop 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 -- 172.21.15.98:0/354601305 shutdown_connections 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 --2- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f97ac06faa0 0x7f97ac06fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 --2- 172.21.15.98:0/354601305 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f97ac06f6c0 0x7f97ac072940 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 -- 172.21.15.98:0/354601305 >> 172.21.15.98:0/354601305 conn(0x7f97ac0acac0 msgr2=0x7f97ac0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 -- 172.21.15.98:0/354601305 shutdown_connections 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 1 -- 172.21.15.98:0/354601305 wait complete. 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 10 monclient: build_initial_monmap 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.846+0000 7f97b35d2640 10 monclient: monmap: 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:52.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:52.865 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 librados: starting msgr at 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 librados: starting objecter 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 Processor -- start 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 -- start start 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 librados: setting wanted keys 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 librados: calling monclient init 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: init 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _add_conns ranks=[1,5,6,8,7,2,3,0,4] 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06f6c0 0x7f97ac1d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): picked mon.noname-f con 0x7f97ac06f6c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:52.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97ac06faa0 0x7f97ac1d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): picked mon.noname-h con 0x7f97ac06faa0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f97ac0703c0 0x7f97ac13dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): picked mon.noname-d con 0x7f97ac0703c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97ac144850 con 0x7f97ac06f6c0 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f97ac1446d0 con 0x7f97ac06faa0 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f97ac1449d0 con 0x7f97ac0703c0 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient(hunting): authenticate will time out at 1486.732717s 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f97ac0703c0 0x7f97ac13dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:52.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f97ac0703c0 0x7f97ac13dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44556/0 (socket says 172.21.15.98:44556) 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 -- 172.21.15.98:0/3777962814 learned_addr learned my addr 172.21.15.98:0/3777962814 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): get_auth_request con 0x7f97ac0703c0 auth_method 0 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient(hunting): handle_auth_done global_id 44367 payload 824 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b0b6d640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97ac06faa0 0x7f97ac1d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 -- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97ac06faa0 msgr2=0x7f97ac1d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97ac06faa0 0x7f97ac1d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:52.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 -- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06f6c0 msgr2=0x7f97ac1d1c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06f6c0 0x7f97ac1d1c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient: _finish_hunting 0 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 monclient: found mon.noname-d 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 -- 172.21.15.98:0/3777962814 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ac122c00 con 0x7f97ac0703c0 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b1b6f640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f97ac0703c0 0x7f97ac13dd20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c390 tx=0x7f97a400c850 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 client.?.objecter ms_handle_connect 0x7f97ac0703c0 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 1 -- 172.21.15.98:0/3777962814 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f97a40047f0 con 0x7f97ac0703c0 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:52.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: dump: 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:52.870 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: _finish_auth 0 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 1 -- 172.21.15.98:0/3777962814 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a40049d0 con 0x7f97ac0703c0 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 5 monclient: authenticate success, global_id 44367 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 1 -- 172.21.15.98:0/3777962814 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f97a40133c0 con 0x7f97ac0703c0 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _renew_subs 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:52.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ac123480 con 0x7f97ac0703c0 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 monclient: dump: 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:52.872 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a37fe640 4 set_mon_vals no callback set 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 client.44367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _renew_subs 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ac0a5030 con 0x7f97ac0703c0 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 librados: init done 2024-09-11T02:58:52.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 1 -- 172.21.15.98:0/3777962814 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f97a4004b80 con 0x7f97ac0703c0 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ac072940 con 0x7f97ac0703c0 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f979003df40 0x7f9790040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 1 -- 172.21.15.98:0/3777962814 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(362..362 src has 1..362) ==== 6202+0+0 (secure 0 0 0) 0x7f97a404f510 con 0x7f97ac0703c0 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 client.44367.objecter ms_dispatch 0x7f97ac000ba0 osd_map(362..362 src has 1..362) 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 3 client.44367.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 3 client.44367.objecter handle_osd_map decoding full epoch 362 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 20 client.44367.objecter dump_active .. 0 homeless 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b136e640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f979003df40 0x7f9790040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b136e640 10 monclient: get_auth_request con 0x7f979003df40 auth_method 0 2024-09-11T02:58:52.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97b136e640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f979003df40 0x7f9790040400 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f97a80af000 tx=0x7f97a80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:52.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.850+0000 7f97a1ffb640 10 client.44367.objecter ms_handle_connect 0x7f979003df40 2024-09-11T02:58:52.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.854+0000 7f97a1ffb640 1 -- 172.21.15.98:0/3777962814 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f97a4011360 con 0x7f97ac0703c0 2024-09-11T02:58:52.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.854+0000 7f97a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:52.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:52.854+0000 7f97a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:53.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.090+0000 7f97b35d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:58:53.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.090+0000 7f97b35d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:53.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.090+0000 7f97b35d2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:53.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.090+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97ac13f790 con 0x7f97ac0703c0 2024-09-11T02:58:53.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.090+0000 7f97a1ffb640 1 -- 172.21.15.98:0/3777962814 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f97a4013a80 con 0x7f97ac0703c0 2024-09-11T02:58:53.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.090+0000 7f97a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:53.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.090+0000 7f97a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:53.093 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":68,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:58:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 10 librados: watch_flush enter 2024-09-11T02:58:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 10 librados: watch_flush exit 2024-09-11T02:58:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 20 client.44367.objecter shutdown clearing up homeless session... 2024-09-11T02:58:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 10 client.44367.objecter successfully canceled tick 2024-09-11T02:58:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f979003df40 msgr2=0x7f9790040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f979003df40 0x7f9790040400 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7f97a80af000 tx=0x7f97a80a7000 comp rx=0 tx=0).stop 2024-09-11T02:58:53.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 10 monclient: shutdown 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f97ac0703c0 msgr2=0x7f97ac13dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f97ac0703c0 0x7f97ac13dd20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c390 tx=0x7f97a400c850 comp rx=0 tx=0).stop 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 shutdown_connections 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f979003df40 0x7f9790040400 unknown :-1 s=CLOSED pgs=3544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f97ac0703c0 0x7f97ac13dd20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f97ac06faa0 0x7f97ac1d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 --2- 172.21.15.98:0/3777962814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97ac06f6c0 0x7f97ac1d1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 >> 172.21.15.98:0/3777962814 conn(0x7f97ac0acac0 msgr2=0x7f97ac140ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 shutdown_connections 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 -- 172.21.15.98:0/3777962814 wait complete. 2024-09-11T02:58:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.094+0000 7f97b35d2640 1 librados: shutdown 2024-09-11T02:58:53.112 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T02:58:53.112 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T02:58:53.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T02:58:53.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:58:53.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.230+0000 7ff34e5bf640 10 monclient: get_monmap_and_config 2024-09-11T02:58:53.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.230+0000 7ff34e5bf640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient: monmap: 2024-09-11T02:58:53.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.235 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.235 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:53.237317+0000 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:53.237317+0000 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.236 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:53.237 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:53.237 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 1 Processor -- start 2024-09-11T02:58:53.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 1 -- start start 2024-09-11T02:58:53.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient: init 2024-09-11T02:58:53.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient: _add_conns ranks=[5,3,1,6,7,4,2,10,8,0,9] 2024-09-11T02:58:53.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff3480a46a0 0x7ff3480a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient(hunting): picked mon.noname-h con 0x7ff3480a46a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a4fc0 0x7ff348139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient(hunting): picked mon.noname-g con 0x7ff3480a4fc0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff348139e70 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3480a17b0 con 0x7ff348139e70 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff3480a0750 con 0x7ff3480a4fc0 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7ff3480775d0 con 0x7ff3480a46a0 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff34e5bf640 10 monclient(hunting): authenticate will time out at 1487.117864s 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff3480a46a0 0x7ff3480a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff3480a46a0 0x7ff3480a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:38466/0 (socket says 172.21.15.98:38466) 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a4fc0 0x7ff348139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 -- 172.21.15.98:0/883136991 learned_addr learned my addr 172.21.15.98:0/883136991 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): get_auth_request con 0x7ff3480a46a0 auth_method 0 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 10 monclient(hunting): get_auth_request con 0x7ff3480a4fc0 auth_method 0 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff346ffd640 1 -- 172.21.15.98:0/883136991 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1529244127 0 0) 0x7ff3480a17b0 con 0x7ff348139e70 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff346ffd640 10 monclient(hunting): _negotiate 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff346ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff346ffd640 10 monclient(hunting): my global_id is 25345 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff346ffd640 1 -- 172.21.15.98:0/883136991 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff330003530 con 0x7ff348139e70 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient(hunting): handle_auth_done global_id 20093 payload 274 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 -- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a4fc0 msgr2=0x7ff348139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:53.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 --2- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a4fc0 0x7ff348139930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 -- 172.21.15.98:0/883136991 >> v1:172.21.15.190:6789/0 conn(0x7ff348139e70 legacy=0x7ff348140bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 monclient: found mon.noname-h 2024-09-11T02:58:53.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T02:58:53.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff347fff640 1 -- 172.21.15.98:0/883136991 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3480a0200 con 0x7ff3480a46a0 2024-09-11T02:58:53.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.234+0000 7ff3477fe640 1 --2- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a4fc0 0x7ff348139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff347fff640 1 --2- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff3480a46a0 0x7ff3480a4a80 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff3380098a0 tx=0x7ff33801c440 comp rx=0 tx=0).ready entity=mon.5 client_cookie=5bfe12a29afc3114 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 1 -- 172.21.15.98:0/883136991 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff3380b0070 con 0x7ff3480a46a0 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: dump: 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.249 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 1 -- 172.21.15.98:0/883136991 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3380aa810 con 0x7ff3480a46a0 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 5 monclient: authenticate success, global_id 20093 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 1 -- 172.21.15.98:0/883136991 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff3380aa9c0 con 0x7ff3480a46a0 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 4 set_mon_vals no callback set 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff346ffd640 10 monclient: dump: 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 10 monclient: shutdown 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 -- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff3480a46a0 msgr2=0x7ff3480a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 --2- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff3480a46a0 0x7ff3480a4a80 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff3380098a0 tx=0x7ff33801c440 comp rx=0 tx=0).stop 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 -- 172.21.15.98:0/883136991 shutdown_connections 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 --2- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a4fc0 0x7ff348139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 --2- 172.21.15.98:0/883136991 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff3480a46a0 0x7ff3480a4a80 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 -- 172.21.15.98:0/883136991 >> 172.21.15.98:0/883136991 conn(0x7ff34806ed00 msgr2=0x7ff34806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 -- 172.21.15.98:0/883136991 shutdown_connections 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 -- 172.21.15.98:0/883136991 wait complete. 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 10 monclient: monmap: 2024-09-11T02:58:53.253 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 librados: starting msgr at 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.238+0000 7ff34e5bf640 1 librados: starting objecter 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 Processor -- start 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 -- start start 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 librados: setting wanted keys 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 librados: calling monclient init 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient: init 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient: _add_conns ranks=[3,4,8,0,5,6,7,2,1] 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a46a0 0x7ff3481d1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): picked mon.noname-g con 0x7ff3480a46a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff3480a4fc0 0x7ff3481d2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): picked mon.noname-c con 0x7ff3480a4fc0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff348139e70 0x7ff34813dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): picked mon.noname-e con 0x7ff348139e70 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7ff3480a53a0 con 0x7ff3480a46a0 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff348077850 con 0x7ff3480a4fc0 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff3481404b0 con 0x7ff348139e70 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): authenticate will time out at 1487.124759s 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff3480a4fc0 0x7ff3481d2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff3480a4fc0 0x7ff3481d2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59764/0 (socket says 172.21.15.98:59764) 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 -- 172.21.15.98:0/676991260 learned_addr learned my addr 172.21.15.98:0/676991260 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): get_auth_request con 0x7ff3480a4fc0 auth_method 0 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff348139e70 0x7ff34813dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient(hunting): get_auth_request con 0x7ff348139e70 auth_method 0 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a46a0 0x7ff3481d1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): get_auth_request con 0x7ff3480a46a0 auth_method 0 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient(hunting): handle_auth_done global_id 24007 payload 824 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff348139e70 msgr2=0x7ff34813dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff348139e70 0x7ff34813dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a46a0 msgr2=0x7ff3481d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7ff3480a46a0 0x7ff3481d1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 monclient: found mon.noname-c 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 -- 172.21.15.98:0/676991260 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3480a8900 con 0x7ff3480a4fc0 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3477fe640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff3480a4fc0 0x7ff3481d2060 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ebb0 tx=0x7ff33c00c520 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 client.?.objecter ms_handle_connect 0x7ff3480a4fc0 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff33c017070 con 0x7ff3480a4fc0 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: dump: 2024-09-11T02:58:53.259 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.260 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33c00cc30 con 0x7ff3480a4fc0 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff33c00cde0 con 0x7ff3480a4fc0 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: dump: 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.261 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.262 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff3467fc640 4 set_mon_vals no callback set 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 5 monclient: authenticate success, global_id 24007 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient: _renew_subs 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34806f530 con 0x7ff3480a4fc0 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 client.24007.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient: _renew_subs 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff348141420 con 0x7ff3480a4fc0 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 librados: init done 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813e1c0 0x7ff34813e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff3480a5520 con 0x7ff34813e1c0 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ff33c024060 con 0x7ff3480a4fc0 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813e1c0 0x7ff34813e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient: get_auth_request con 0x7ff34813e1c0 auth_method 0 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff33403df40 0x7ff334040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 10 monclient(hunting): handle_auth_done global_id 44496 payload 824 2024-09-11T02:58:53.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(362..362 src has 1..362) ==== 6202+0+0 (secure 0 0 0) 0x7ff33c00f040 con 0x7ff3480a4fc0 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 client.24007.objecter ms_dispatch 0x7ff348000ba0 osd_map(362..362 src has 1..362) 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 3 client.24007.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 3 client.24007.objecter handle_osd_map decoding full epoch 362 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 1 -- 172.21.15.98:0/676991260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff34813ff60 con 0x7ff34813e1c0 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 20 client.24007.objecter dump_active .. 0 homeless 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 client.24007.objecter ms_handle_connect 0x7ff34813e1c0 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 client.24007.objecter resend_mon_ops 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff347fff640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813e1c0 0x7ff34813e5a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3380ad000 tx=0x7ff3380191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff3380b9070 con 0x7ff34813e1c0 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: dump: 2024-09-11T02:58:53.265 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.266 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff33403df40 0x7ff334040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff34cb5c640 10 monclient: get_auth_request con 0x7ff33403df40 auth_method 0 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff3380b1040 con 0x7ff34813e1c0 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813e1c0 msgr2=0x7ff34813e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813e1c0 0x7ff34813e5a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3380ad000 tx=0x7ff3380191e0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813e1c0 msgr2=0x7ff34813e5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.242+0000 7ff344ff9640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813e1c0 0x7ff34813e5a0 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff3380ad000 tx=0x7ff3380191e0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.246+0000 7ff34cb5c640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff33403df40 0x7ff334040400 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7ff340001e10 tx=0x7ff340009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.246+0000 7ff344ff9640 10 client.24007.objecter ms_handle_connect 0x7ff33403df40 2024-09-11T02:58:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813eae0 0x7ff34809d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff34806ca10 con 0x7ff34813eae0 2024-09-11T02:58:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813eae0 0x7ff34809d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient: get_auth_request con 0x7ff34813eae0 auth_method 0 2024-09-11T02:58:53.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 10 monclient(hunting): handle_auth_done global_id 44505 payload 824 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 1 -- 172.21.15.98:0/676991260 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff3480a4a80 con 0x7ff34813eae0 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 10 client.24007.objecter ms_handle_connect 0x7ff34813eae0 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 10 client.24007.objecter resend_mon_ops 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff347fff640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813eae0 0x7ff34809d690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff33801c480 tx=0x7ff3380a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff3380b0050 con 0x7ff34813eae0 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 10 monclient: dump: 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.271 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7ff3380c2070 con 0x7ff34813eae0 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813eae0 msgr2=0x7ff34809d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813eae0 0x7ff34809d690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff33801c480 tx=0x7ff3380a8000 comp rx=0 tx=0).stop 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813eae0 msgr2=0x7ff34809d690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff344ff9640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff34813eae0 0x7ff34809d690 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff33801c480 tx=0x7ff3380a8000 comp rx=0 tx=0).stop 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 159690, 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:53.273 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.274 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.275 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:53.276 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:53.277 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.278 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:53.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.280 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.282 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.283 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.287 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.289 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.294 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.295 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:53.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.297 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.297 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:53.297 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:53.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 10 librados: watch_flush enter 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 10 librados: watch_flush exit 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 20 client.24007.objecter shutdown clearing up homeless session... 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 10 client.24007.objecter successfully canceled tick 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff33403df40 msgr2=0x7ff334040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff33403df40 0x7ff334040400 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7ff340001e10 tx=0x7ff340009040 comp rx=0 tx=0).stop 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 10 monclient: shutdown 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff3480a4fc0 msgr2=0x7ff3481d2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 --2- 172.21.15.98:0/676991260 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff3480a4fc0 0x7ff3481d2060 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00ebb0 tx=0x7ff33c00c520 comp rx=0 tx=0).stop 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34cb5c640 1 -- 172.21.15.98:0/676991260 reap_dead start 2024-09-11T02:58:53.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34cb5c640 1 -- 172.21.15.98:0/676991260 reap_dead start 2024-09-11T02:58:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 shutdown_connections 2024-09-11T02:58:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 >> 172.21.15.98:0/676991260 conn(0x7ff34806ed00 msgr2=0x7ff3480a81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 shutdown_connections 2024-09-11T02:58:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 -- 172.21.15.98:0/676991260 wait complete. 2024-09-11T02:58:53.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.250+0000 7ff34e5bf640 1 librados: shutdown 2024-09-11T02:58:53.300 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:53.300 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T02:58:53.300 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:58:53.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient: get_monmap_and_config 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient: monmap: 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:53.426539+0000 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:53.426539+0000 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.425 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:53.426 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:53.426 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 Processor -- start 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 -- start start 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient: init 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient: _add_conns ranks=[2,4,10,6,8,7,0,9,3,5,1] 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8139b80 0x7fa7d8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): picked mon.noname-b con 0x7fa7d8139b80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d806fe50 0x7fa7d80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d806fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa7d8072640 0x7fa7d80a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): picked mon.noname-f con 0x7fa7d8072640 addr v2:172.21.15.190:3300/0 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d80719c0 con 0x7fa7d8072640 2024-09-11T02:58:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d8071c10 con 0x7fa7d8139b80 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa7d80702d0 con 0x7fa7d806fe50 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7dd985640 10 monclient(hunting): authenticate will time out at 1487.306762s 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d806fe50 0x7fa7d80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d806fe50 0x7fa7d80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59780/0 (socket says 172.21.15.98:59780) 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 -- 172.21.15.98:0/2102550867 learned_addr learned my addr 172.21.15.98:0/2102550867 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): get_auth_request con 0x7fa7d806fe50 auth_method 0 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d6ffd640 1 --2- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8139b80 0x7fa7d8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d6ffd640 10 monclient(hunting): get_auth_request con 0x7fa7d8139b80 auth_method 0 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d77fe640 1 --2- 172.21.15.98:0/2102550867 >> v2:172.21.15.190:3300/0 conn(0x7fa7d8072640 0x7fa7d80a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d77fe640 10 monclient(hunting): get_auth_request con 0x7fa7d8072640 auth_method 0 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_done global_id 24016 payload 274 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 -- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8139b80 msgr2=0x7fa7d8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8139b80 0x7fa7d8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 -- 172.21.15.98:0/2102550867 >> v2:172.21.15.190:3300/0 conn(0x7fa7d8072640 msgr2=0x7fa7d80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/2102550867 >> v2:172.21.15.190:3300/0 conn(0x7fa7d8072640 0x7fa7d80a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 monclient: found mon.noname-c 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d67fc640 1 -- 172.21.15.98:0/2102550867 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d8071370 con 0x7fa7d806fe50 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.422+0000 7fa7d77fe640 1 --2- 172.21.15.98:0/2102550867 >> v2:172.21.15.190:3300/0 conn(0x7fa7d8072640 0x7fa7d80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d806fe50 0x7fa7d80a7a10 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0098a0 tx=0x7fa7cc01c400 comp rx=0 tx=0).ready entity=mon.4 client_cookie=bb6083e88d6ab3b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 1 -- 172.21.15.98:0/2102550867 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7cc0b0070 con 0x7fa7d806fe50 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: dump: 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.435 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 1 -- 172.21.15.98:0/2102550867 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7cc0aa810 con 0x7fa7d806fe50 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 5 monclient: authenticate success, global_id 24016 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 4 set_mon_vals no callback set 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 1 -- 172.21.15.98:0/2102550867 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7cc0a9300 con 0x7fa7d806fe50 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7d5ffb640 10 monclient: dump: 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.437 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 10 monclient: shutdown 2024-09-11T02:58:53.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 -- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d806fe50 msgr2=0x7fa7d80a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 --2- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d806fe50 0x7fa7d80a7a10 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0098a0 tx=0x7fa7cc01c400 comp rx=0 tx=0).stop 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 -- 172.21.15.98:0/2102550867 shutdown_connections 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 --2- 172.21.15.98:0/2102550867 >> v2:172.21.15.190:3300/0 conn(0x7fa7d8072640 0x7fa7d80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 --2- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d806fe50 0x7fa7d80a7a10 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 --2- 172.21.15.98:0/2102550867 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8139b80 0x7fa7d8139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 -- 172.21.15.98:0/2102550867 >> 172.21.15.98:0/2102550867 conn(0x7fa7d80ac730 msgr2=0x7fa7d80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 -- 172.21.15.98:0/2102550867 shutdown_connections 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 -- 172.21.15.98:0/2102550867 wait complete. 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 10 monclient: monmap: 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.440 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 librados: starting msgr at 2024-09-11T02:58:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 librados: starting objecter 2024-09-11T02:58:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 Processor -- start 2024-09-11T02:58:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 -- start start 2024-09-11T02:58:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 librados: setting wanted keys 2024-09-11T02:58:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 1 librados: calling monclient init 2024-09-11T02:58:53.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.426+0000 7fa7dd985640 10 monclient: init 2024-09-11T02:58:53.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient: _add_conns ranks=[8,4,1,0,5,7,6,2,3] 2024-09-11T02:58:53.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7d806fe50 0x7fa7d813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): picked mon.noname-e con 0x7fa7d806fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d8072640 0x7fa7d813abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): picked mon.noname-c con 0x7fa7d8072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d8139b80 0x7fa7d813b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): picked mon.noname-f con 0x7fa7d8139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d8143440 con 0x7fa7d8139b80 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa7d80a88a0 con 0x7fa7d8072640 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa7d81435c0 con 0x7fa7d806fe50 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): authenticate will time out at 1487.312338s 2024-09-11T02:58:53.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d8072640 0x7fa7d813abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7d806fe50 0x7fa7d813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d8072640 0x7fa7d813abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59794/0 (socket says 172.21.15.98:59794) 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 1 -- 172.21.15.98:0/1530097014 learned_addr learned my addr 172.21.15.98:0/1530097014 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): get_auth_request con 0x7fa7d806fe50 auth_method 0 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): get_auth_request con 0x7fa7d8072640 auth_method 0 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d8139b80 0x7fa7d813b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 10 monclient(hunting): get_auth_request con 0x7fa7d8139b80 auth_method 0 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient(hunting): handle_auth_done global_id 24551 payload 824 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d8072640 msgr2=0x7fa7d813abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d8072640 0x7fa7d813abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d8139b80 msgr2=0x7fa7d813b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7d8139b80 0x7fa7d813b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 monclient: found mon.noname-e 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 -- 172.21.15.98:0/1530097014 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d80aa2b0 con 0x7fa7d806fe50 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7d8072640 0x7fa7d813abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d6ffd640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7d806fe50 0x7fa7d813a6b0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800d5d0 tx=0x7fa7c800da90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 client.?.objecter ms_handle_connect 0x7fa7d806fe50 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c8012a60 con 0x7fa7d806fe50 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: dump: 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c8019070 con 0x7fa7d806fe50 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 5 monclient: authenticate success, global_id 24551 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient: _renew_subs 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d80a5950 con 0x7fa7d806fe50 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c8012c40 con 0x7fa7d806fe50 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 client.24551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d57fa640 4 set_mon_vals no callback set 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: dump: 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.454 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient: _renew_subs 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d80a5260 con 0x7fa7d806fe50 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 librados: init done 2024-09-11T02:58:53.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d813bc60 0x7fa7d8133860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d806ca10 con 0x7fa7d813bc60 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d813bc60 0x7fa7d8133860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient: get_auth_request con 0x7fa7d813bc60 auth_method 0 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa7c8013070 con 0x7fa7d806fe50 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_done global_id 64199 payload 824 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 1 -- 172.21.15.98:0/1530097014 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7d80aa800 con 0x7fa7d813bc60 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7c403df40 0x7fa7c4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(362..362 src has 1..362) ==== 6202+0+0 (secure 0 0 0) 0x7fa7c800f040 con 0x7fa7d806fe50 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 client.24551.objecter ms_dispatch 0x7fa7d8000ba0 osd_map(362..362 src has 1..362) 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 3 client.24551.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 3 client.24551.objecter handle_osd_map decoding full epoch 362 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 20 client.24551.objecter dump_active .. 0 homeless 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 client.24551.objecter ms_handle_connect 0x7fa7d813bc60 2024-09-11T02:58:53.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 client.24551.objecter resend_mon_ops 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d813bc60 0x7fa7d8133860 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0097c0 tx=0x7fa7cc0191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7cc0ba070 con 0x7fa7d813bc60 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: dump: 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.458 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7c403df40 0x7fa7c4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 10 monclient: get_auth_request con 0x7fa7c403df40 auth_method 0 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7cc0aa750 con 0x7fa7d813bc60 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d813bc60 msgr2=0x7fa7d8133860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d813bc60 0x7fa7d8133860 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0097c0 tx=0x7fa7cc0191a0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d813bc60 msgr2=0x7fa7d8133860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d813bc60 0x7fa7d8133860 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc0097c0 tx=0x7fa7cc0191a0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7d77fe640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7c403df40 0x7fa7c4040400 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8135c80 tx=0x7fa7d0003cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.430+0000 7fa7bffff640 10 client.24551.objecter ms_handle_connect 0x7fa7c403df40 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8133da0 0x7fa7d8134180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d806e870 con 0x7fa7d8133da0 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8133da0 0x7fa7d8134180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient: get_auth_request con 0x7fa7d8133da0 auth_method 0 2024-09-11T02:58:53.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 10 monclient(hunting): handle_auth_done global_id 64208 payload 824 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 1 -- 172.21.15.98:0/1530097014 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7d81346c0 con 0x7fa7d8133da0 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 10 client.24551.objecter ms_handle_connect 0x7fa7d8133da0 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 10 client.24551.objecter resend_mon_ops 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d67fc640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8133da0 0x7fa7d8134180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc01cca0 tx=0x7fa7cc0a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7cc0b0040 con 0x7fa7d8133da0 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 10 monclient: dump: 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7fa7cc003830 con 0x7fa7d8133da0 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8133da0 msgr2=0x7fa7d8134180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8133da0 0x7fa7d8134180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc01cca0 tx=0x7fa7cc0a8000 comp rx=0 tx=0).stop 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8133da0 msgr2=0x7fa7d8134180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7bffff640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa7d8133da0 0x7fa7d8134180 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc01cca0 tx=0x7fa7cc0a8000 comp rx=0 tx=0).stop 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 25631, 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:53.464 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.465 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.466 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:53.467 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.468 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.469 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.472 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:53.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.476 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:53.482 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.485 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:53.487 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 10 librados: watch_flush enter 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 10 librados: watch_flush exit 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 20 client.24551.objecter shutdown clearing up homeless session... 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 10 client.24551.objecter successfully canceled tick 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7c403df40 msgr2=0x7fa7c4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7c403df40 0x7fa7c4040400 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8135c80 tx=0x7fa7d0003cf0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 10 monclient: shutdown 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7d806fe50 msgr2=0x7fa7d813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7d806fe50 0x7fa7d813a6b0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fa7c800d5d0 tx=0x7fa7c800da90 comp rx=0 tx=0).stop 2024-09-11T02:58:53.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7d77fe640 1 -- 172.21.15.98:0/1530097014 reap_dead start 2024-09-11T02:58:53.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 shutdown_connections 2024-09-11T02:58:53.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 --2- 172.21.15.98:0/1530097014 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa7d806fe50 0x7fa7d813a6b0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 >> 172.21.15.98:0/1530097014 conn(0x7fa7d80ac730 msgr2=0x7fa7d80aaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 shutdown_connections 2024-09-11T02:58:53.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 -- 172.21.15.98:0/1530097014 wait complete. 2024-09-11T02:58:53.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.438+0000 7fa7dd985640 1 librados: shutdown 2024-09-11T02:58:53.490 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:53.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T02:58:53.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:58:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient: get_monmap_and_config 2024-09-11T02:58:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient: monmap: 2024-09-11T02:58:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.612 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.613 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:53.614225+0000 2024-09-11T02:58:53.613 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:53.614225+0000 2024-09-11T02:58:53.616 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.616 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.616 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:53.616 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:53.617 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 1 Processor -- start 2024-09-11T02:58:53.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 1 -- start start 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient: init 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient: _add_conns ranks=[6,0,9,5,8,4,10,1,3,2,7] 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c0a5930 0x7eff8c0a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient(hunting): picked mon.noname-d con 0x7eff8c0a5930 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff8c06fbe0 addr v1:172.21.15.98:6789/0 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff8c0a6250 0x7eff8c138e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient(hunting): picked mon.noname-a con 0x7eff8c0a6250 addr v2:172.21.15.98:3300/0 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8c06ee20 con 0x7eff8c06fbe0 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7eff8c06f4e0 con 0x7eff8c0a6250 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7eff8c0a66e0 con 0x7eff8c0a5930 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff93bf1640 10 monclient(hunting): authenticate will time out at 1487.494523s 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c0a5930 0x7eff8c0a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff8c0a6250 0x7eff8c138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7eff8c0a6250 0x7eff8c138e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38680/0 (socket says 172.21.15.98:38680) 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 -- 172.21.15.98:0/75803249 learned_addr learned my addr 172.21.15.98:0/75803249 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 10 monclient(hunting): get_auth_request con 0x7eff8c0a5930 auth_method 0 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9118c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7eff8c06fbe0 0x7eff8c1386f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50030/0 (socket says 172.21.15.98:50030) 2024-09-11T02:58:53.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): get_auth_request con 0x7eff8c0a6250 auth_method 0 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient(hunting): handle_auth_done global_id 44514 payload 274 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 -- 172.21.15.98:0/75803249 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c0a5930 msgr2=0x7eff8c0a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 --2- 172.21.15.98:0/75803249 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c0a5930 0x7eff8c0a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 -- 172.21.15.98:0/75803249 >> v1:172.21.15.98:6789/0 conn(0x7eff8c06fbe0 legacy=0x7eff8c1386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 monclient: found mon.noname-a 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.610+0000 7eff9218e640 1 -- 172.21.15.98:0/75803249 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c06f230 con 0x7eff8c0a6250 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9218e640 1 --2- 172.21.15.98:0/75803249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff8c0a6250 0x7eff8c138e00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff84009e50 tx=0x7eff8401c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdece28d1e6bbc01 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 1 -- 172.21.15.98:0/75803249 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff840090d0 con 0x7eff8c0a6250 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: dump: 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.629 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.630 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 1 -- 172.21.15.98:0/75803249 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff840a5a20 con 0x7eff8c0a6250 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 1 -- 172.21.15.98:0/75803249 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff840a5c10 con 0x7eff8c0a6250 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 5 monclient: authenticate success, global_id 44514 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 4 set_mon_vals no callback set 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff9098b640 10 monclient: dump: 2024-09-11T02:58:53.631 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.632 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 10 monclient: shutdown 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 -- 172.21.15.98:0/75803249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff8c0a6250 msgr2=0x7eff8c138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 --2- 172.21.15.98:0/75803249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff8c0a6250 0x7eff8c138e00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7eff84009e50 tx=0x7eff8401c8f0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 -- 172.21.15.98:0/75803249 shutdown_connections 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 --2- 172.21.15.98:0/75803249 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff8c0a6250 0x7eff8c138e00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 --2- 172.21.15.98:0/75803249 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c0a5930 0x7eff8c0a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 -- 172.21.15.98:0/75803249 >> 172.21.15.98:0/75803249 conn(0x7eff8c0aa520 msgr2=0x7eff8c0a6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 -- 172.21.15.98:0/75803249 shutdown_connections 2024-09-11T02:58:53.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 -- 172.21.15.98:0/75803249 wait complete. 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 10 monclient: monmap: 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.634 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 librados: starting msgr at 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 librados: starting objecter 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 Processor -- start 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 -- start start 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 librados: setting wanted keys 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 1 librados: calling monclient init 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.614+0000 7eff93bf1640 10 monclient: init 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient: _add_conns ranks=[6,8,5,0,2,3,4,7,1] 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c06fbe0 0x7eff8c1211a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): picked mon.noname-d con 0x7eff8c06fbe0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff8c0a5930 0x7eff8c1d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): picked mon.noname-e con 0x7eff8c0a5930 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff8c0a6250 0x7eff8c13f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): picked mon.noname-h con 0x7eff8c0a6250 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7eff8c139550 con 0x7eff8c0a6250 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7eff8c144550 con 0x7eff8c06fbe0 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7eff8c1c93e0 con 0x7eff8c0a5930 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): authenticate will time out at 1487.500410s 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff8c0a5930 0x7eff8c1d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c06fbe0 0x7eff8c1211a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff8c0a5930 0x7eff8c1d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39728/0 (socket says 172.21.15.98:39728) 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 -- 172.21.15.98:0/1874462650 learned_addr learned my addr 172.21.15.98:0/1874462650 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): get_auth_request con 0x7eff8c0a5930 auth_method 0 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): get_auth_request con 0x7eff8c06fbe0 auth_method 0 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient(hunting): handle_auth_done global_id 24560 payload 824 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9218e640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff8c0a6250 0x7eff8c13f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c06fbe0 msgr2=0x7eff8c1211a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7eff8c06fbe0 0x7eff8c1211a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff8c0a6250 msgr2=0x7eff8c13f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff8c0a6250 0x7eff8c13f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 monclient: found mon.noname-e 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 -- 172.21.15.98:0/1874462650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c0a9860 con 0x7eff8c0a5930 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9218e640 10 monclient: get_auth_request con 0x7eff8c0a6250 auth_method 0 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9218e640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff8c0a6250 0x7eff8c13f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9118c640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff8c0a5930 0x7eff8c1d1a40 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c3f0 tx=0x7eff7c00c8b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 client.?.objecter ms_handle_connect 0x7eff8c0a5930 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff7c0047f0 con 0x7eff8c0a5930 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:53.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: dump: 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.640 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c0049d0 con 0x7eff8c0a5930 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff7c013400 con 0x7eff8c0a5930 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff83fff640 4 set_mon_vals no callback set 2024-09-11T02:58:53.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: dump: 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 5 monclient: authenticate success, global_id 24560 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient: _renew_subs 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8c139140 con 0x7eff8c0a5930 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 client.24560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient: _renew_subs 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8c0a9360 con 0x7eff8c0a5930 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 librados: init done 2024-09-11T02:58:53.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c1d1f80 0x7eff8c122340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7eff8c06f930 con 0x7eff8c1d1f80 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7eff7c004b80 con 0x7eff8c0a5930 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c1d1f80 0x7eff8c122340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient: get_auth_request con 0x7eff8c1d1f80 auth_method 0 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff7003df40 0x7eff70040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 10 monclient(hunting): handle_auth_done global_id 24034 payload 824 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 1 -- 172.21.15.98:0/1874462650 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff8c0a5290 con 0x7eff8c1d1f80 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(362..362 src has 1..362) ==== 6202+0+0 (secure 0 0 0) 0x7eff7c04f0c0 con 0x7eff8c0a5930 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 client.24560.objecter ms_dispatch 0x7eff8c000ba0 osd_map(362..362 src has 1..362) 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 3 client.24560.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 3 client.24560.objecter handle_osd_map decoding full epoch 362 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 20 client.24560.objecter dump_active .. 0 homeless 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 client.24560.objecter ms_handle_connect 0x7eff8c1d1f80 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 client.24560.objecter resend_mon_ops 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9198d640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c1d1f80 0x7eff8c122340 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7eff88011900 tx=0x7eff88011dc0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff88019070 con 0x7eff8c1d1f80 2024-09-11T02:58:53.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: dump: 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.646 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9218e640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff7003df40 0x7eff70040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff9218e640 10 monclient: get_auth_request con 0x7eff7003df40 auth_method 0 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7eff88004300 con 0x7eff8c1d1f80 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c1d1f80 msgr2=0x7eff8c122340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c1d1f80 0x7eff8c122340 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7eff88011900 tx=0x7eff88011dc0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c1d1f80 msgr2=0x7eff8c122340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.618+0000 7eff827fc640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c1d1f80 0x7eff8c122340 secure :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7eff88011900 tx=0x7eff88011dc0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.622+0000 7eff827fc640 10 client.24560.objecter ms_handle_connect 0x7eff7003df40 2024-09-11T02:58:53.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.622+0000 7eff9218e640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff7003df40 0x7eff70040400 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7eff840ae000 tx=0x7eff840a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c13f8a0 0x7eff8c13fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7eff8c1401f0 con 0x7eff8c13f8a0 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c13f8a0 0x7eff8c13fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient: get_auth_request con 0x7eff8c13f8a0 auth_method 0 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 10 monclient(hunting): handle_auth_done global_id 24043 payload 824 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 1 -- 172.21.15.98:0/1874462650 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7eff8c0a5d10 con 0x7eff8c13f8a0 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9198d640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c13f8a0 0x7eff8c13fcb0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7eff88011900 tx=0x7eff880046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 10 client.24560.objecter ms_handle_connect 0x7eff8c13f8a0 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 10 client.24560.objecter resend_mon_ops 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff88019040 con 0x7eff8c13f8a0 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 10 monclient: dump: 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.651 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7eff88003c40 con 0x7eff8c13f8a0 2024-09-11T02:58:53.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c13f8a0 msgr2=0x7eff8c13fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c13f8a0 0x7eff8c13fcb0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7eff88011900 tx=0x7eff880046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c13f8a0 msgr2=0x7eff8c13fcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff827fc640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7eff8c13f8a0 0x7eff8c13fcb0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:53.653 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 549410, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.654 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.655 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.656 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.657 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.658 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.659 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.660 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.664 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.667 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:53.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.669 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.670 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.671 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.672 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:53.674 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:53.675 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.676 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:53.677 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 10 librados: watch_flush enter 2024-09-11T02:58:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 10 librados: watch_flush exit 2024-09-11T02:58:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 20 client.24560.objecter shutdown clearing up homeless session... 2024-09-11T02:58:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 10 client.24560.objecter successfully canceled tick 2024-09-11T02:58:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff7003df40 msgr2=0x7eff70040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff7003df40 0x7eff70040400 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7eff840ae000 tx=0x7eff840a7070 comp rx=0 tx=0).stop 2024-09-11T02:58:53.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 10 monclient: shutdown 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff8c0a5930 msgr2=0x7eff8c1d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 --2- 172.21.15.98:0/1874462650 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff8c0a5930 0x7eff8c1d1a40 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00c3f0 tx=0x7eff7c00c8b0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9218e640 1 -- 172.21.15.98:0/1874462650 reap_dead start 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff9218e640 1 -- 172.21.15.98:0/1874462650 reap_dead start 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 shutdown_connections 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 >> 172.21.15.98:0/1874462650 conn(0x7eff8c0aa520 msgr2=0x7eff8c0a4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 shutdown_connections 2024-09-11T02:58:53.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 -- 172.21.15.98:0/1874462650 wait complete. 2024-09-11T02:58:53.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.626+0000 7eff93bf1640 1 librados: shutdown 2024-09-11T02:58:53.681 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:53.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T02:58:53.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:58:53.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient: get_monmap_and_config 2024-09-11T02:58:53.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient: monmap: 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:53.801469+0000 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:53.801469+0000 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.801 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.801 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.801 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:53.801 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:53.801 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 1 Processor -- start 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 1 -- start start 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient: init 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient: _add_conns ranks=[10,8,0,7,1,4,3,9,6,5,2] 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcde0139b80 0x7fcde0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient(hunting): picked mon.noname-f con 0x7fcde0139b80 addr v2:172.21.15.190:3300/0 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcde006fe50 0x7fcde00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient(hunting): picked mon.noname-e con 0x7fcde006fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcde0072640 addr v1:172.21.15.98:6789/0 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde0071370 con 0x7fcde0072640 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fcde00719c0 con 0x7fcde0139b80 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcde0071c10 con 0x7fcde006fe50 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde71cc640 10 monclient(hunting): authenticate will time out at 1487.681828s 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcde006fe50 0x7fcde00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde5769640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcde0072640 0x7fcde00a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50040/0 (socket says 172.21.15.98:50040) 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde5769640 1 -- 172.21.15.98:0/623743297 learned_addr learned my addr 172.21.15.98:0/623743297 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:58:53.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): get_auth_request con 0x7fcde006fe50 auth_method 0 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient(hunting): handle_auth_done global_id 24569 payload 274 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcde4f68640 1 --2- 172.21.15.98:0/623743297 >> v2:172.21.15.190:3300/0 conn(0x7fcde0139b80 0x7fcde0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 1 -- 172.21.15.98:0/623743297 >> v2:172.21.15.190:3300/0 conn(0x7fcde0139b80 msgr2=0x7fcde0139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 1 --2- 172.21.15.98:0/623743297 >> v2:172.21.15.190:3300/0 conn(0x7fcde0139b80 0x7fcde0139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 1 -- 172.21.15.98:0/623743297 >> v1:172.21.15.98:6789/0 conn(0x7fcde0072640 legacy=0x7fcde00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 1 monclient: found mon.noname-e 2024-09-11T02:58:53.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddf7fe640 1 -- 172.21.15.98:0/623743297 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2294144659 0 0) 0x7fcde0071370 con 0x7fcde0072640 2024-09-11T02:58:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddffff640 1 -- 172.21.15.98:0/623743297 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde00a9600 con 0x7fcde006fe50 2024-09-11T02:58:53.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.798+0000 7fcddf7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:58:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddffff640 1 --2- 172.21.15.98:0/623743297 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcde006fe50 0x7fcde00a7a10 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009d90 tx=0x7fcdd001c830 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b31a9ea7e4e49978 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 1 -- 172.21.15.98:0/623743297 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdd00b2070 con 0x7fcde006fe50 2024-09-11T02:58:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:53.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: dump: 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.813 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 1 -- 172.21.15.98:0/623743297 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd00a5a70 con 0x7fcde006fe50 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 5 monclient: authenticate success, global_id 24569 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 1 -- 172.21.15.98:0/623743297 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdd00a5c60 con 0x7fcde006fe50 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 4 set_mon_vals no callback set 2024-09-11T02:58:53.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddf7fe640 10 monclient: dump: 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.815 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: shutdown 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- 172.21.15.98:0/623743297 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcde006fe50 msgr2=0x7fcde00a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 --2- 172.21.15.98:0/623743297 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcde006fe50 0x7fcde00a7a10 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009d90 tx=0x7fcdd001c830 comp rx=0 tx=0).stop 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- 172.21.15.98:0/623743297 shutdown_connections 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 --2- 172.21.15.98:0/623743297 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcde006fe50 0x7fcde00a7a10 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 --2- 172.21.15.98:0/623743297 >> v2:172.21.15.190:3300/0 conn(0x7fcde0139b80 0x7fcde0139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- 172.21.15.98:0/623743297 >> 172.21.15.98:0/623743297 conn(0x7fcde00ac730 msgr2=0x7fcde00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- 172.21.15.98:0/623743297 shutdown_connections 2024-09-11T02:58:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- 172.21.15.98:0/623743297 wait complete. 2024-09-11T02:58:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: monmap: 2024-09-11T02:58:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.817 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:53.819 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 librados: starting msgr at 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 librados: starting objecter 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 Processor -- start 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- start start 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 librados: setting wanted keys 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 librados: calling monclient init 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: init 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: _add_conns ranks=[6,5,7,8,2,1,0,4,3] 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde006fe50 0x7fcde0138120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): picked mon.noname-d con 0x7fcde006fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcde0072640 0x7fcde0138660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): picked mon.noname-h con 0x7fcde0072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcde0139b80 0x7fcde013e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): picked mon.noname-i con 0x7fcde0139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcde0144450 con 0x7fcde0072640 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcde00a8870 con 0x7fcde006fe50 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fcde01445d0 con 0x7fcde0139b80 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient(hunting): authenticate will time out at 1487.686965s 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde006fe50 0x7fcde0138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde006fe50 0x7fcde0138120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44602/0 (socket says 172.21.15.98:44602) 2024-09-11T02:58:53.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 -- 172.21.15.98:0/779633999 learned_addr learned my addr 172.21.15.98:0/779633999 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): get_auth_request con 0x7fcde006fe50 auth_method 0 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient(hunting): handle_auth_done global_id 44394 payload 824 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddffff640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcde0072640 0x7fcde0138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcde0139b80 msgr2=0x7fcde013e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcde0139b80 0x7fcde013e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcde0072640 msgr2=0x7fcde0138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcde0072640 0x7fcde0138660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 monclient: found mon.noname-d 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 -- 172.21.15.98:0/779633999 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde00aba70 con 0x7fcde006fe50 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde4f68640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde006fe50 0x7fcde0138120 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400c7f0 tx=0x7fcdd400ccb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 client.?.objecter ms_handle_connect 0x7fcde006fe50 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdd4015070 con 0x7fcde006fe50 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: dump: 2024-09-11T02:58:53.824 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.825 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdd4007a90 con 0x7fcde006fe50 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdd4007c40 con 0x7fcde006fe50 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 5 monclient: authenticate success, global_id 44394 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: _renew_subs 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde0122830 con 0x7fcde006fe50 2024-09-11T02:58:53.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddeffd640 4 set_mon_vals no callback set 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcddd7fa640 10 monclient: dump: 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.827 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 client.44394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: _renew_subs 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde01230d0 con 0x7fcde006fe50 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.802+0000 7fcde71cc640 1 librados: init done 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcde71cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcde71cc640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013eba0 0x7fcde013ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcde71cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcde0140900 con 0x7fcde013eba0 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013eba0 0x7fcde013ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient: get_auth_request con 0x7fcde013eba0 auth_method 0 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fcdd4007620 con 0x7fcde006fe50 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 10 monclient(hunting): handle_auth_done global_id 44403 payload 824 2024-09-11T02:58:53.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 1 -- 172.21.15.98:0/779633999 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcde00a8020 con 0x7fcde013eba0 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcdcc03df40 0x7fcdcc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(362..362 src has 1..362) ==== 6202+0+0 (secure 0 0 0) 0x7fcdd404f930 con 0x7fcde006fe50 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 client.44394.objecter ms_dispatch 0x7fcde0000ba0 osd_map(362..362 src has 1..362) 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 3 client.44394.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 3 client.44394.objecter handle_osd_map decoding full epoch 362 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddffff640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013eba0 0x7fcde013ef80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcde00a9e00 tx=0x7fcdd00195d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 20 client.44394.objecter dump_active .. 0 homeless 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 client.44394.objecter ms_handle_connect 0x7fcde013eba0 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 client.44394.objecter resend_mon_ops 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdd00bc030 con 0x7fcde013eba0 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:53.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 monclient: dump: 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.831 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcde5769640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcdcc03df40 0x7fcdcc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcde5769640 10 monclient: get_auth_request con 0x7fcdcc03df40 auth_method 0 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcdd0004040 con 0x7fcde013eba0 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013eba0 msgr2=0x7fcde013ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013eba0 0x7fcde013ef80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcde00a9e00 tx=0x7fcdd00195d0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013eba0 msgr2=0x7fcde013ef80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013eba0 0x7fcde013ef80 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcde00a9e00 tx=0x7fcdd00195d0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcde5769640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcdcc03df40 0x7fcdcc040400 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8000c10 tx=0x7fcdd80078e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.806+0000 7fcddd7fa640 10 client.44394.objecter ms_handle_connect 0x7fcdcc03df40 2024-09-11T02:58:53.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcde71cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:53.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcde71cc640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013f4c0 0x7fcde013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcde71cc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcde006ca10 con 0x7fcde013f4c0 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcddffff640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013f4c0 0x7fcde013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcddffff640 10 monclient: get_auth_request con 0x7fcde013f4c0 auth_method 0 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcddffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.810+0000 7fcddffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddffff640 10 monclient(hunting): handle_auth_done global_id 44412 payload 824 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddffff640 1 -- 172.21.15.98:0/779633999 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcde00a8350 con 0x7fcde013f4c0 2024-09-11T02:58:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 10 client.44394.objecter ms_handle_connect 0x7fcde013f4c0 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 10 client.44394.objecter resend_mon_ops 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddffff640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013f4c0 0x7fcde013a7d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00b0040 tx=0x7fcdd00a7000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcdd00bc070 con 0x7fcde013f4c0 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 10 monclient: dump: 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7fcdd0004850 con 0x7fcde013f4c0 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013f4c0 msgr2=0x7fcde013a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013f4c0 0x7fcde013a7d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00b0040 tx=0x7fcdd00a7000 comp rx=0 tx=0).stop 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013f4c0 msgr2=0x7fcde013a7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcddd7fa640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde013f4c0 0x7fcde013a7d0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00b0040 tx=0x7fcdd00a7000 comp rx=0 tx=0).stop 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 160242, 2024-09-11T02:58:53.838 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.839 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.840 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.841 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:53.842 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:53.843 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:53.844 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.847 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:53.850 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.854 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:53.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.857 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:53.858 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:53.859 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.860 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:53.861 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 10 librados: watch_flush enter 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 10 librados: watch_flush exit 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 20 client.44394.objecter shutdown clearing up homeless session... 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 10 client.44394.objecter successfully canceled tick 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcdcc03df40 msgr2=0x7fcdcc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcdcc03df40 0x7fcdcc040400 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8000c10 tx=0x7fcdd80078e0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 10 monclient: shutdown 2024-09-11T02:58:53.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde006fe50 msgr2=0x7fcde0138120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde006fe50 0x7fcde0138120 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400c7f0 tx=0x7fcdd400ccb0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde5769640 1 -- 172.21.15.98:0/779633999 reap_dead start 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 shutdown_connections 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 --2- 172.21.15.98:0/779633999 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcde006fe50 0x7fcde0138120 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 >> 172.21.15.98:0/779633999 conn(0x7fcde00ac730 msgr2=0x7fcde00aa480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 shutdown_connections 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 -- 172.21.15.98:0/779633999 wait complete. 2024-09-11T02:58:53.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.814+0000 7fcde71cc640 1 librados: shutdown 2024-09-11T02:58:53.864 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:53.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T02:58:53.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:58:53.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.978+0000 7efc5bca7640 10 monclient: get_monmap_and_config 2024-09-11T02:58:53.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.978+0000 7efc5bca7640 10 monclient: build_initial_monmap 2024-09-11T02:58:53.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.978+0000 7efc5bca7640 10 monclient: monmap: 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:53.984995+0000 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:53.984995+0000 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:53.984 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:53.985 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:53.985 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 1 Processor -- start 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 1 -- start start 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient: init 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient: _add_conns ranks=[1,10,7,0,6,9,5,4,2,8,3] 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7efc54139b80 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efc5406fce0 0x7efc5413ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient(hunting): picked mon.noname-f con 0x7efc5406fce0 addr v2:172.21.15.190:3300/0 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc540724b0 0x7efc54140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient(hunting): picked mon.noname-i con 0x7efc540724b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc540a94a0 con 0x7efc54139b80 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7efc54071850 con 0x7efc5406fce0 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7efc54071aa0 con 0x7efc540724b0 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5bca7640 10 monclient(hunting): authenticate will time out at 1487.865263s 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc540724b0 0x7efc54140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7efc5406fce0 0x7efc5413ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59a43640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7efc54139b80 0x7efc54139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:45786/0 (socket says 172.21.15.98:45786) 2024-09-11T02:58:53.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59a43640 1 -- 172.21.15.98:0/126537205 learned_addr learned my addr 172.21.15.98:0/126537205 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): get_auth_request con 0x7efc5406fce0 auth_method 0 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 10 monclient(hunting): get_auth_request con 0x7efc540724b0 auth_method 0 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:53.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient(hunting): handle_auth_done global_id 25372 payload 274 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 1 -- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc540724b0 msgr2=0x7efc54140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 1 --2- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc540724b0 0x7efc54140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 1 -- 172.21.15.98:0/126537205 >> v1:172.21.15.190:6789/0 conn(0x7efc54139b80 legacy=0x7efc54139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient: _finish_hunting 0 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 1 monclient: found mon.noname-f 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc59242640 1 -- 172.21.15.98:0/126537205 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc541424a0 con 0x7efc5406fce0 2024-09-11T02:58:53.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.982+0000 7efc5a244640 1 --2- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc540724b0 0x7efc54140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 1 --2- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc5406fce0 0x7efc5413ea00 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7efc500097f0 tx=0x7efc5001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3bbdeebf763e95c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 1 -- 172.21.15.98:0/126537205 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc500b0070 con 0x7efc5406fce0 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: dump: 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: _finish_auth 0 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: _check_auth_tickets 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 5 monclient: authenticate success, global_id 25372 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 1 -- 172.21.15.98:0/126537205 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc500aa810 con 0x7efc5406fce0 2024-09-11T02:58:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 4 set_mon_vals no callback set 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 1 -- 172.21.15.98:0/126537205 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc500a9300 con 0x7efc5406fce0 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc58a41640 10 monclient: dump: 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.000 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient: shutdown 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc5406fce0 msgr2=0x7efc5413ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 --2- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc5406fce0 0x7efc5413ea00 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7efc500097f0 tx=0x7efc5001c360 comp rx=0 tx=0).stop 2024-09-11T02:58:54.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- 172.21.15.98:0/126537205 shutdown_connections 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 --2- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc540724b0 0x7efc54140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 --2- 172.21.15.98:0/126537205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efc5406fce0 0x7efc5413ea00 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- 172.21.15.98:0/126537205 >> 172.21.15.98:0/126537205 conn(0x7efc540ac6b0 msgr2=0x7efc540a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- 172.21.15.98:0/126537205 shutdown_connections 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- 172.21.15.98:0/126537205 wait complete. 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient: monmap: 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.002 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 librados: starting msgr at 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 librados: starting objecter 2024-09-11T02:58:54.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 Processor -- start 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- start start 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 librados: setting wanted keys 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 librados: calling monclient init 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient: init 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient: _add_conns ranks=[3,2,4,6,7,1,5,8,0] 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc5406fce0 0x7efc5413d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): picked mon.noname-g con 0x7efc5406fce0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc540724b0 0x7efc5413def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): picked mon.noname-b con 0x7efc540724b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc54139b80 0x7efc5413e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): picked mon.noname-c con 0x7efc54139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:54.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efc54142030 con 0x7efc540724b0 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efc54141eb0 con 0x7efc5406fce0 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7efc541421b0 con 0x7efc54139b80 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5bca7640 10 monclient(hunting): authenticate will time out at 1487.871295s 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc54139b80 0x7efc5413e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc54139b80 0x7efc5413e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59828/0 (socket says 172.21.15.98:59828) 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 -- 172.21.15.98:0/4241730015 learned_addr learned my addr 172.21.15.98:0/4241730015 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): get_auth_request con 0x7efc54139b80 auth_method 0 2024-09-11T02:58:54.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc540724b0 0x7efc5413def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 10 monclient(hunting): get_auth_request con 0x7efc540724b0 auth_method 0 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59a43640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc5406fce0 0x7efc5413d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59a43640 10 monclient(hunting): get_auth_request con 0x7efc5406fce0 auth_method 0 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59a43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59a43640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient(hunting): handle_auth_done global_id 24052 payload 824 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc5406fce0 msgr2=0x7efc5413d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc5406fce0 0x7efc5413d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc540724b0 msgr2=0x7efc5413def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc540724b0 0x7efc5413def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 monclient: found mon.noname-c 2024-09-11T02:58:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc5a244640 1 -- 172.21.15.98:0/4241730015 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc54122c70 con 0x7efc54139b80 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.986+0000 7efc59242640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc540724b0 0x7efc5413def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5a244640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc54139b80 0x7efc5413e430 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00efc0 tx=0x7efc4c009f90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 client.?.objecter ms_handle_connect 0x7efc54139b80 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc4c018030 con 0x7efc54139b80 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: dump: 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc4c00cde0 con 0x7efc54139b80 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 5 monclient: authenticate success, global_id 24052 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 10 monclient: _renew_subs 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc54139620 con 0x7efc54139b80 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc4c014d60 con 0x7efc54139b80 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 10 client.24052.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4b7fe640 4 set_mon_vals no callback set 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: dump: 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 10 monclient: _renew_subs 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7efc54140f60 con 0x7efc54139b80 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 1 librados: init done 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc54139f60 0x7efc5413a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7efc540737c0 con 0x7efc54139f60 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc54139f60 0x7efc5413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient: get_auth_request con 0x7efc54139f60 auth_method 0 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7efc4c014480 con 0x7efc54139b80 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 10 monclient(hunting): handle_auth_done global_id 24578 payload 824 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 1 -- 172.21.15.98:0/4241730015 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc540aa240 con 0x7efc54139f60 2024-09-11T02:58:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59a43640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc54139f60 0x7efc5413a340 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f70 tx=0x7efc4400e8b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3803df70 0x7efc38040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 client.24052.objecter ms_handle_connect 0x7efc54139f60 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 client.24052.objecter resend_mon_ops 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(362..362 src has 1..362) ==== 6202+0+0 (secure 0 0 0) 0x7efc4c019070 con 0x7efc54139b80 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 client.24052.objecter ms_dispatch 0x7efc54000ba0 osd_map(362..362 src has 1..362) 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 3 client.24052.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 3 client.24052.objecter handle_osd_map decoding full epoch 362 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 20 client.24052.objecter dump_active .. 0 homeless 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc44019070 con 0x7efc54139f60 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: dump: 2024-09-11T02:58:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59242640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3803df70 0x7efc38040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59242640 10 monclient: get_auth_request con 0x7efc3803df70 auth_method 0 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efc44005a70 con 0x7efc54139f60 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc54139f60 msgr2=0x7efc5413a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc54139f60 0x7efc5413a340 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f70 tx=0x7efc4400e8b0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc54139f60 msgr2=0x7efc5413a340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc54139f60 0x7efc5413a340 secure :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f70 tx=0x7efc4400e8b0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc4a7fc640 10 client.24052.objecter ms_handle_connect 0x7efc3803df70 2024-09-11T02:58:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.990+0000 7efc59242640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3803df70 0x7efc38040430 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7efc500ad000 tx=0x7efc500a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc5413a880 0x7efc5413ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7efc5406ca10 con 0x7efc5413a880 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc5413a880 0x7efc5413ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient: get_auth_request con 0x7efc5413a880 auth_method 0 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 10 monclient(hunting): handle_auth_done global_id 24587 payload 824 2024-09-11T02:58:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 1 -- 172.21.15.98:0/4241730015 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efc54073d90 con 0x7efc5413a880 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 10 client.24052.objecter ms_handle_connect 0x7efc5413a880 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 10 client.24052.objecter resend_mon_ops 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc59a43640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc5413a880 0x7efc5413ac90 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f70 tx=0x7efc44000ee0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc44019030 con 0x7efc5413a880 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 10 monclient: dump: 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9269 (secure 0 0 0) 0x7efc44007610 con 0x7efc5413a880 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc5413a880 msgr2=0x7efc5413ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc5413a880 0x7efc5413ac90 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f70 tx=0x7efc44000ee0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc5413a880 msgr2=0x7efc5413ac90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc4a7fc640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7efc5413a880 0x7efc5413ac90 secure :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7efc44000f70 tx=0x7efc44000ee0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 549790, 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:54.023 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.024 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:54.025 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:54.026 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:54.027 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.028 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:54.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.030 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.035 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.039 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.040 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.042 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.044 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.046 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:54.046 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.046 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.046 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:54.046 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 10 librados: watch_flush enter 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 10 librados: watch_flush exit 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 20 client.24052.objecter shutdown clearing up homeless session... 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 10 client.24052.objecter successfully canceled tick 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3803df70 msgr2=0x7efc38040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3803df70 0x7efc38040430 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7efc500ad000 tx=0x7efc500a6000 comp rx=0 tx=0).stop 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 10 monclient: shutdown 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc54139b80 msgr2=0x7efc5413e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 --2- 172.21.15.98:0/4241730015 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc54139b80 0x7efc5413e430 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00efc0 tx=0x7efc4c009f90 comp rx=0 tx=0).stop 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5a244640 1 -- 172.21.15.98:0/4241730015 reap_dead start 2024-09-11T02:58:54.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5a244640 1 -- 172.21.15.98:0/4241730015 reap_dead start 2024-09-11T02:58:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 shutdown_connections 2024-09-11T02:58:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 >> 172.21.15.98:0/4241730015 conn(0x7efc540ac6b0 msgr2=0x7efc540a7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 shutdown_connections 2024-09-11T02:58:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 -- 172.21.15.98:0/4241730015 wait complete. 2024-09-11T02:58:54.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:53.998+0000 7efc5bca7640 1 librados: shutdown 2024-09-11T02:58:54.049 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:54.049 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T02:58:54.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:58:54.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient: get_monmap_and_config 2024-09-11T02:58:54.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient: monmap: 2024-09-11T02:58:54.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:54.173361+0000 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:54.173361+0000 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.172 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:54.173 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:54.173 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 1 Processor -- start 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 1 -- start start 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient: init 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient: _add_conns ranks=[3,2,1,9,0,4,8,5,7,10,6] 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 0x7fd89809bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient(hunting): picked mon.noname-g con 0x7fd89809b8c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd89809df80 0x7fd8980a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient(hunting): picked mon.noname-b con 0x7fd89809df80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd8980a4320 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89809d2c0 con 0x7fd8980a4320 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd89809d5a0 con 0x7fd89809df80 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd89809d880 con 0x7fd89809b8c0 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd8a02fc640 10 monclient(hunting): authenticate will time out at 1488.053624s 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd89809df80 0x7fd8980a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd89809df80 0x7fd8980a3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:45404/0 (socket says 172.21.15.98:45404) 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 -- 172.21.15.98:0/1268355048 learned_addr learned my addr 172.21.15.98:0/1268355048 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): get_auth_request con 0x7fd89809df80 auth_method 0 2024-09-11T02:58:54.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89e098640 1 --2- 172.21.15.98:0/1268355048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 0x7fd89809bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89e098640 10 monclient(hunting): get_auth_request con 0x7fd89809b8c0 auth_method 0 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89e098640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89e098640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89e098640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient(hunting): handle_auth_done global_id 64226 payload 274 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 -- 172.21.15.98:0/1268355048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 msgr2=0x7fd89809bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 --2- 172.21.15.98:0/1268355048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 0x7fd89809bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 -- 172.21.15.98:0/1268355048 >> v1:172.21.15.190:6789/0 conn(0x7fd8980a4320 legacy=0x7fd898140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 monclient: found mon.noname-b 2024-09-11T02:58:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:58:54.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.170+0000 7fd89d897640 1 -- 172.21.15.98:0/1268355048 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8980a5d10 con 0x7fd89809df80 2024-09-11T02:58:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 1 --2- 172.21.15.98:0/1268355048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd89809df80 0x7fd8980a3de0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0098a0 tx=0x7fd88c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83b44c6ed53cf614 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 1 -- 172.21.15.98:0/1268355048 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd88c0b0070 con 0x7fd89809df80 2024-09-11T02:58:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: dump: 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.186 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 1 -- 172.21.15.98:0/1268355048 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd88c0aa810 con 0x7fd89809df80 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 5 monclient: authenticate success, global_id 64226 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:54.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 1 -- 172.21.15.98:0/1268355048 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd88c0a9300 con 0x7fd89809df80 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 4 set_mon_vals no callback set 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d096640 10 monclient: dump: 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.188 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient: shutdown 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- 172.21.15.98:0/1268355048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd89809df80 msgr2=0x7fd8980a3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 --2- 172.21.15.98:0/1268355048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd89809df80 0x7fd8980a3de0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0098a0 tx=0x7fd88c01c3e0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- 172.21.15.98:0/1268355048 shutdown_connections 2024-09-11T02:58:54.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 --2- 172.21.15.98:0/1268355048 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd89809df80 0x7fd8980a3de0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 --2- 172.21.15.98:0/1268355048 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 0x7fd89809bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- 172.21.15.98:0/1268355048 >> 172.21.15.98:0/1268355048 conn(0x7fd8980a4ed0 msgr2=0x7fd8980a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- 172.21.15.98:0/1268355048 shutdown_connections 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- 172.21.15.98:0/1268355048 wait complete. 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient: monmap: 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:54.190 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 librados: starting msgr at 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 librados: starting objecter 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 Processor -- start 2024-09-11T02:58:54.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- start start 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 librados: setting wanted keys 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 librados: calling monclient init 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient: init 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient: _add_conns ranks=[3,1,7,4,5,6,8,0,2] 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 0x7fd898071a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): picked mon.noname-g con 0x7fd89809b8c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89809df80 0x7fd898071f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): picked mon.noname-f con 0x7fd89809df80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd8980a4320 0x7fd8980724d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): picked mon.noname-i con 0x7fd8980a4320 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:54.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd89809e4e0 con 0x7fd89809df80 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd89809e360 con 0x7fd89809b8c0 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fd8981444d0 con 0x7fd8980a4320 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd8a02fc640 10 monclient(hunting): authenticate will time out at 1488.059535s 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89809df80 0x7fd898071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89e098640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 0x7fd898071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89809df80 0x7fd898071f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:39318/0 (socket says 172.21.15.98:39318) 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 1 -- 172.21.15.98:0/3081469113 learned_addr learned my addr 172.21.15.98:0/3081469113 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 10 monclient(hunting): get_auth_request con 0x7fd89809df80 auth_method 0 2024-09-11T02:58:54.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89d897640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89e899640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd8980a4320 0x7fd8980724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89e098640 10 monclient(hunting): get_auth_request con 0x7fd89809b8c0 auth_method 0 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89e098640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89e098640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.174+0000 7fd89e098640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient(hunting): get_auth_request con 0x7fd8980a4320 auth_method 0 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 10 monclient(hunting): handle_auth_done global_id 25390 payload 824 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd8980a4320 msgr2=0x7fd8980724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd8980a4320 0x7fd8980724d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 msgr2=0x7fd898071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd89809b8c0 0x7fd898071a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 1 monclient: found mon.noname-f 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 1 -- 172.21.15.98:0/3081469113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd898092820 con 0x7fd89809df80 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89d897640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89809df80 0x7fd898071f90 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0ae040 tx=0x7fd88c0191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 client.?.objecter ms_handle_connect 0x7fd89809df80 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd88c0b9070 con 0x7fd89809df80 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:58:54.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: dump: 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.197 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 5 monclient: authenticate success, global_id 25390 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd88c0044b0 con 0x7fd89809df80 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd88c0b0040 con 0x7fd89809df80 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89c895640 4 set_mon_vals no callback set 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:58:54.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 monclient: dump: 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.199 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 10 monclient: _renew_subs 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd898092b00 con 0x7fd89809df80 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 10 client.25390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 10 monclient: _renew_subs 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8980732d0 con 0x7fd89809df80 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 1 librados: init done 2024-09-11T02:58:54.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1550 0x7fd8981d1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd89806ca10 con 0x7fd8981d1550 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1550 0x7fd8981d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient: get_auth_request con 0x7fd8981d1550 auth_method 0 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd88c004040 con 0x7fd89809df80 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd88403df40 0x7fd884040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(362..362 src has 1..362) ==== 6202+0+0 (secure 0 0 0) 0x7fd88c0e8cc0 con 0x7fd89809df80 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 10 client.25390.objecter ms_dispatch 0x7fd898000ba0 osd_map(362..362 src has 1..362) 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 3 client.25390.objecter handle_osd_map got epochs [362,362] > 0 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 3 client.25390.objecter handle_osd_map decoding full epoch 362 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd896ffd640 20 client.25390.objecter dump_active .. 0 homeless 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 10 monclient(hunting): handle_auth_done global_id 25399 payload 824 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e098640 1 -- 172.21.15.98:0/3081469113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd89809bca0 con 0x7fd8981d1550 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd88403df40 0x7fd884040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.178+0000 7fd89e899640 10 monclient: get_auth_request con 0x7fd88403df40 auth_method 0 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 client.25390.objecter ms_handle_connect 0x7fd8981d1550 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 client.25390.objecter resend_mon_ops 2024-09-11T02:58:54.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd89e098640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1550 0x7fd8981d1930 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fd890004680 tx=0x7fd890007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd890015890 con 0x7fd8981d1550 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 monclient: dump: 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.203 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 client.25390.objecter ms_handle_connect 0x7fd88403df40 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd89e899640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd88403df40 0x7fd884040400 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7fd888007fd0 tx=0x7fd888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd89001a070 con 0x7fd8981d1550 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:54.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1550 msgr2=0x7fd8981d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1550 0x7fd8981d1930 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fd890004680 tx=0x7fd890007700 comp rx=0 tx=0).stop 2024-09-11T02:58:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1550 msgr2=0x7fd8981d1930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.182+0000 7fd896ffd640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1550 0x7fd8981d1930 secure :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fd890004680 tx=0x7fd890007700 comp rx=0 tx=0).stop 2024-09-11T02:58:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd8a02fc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd8a02fc640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1e70 0x7fd89813ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd8a02fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd89806e870 con 0x7fd8981d1e70 2024-09-11T02:58:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1e70 0x7fd89813ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 10 monclient: get_auth_request con 0x7fd8981d1e70 auth_method 0 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.186+0000 7fd89e098640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd89e098640 10 monclient(hunting): handle_auth_done global_id 25408 payload 824 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd89e098640 1 -- 172.21.15.98:0/3081469113 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd898092f90 con 0x7fd8981d1e70 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd89e098640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1e70 0x7fd89813ead0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fd890004680 tx=0x7fd890004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 10 client.25390.objecter ms_handle_connect 0x7fd8981d1e70 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 10 client.25390.objecter resend_mon_ops 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd890015c40 con 0x7fd8981d1e70 2024-09-11T02:58:54.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 10 monclient: dump: 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.208 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7fd8900157d0 con 0x7fd8981d1e70 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1e70 msgr2=0x7fd89813ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1e70 0x7fd89813ead0 secure :-1 s=READY pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fd890004680 tx=0x7fd890004a50 comp rx=0 tx=0).stop 2024-09-11T02:58:54.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1e70 msgr2=0x7fd89813ead0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd896ffd640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8981d1e70 0x7fd89813ead0 secure :-1 s=CLOSED pgs=2281 cs=0 l=1 rev1=1 crypto rx=0x7fd890004680 tx=0x7fd890004a50 comp rx=0 tx=0).stop 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 549960, 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:54.210 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:54.211 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:54.214 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.215 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:54.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:54.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.220 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.222 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:54.224 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.227 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:54.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.233 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:54.234 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 10 librados: watch_flush enter 2024-09-11T02:58:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 10 librados: watch_flush exit 2024-09-11T02:58:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 20 client.25390.objecter shutdown clearing up homeless session... 2024-09-11T02:58:54.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 10 client.25390.objecter successfully canceled tick 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd88403df40 msgr2=0x7fd884040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd88403df40 0x7fd884040400 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7fd888007fd0 tx=0x7fd888009040 comp rx=0 tx=0).stop 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 10 monclient: shutdown 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89809df80 msgr2=0x7fd898071f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 --2- 172.21.15.98:0/3081469113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd89809df80 0x7fd898071f90 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0ae040 tx=0x7fd88c0191e0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd89e899640 1 -- 172.21.15.98:0/3081469113 reap_dead start 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd89e899640 1 -- 172.21.15.98:0/3081469113 reap_dead start 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 shutdown_connections 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 >> 172.21.15.98:0/3081469113 conn(0x7fd8980a4ed0 msgr2=0x7fd8980a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 shutdown_connections 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 -- 172.21.15.98:0/3081469113 wait complete. 2024-09-11T02:58:54.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.190+0000 7fd8a02fc640 1 librados: shutdown 2024-09-11T02:58:54.238 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:54.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T02:58:54.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T02:58:54.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient: get_monmap_and_config 2024-09-11T02:58:54.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient: monmap: 2024-09-11T02:58:54.373 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.373 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:54.375141+0000 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:54.375141+0000 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.374 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.375 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:54.375 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:54.375 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 1 Processor -- start 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 1 -- start start 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient: init 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient: _add_conns ranks=[4,0,1,6,2,3,10,9,7,8,5] 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 0x7f729809d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient(hunting): picked mon.noname-c con 0x7f72980a46a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:54.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72980a4a80 addr v1:172.21.15.98:6789/0 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f729813c8b0 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72980779a0 con 0x7f72980a4a80 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7298070130 con 0x7f729813c8b0 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7298076fc0 con 0x7f72980a46a0 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f729cb0f640 10 monclient(hunting): authenticate will time out at 1488.255497s 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f7295d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f72980a4a80 0x7f72980a4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:50044/0 (socket says 172.21.15.98:50044) 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.370+0000 7f7295d9b640 1 -- 172.21.15.98:0/1280473092 learned_addr learned my addr 172.21.15.98:0/1280473092 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3700084016 0 0) 0x7f72980779a0 con 0x7f72980a4a80 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): _negotiate 2024-09-11T02:58:54.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): my global_id is 44541 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7284003530 con 0x7f72980a4a80 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2803806854 0 0) 0x7f7284003530 con 0x7f72980a4a80 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 msgr2=0x7f729809d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 0x7f729809d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 >> v1:172.21.15.190:6789/0 conn(0x7f729813c8b0 legacy=0x7f729813eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 monclient: found mon.noname-a-legacy 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72980a2d60 con 0x7f72980a4a80 2024-09-11T02:58:54.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f7288001cf0 con 0x7f72980a4a80 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729cb0f640 5 monclient: authenticate success, global_id 44541 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729cb0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: dump: 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.382 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T02:58:54.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 >> v1:172.21.15.98:6789/0 conn(0x7f72980a4a80 legacy=0x7f72980a4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: _add_conns ranks=[5,4,2,3,0,6,1,8,7] 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f72840041a0 0x7f7284004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): picked mon.h con 0x7f72840041a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 0x7f7284004ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): picked mon.c con 0x7f72980a46a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72840099c0 0x7f728400bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): picked mon.b con 0x7f72840099c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7298076fc0 con 0x7f72840099c0 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f728400c4d0 con 0x7f72980a46a0 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f728400c680 con 0x7f72840041a0 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient(hunting): _finish_auth 0 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729cb0f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 0x7f7284004ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): get_auth_request con 0x7f72980a46a0 auth_method 0 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729659c640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f72840041a0 0x7f7284004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729659c640 10 monclient(hunting): get_auth_request con 0x7f72840041a0 auth_method 0 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient(hunting): handle_auth_done global_id 44541 payload 293 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 -- 172.21.15.98:0/1280473092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f72840041a0 msgr2=0x7f7284004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f72840041a0 0x7f7284004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 -- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72840099c0 msgr2=0x7f728400bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f72840099c0 0x7f728400bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 monclient: found mon.c 2024-09-11T02:58:54.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 -- 172.21.15.98:0/1280473092 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72980a2d60 con 0x7f72980a46a0 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f7295d9b640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 0x7f7284004ed0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f728800a7b0 tx=0x7f728801d260 comp rx=0 tx=0).ready entity=mon.4 client_cookie=af9934a6a83d2cf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f728800a3d0 con 0x7f72980a46a0 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: dump: 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.387 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.388 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72880a9070 con 0x7f72980a46a0 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729cb0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729cb0f640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729cb0f640 4 set_mon_vals no callback set 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 1 -- 172.21.15.98:0/1280473092 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72880a5c30 con 0x7f72980a46a0 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.374+0000 7f729559a640 10 monclient: dump: 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.389 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.390 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.378+0000 7f729cb0f640 10 monclient: shutdown 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.378+0000 7f729cb0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.378+0000 7f729cb0f640 1 -- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 msgr2=0x7f7284004ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.378+0000 7f729cb0f640 1 --2- 172.21.15.98:0/1280473092 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f72980a46a0 0x7f7284004ed0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f728800a7b0 tx=0x7f728801d260 comp rx=0 tx=0).stop 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.382+0000 7f7296d9d640 1 -- 172.21.15.98:0/1280473092 reap_dead start 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.382+0000 7f729cb0f640 1 -- 172.21.15.98:0/1280473092 shutdown_connections 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.382+0000 7f729cb0f640 1 -- 172.21.15.98:0/1280473092 >> 172.21.15.98:0/1280473092 conn(0x7f729806ed00 msgr2=0x7f729806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 -- 172.21.15.98:0/1280473092 shutdown_connections 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 -- 172.21.15.98:0/1280473092 wait complete. 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient: monmap: 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.395 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.396 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 librados: starting msgr at 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 librados: starting objecter 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 Processor -- start 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 -- start start 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 librados: setting wanted keys 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 librados: calling monclient init 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient: init 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient: _add_conns ranks=[5,8,0,1,6,4,7,3,2] 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f72981377c0 0x7f7298137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): picked mon.noname-h con 0x7f72981377c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:54.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f729811f2e0 0x7f729811f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): picked mon.noname-e con 0x7f729811f2e0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f729811fc40 0x7f7298120080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): picked mon.noname-a con 0x7f729811fc40 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7298140de0 con 0x7f729811fc40 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f72980517e0 con 0x7f72981377c0 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7298140f60 con 0x7f729811f2e0 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f729cb0f640 10 monclient(hunting): authenticate will time out at 1488.270811s 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f729811f2e0 0x7f729811f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f729811f2e0 0x7f729811f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39778/0 (socket says 172.21.15.98:39778) 2024-09-11T02:58:54.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 -- 172.21.15.98:0/1474628072 learned_addr learned my addr 172.21.15.98:0/1474628072 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): get_auth_request con 0x7f729811f2e0 auth_method 0 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient(hunting): handle_auth_done global_id 24596 payload 824 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f72981377c0 msgr2=0x7f7298137ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f72981377c0 0x7f7298137ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f729811fc40 msgr2=0x7f7298120080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7296d9d640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f729811fc40 0x7f7298120080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f729811fc40 0x7f7298120080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 monclient: found mon.noname-e 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 -- 172.21.15.98:0/1474628072 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729813fe50 con 0x7f729811f2e0 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7296d9d640 10 monclient: get_auth_request con 0x7f729811fc40 auth_method 0 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7296d9d640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f729811fc40 0x7f7298120080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f7295d9b640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f729811f2e0 0x7f729811f700 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f7288000f90 tx=0x7f728801a000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f728effd640 10 client.?.objecter ms_handle_connect 0x7f729811f2e0 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.386+0000 7f728effd640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f728800a3d0 con 0x7f729811f2e0 2024-09-11T02:58:54.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: dump: 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.401 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 5 monclient: authenticate success, global_id 24596 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7288004d80 con 0x7f729811f2e0 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7288001cf0 con 0x7f729811f2e0 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 10 monclient: _renew_subs 2024-09-11T02:58:54.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 1 -- 172.21.15.98:0/1474628072 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7298138560 con 0x7f729811f2e0 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f7294d99640 4 set_mon_vals no callback set 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 10 client.24596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 10 monclient: _renew_subs 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 1 -- 172.21.15.98:0/1474628072 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f7298121960 con 0x7f729811f2e0 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f729cb0f640 1 librados: init done 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 monclient: dump: 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.403 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728cff9640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270002890 0x7f7270004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728cff9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7270007cc0 con 0x7f7270002890 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f72880ae8a0 con 0x7f729811f2e0 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f727803df70 0x7f7278040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(363..363 src has 1..363) ==== 6202+0+0 (secure 0 0 0) 0x7f72880e9b50 con 0x7f729811f2e0 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 10 client.24596.objecter ms_dispatch 0x7f7298000ba0 osd_map(363..363 src has 1..363) 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 3 client.24596.objecter handle_osd_map got epochs [363,363] > 0 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 3 client.24596.objecter handle_osd_map decoding full epoch 363 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f728effd640 20 client.24596.objecter dump_active .. 0 homeless 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f7296d9d640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f727803df70 0x7f7278040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.390+0000 7f7296d9d640 10 monclient: get_auth_request con 0x7f727803df70 auth_method 0 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 client.24596.objecter ms_handle_connect 0x7f727803df70 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f7296d9d640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f727803df70 0x7f7278040430 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7f72840039c0 tx=0x7f7290008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270002890 0x7f7270004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient: get_auth_request con 0x7f7270002890 auth_method 0 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 10 monclient(hunting): handle_auth_done global_id 64245 payload 824 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 1 -- 172.21.15.98:0/1474628072 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7270007ec0 con 0x7f7270002890 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 client.24596.objecter ms_handle_connect 0x7f7270002890 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 client.24596.objecter resend_mon_ops 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f729659c640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270002890 0x7f7270004d30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f728000d400 tx=0x7f728000d8c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7280012a60 con 0x7f7270002890 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 monclient: dump: 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.410 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.411 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7280019070 con 0x7f7270002890 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270002890 msgr2=0x7f7270004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270002890 0x7f7270004d30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f728000d400 tx=0x7f728000d8c0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270002890 msgr2=0x7f7270004d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.394+0000 7f728effd640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270002890 0x7f7270004d30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270006ce0 0x7f7270007170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f727000a200 con 0x7f7270006ce0 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270006ce0 0x7f7270007170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient: get_auth_request con 0x7f7270006ce0 auth_method 0 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 10 monclient(hunting): handle_auth_done global_id 64254 payload 824 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 1 -- 172.21.15.98:0/1474628072 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f72700076b0 con 0x7f7270006ce0 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f729659c640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270006ce0 0x7f7270007170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7280018000 tx=0x7f7280004850 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 10 client.24596.objecter ms_handle_connect 0x7f7270006ce0 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 10 client.24596.objecter resend_mon_ops 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f728001ec30 con 0x7f7270006ce0 2024-09-11T02:58:54.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 10 monclient: dump: 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f728001e7c0 con 0x7f7270006ce0 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270006ce0 msgr2=0x7f7270007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270006ce0 0x7f7270007170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7280018000 tx=0x7f7280004850 comp rx=0 tx=0).stop 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270006ce0 msgr2=0x7f7270007170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728effd640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7270006ce0 0x7f7270007170 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7280018000 tx=0x7f7280004850 comp rx=0 tx=0).stop 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 26591, 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:54.416 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.417 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.418 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:54.420 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.421 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.423 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:54.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.425 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.427 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.428 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.429 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.431 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.436 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.439 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.440 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:54.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.440 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:54.440 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 10 librados: watch_flush enter 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 10 librados: watch_flush exit 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 20 client.24596.objecter shutdown clearing up homeless session... 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 10 client.24596.objecter successfully canceled tick 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f727803df70 msgr2=0x7f7278040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f727803df70 0x7f7278040430 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7f72840039c0 tx=0x7f7290008040 comp rx=0 tx=0).stop 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 10 monclient: shutdown 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f729811f2e0 msgr2=0x7f729811f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f729811f2e0 0x7f729811f700 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f7288000f90 tx=0x7f728801a000 comp rx=0 tx=0).stop 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f7296d9d640 1 -- 172.21.15.98:0/1474628072 reap_dead start 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 shutdown_connections 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 --2- 172.21.15.98:0/1474628072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f729811f2e0 0x7f729811f700 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 >> 172.21.15.98:0/1474628072 conn(0x7f729806ed00 msgr2=0x7f72980726d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.402+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 shutdown_connections 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.406+0000 7f728cff9640 1 -- 172.21.15.98:0/1474628072 wait complete. 2024-09-11T02:58:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.406+0000 7f728cff9640 1 librados: shutdown 2024-09-11T02:58:54.443 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:54.443 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T02:58:54.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient: get_monmap_and_config 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient: monmap: 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:54.562075+0000 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:54.562075+0000 2024-09-11T02:58:54.560 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.561 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:54.562 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:54.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 Processor -- start 2024-09-11T02:58:54.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 -- start start 2024-09-11T02:58:54.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient: init 2024-09-11T02:58:54.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient: _add_conns ranks=[7,3,6,4,5,8,1,9,10,2,0] 2024-09-11T02:58:54.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139b10 0x7f8210072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): picked mon.noname-i con 0x7f8210139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139ef0 0x7f82100a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): picked mon.noname-g con 0x7f8210139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f821013a2d0 0x7f82100a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): picked mon.noname-d con 0x7f821013a2d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f82100702b0 con 0x7f8210139ef0 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8210071bf0 con 0x7f821013a2d0 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f8210071350 con 0x7f8210139b10 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8216e3f640 10 monclient(hunting): authenticate will time out at 1488.442434s 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f821013a2d0 0x7f82100a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f821013a2d0 0x7f82100a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44614/0 (socket says 172.21.15.98:44614) 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 -- 172.21.15.98:0/1049415023 learned_addr learned my addr 172.21.15.98:0/1049415023 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): get_auth_request con 0x7f821013a2d0 auth_method 0 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139ef0 0x7f82100a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8214bdb640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139b10 0x7f8210072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 10 monclient(hunting): get_auth_request con 0x7f8210139ef0 auth_method 0 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8214bdb640 10 monclient(hunting): get_auth_request con 0x7f8210139b10 auth_method 0 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8214bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8214bdb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8214bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f820ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient(hunting): handle_auth_done global_id 44421 payload 274 2024-09-11T02:58:54.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 -- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139b10 msgr2=0x7f8210072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139b10 0x7f8210072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 -- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139ef0 msgr2=0x7f82100a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139ef0 0x7f82100a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 monclient: found mon.noname-d 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f82153dc640 1 -- 172.21.15.98:0/1049415023 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82100a9620 con 0x7f821013a2d0 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.558+0000 7f8214bdb640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139b10 0x7f8210072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f82153dc640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f821013a2d0 0x7f82100a7f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f82080098a0 tx=0x7f820801c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=72837d2d6312355c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 1 -- 172.21.15.98:0/1049415023 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82080b0070 con 0x7f821013a2d0 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:54.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: dump: 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.571 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 1 -- 172.21.15.98:0/1049415023 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82080aa810 con 0x7f821013a2d0 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 5 monclient: authenticate success, global_id 44421 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 1 -- 172.21.15.98:0/1049415023 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82080a9300 con 0x7f821013a2d0 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 4 set_mon_vals no callback set 2024-09-11T02:58:54.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f820f7fe640 10 monclient: dump: 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.573 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 10 monclient: shutdown 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 -- 172.21.15.98:0/1049415023 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f821013a2d0 msgr2=0x7f82100a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f821013a2d0 0x7f82100a7f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f82080098a0 tx=0x7f820801c440 comp rx=0 tx=0).stop 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 -- 172.21.15.98:0/1049415023 shutdown_connections 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f821013a2d0 0x7f82100a7f50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139ef0 0x7f82100a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1049415023 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139b10 0x7f8210072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 -- 172.21.15.98:0/1049415023 >> 172.21.15.98:0/1049415023 conn(0x7f82100ac790 msgr2=0x7f82100a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 -- 172.21.15.98:0/1049415023 shutdown_connections 2024-09-11T02:58:54.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 1 -- 172.21.15.98:0/1049415023 wait complete. 2024-09-11T02:58:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.562+0000 7f8216e3f640 10 monclient: monmap: 2024-09-11T02:58:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:54.583 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.584 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 librados: starting msgr at 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 librados: starting objecter 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 Processor -- start 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 -- start start 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 librados: setting wanted keys 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 librados: calling monclient init 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient: init 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient: _add_conns ranks=[3,7,1,6,4,0,5,8,2] 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139b10 0x7f8210134a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): picked mon.noname-g con 0x7f8210139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139ef0 0x7f8210134f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): picked mon.noname-i con 0x7f8210139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f821013a2d0 0x7f821013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): picked mon.noname-f con 0x7f821013a2d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8210143460 con 0x7f821013a2d0 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f82100a88a0 con 0x7f8210139b10 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f82101435e0 con 0x7f8210139ef0 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): authenticate will time out at 1488.448938s 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f821013a2d0 0x7f821013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139b10 0x7f8210134a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139ef0 0x7f8210134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f821013a2d0 0x7f821013a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:39352/0 (socket says 172.21.15.98:39352) 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139ef0 0x7f8210134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:51616/0 (socket says 172.21.15.98:51616) 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 -- 172.21.15.98:0/1749378595 learned_addr learned my addr 172.21.15.98:0/1749378595 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): get_auth_request con 0x7f821013a2d0 auth_method 0 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 10 monclient(hunting): get_auth_request con 0x7f8210139ef0 auth_method 0 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 10 monclient(hunting): get_auth_request con 0x7f8210139b10 auth_method 0 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient(hunting): handle_auth_done global_id 25417 payload 824 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139ef0 msgr2=0x7f8210134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139ef0 0x7f8210134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139b10 msgr2=0x7f8210134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139b10 0x7f8210134a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 monclient: found mon.noname-f 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 -- 172.21.15.98:0/1749378595 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82100aa3a0 con 0x7f821013a2d0 2024-09-11T02:58:54.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820ffff640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8210139ef0 0x7f8210134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8214bdb640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8210139b10 0x7f8210134a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f82153dc640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f821013a2d0 0x7f821013a6b0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f820801c270 tx=0x7f82080191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 client.?.objecter ms_handle_connect 0x7f821013a2d0 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82080b8020 con 0x7f821013a2d0 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: dump: 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.590 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 5 monclient: authenticate success, global_id 25417 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient: _renew_subs 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82100a9f00 con 0x7f821013a2d0 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82080aacf0 con 0x7f821013a2d0 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82080b0040 con 0x7f821013a2d0 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 client.25417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820effd640 4 set_mon_vals no callback set 2024-09-11T02:58:54.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f820d7fa640 10 monclient: dump: 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.593 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient: _renew_subs 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821013d7f0 con 0x7f821013a2d0 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 librados: init done 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821013d9c0 0x7f821013b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.566+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f821006ca10 con 0x7f821013d9c0 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821013d9c0 0x7f821013b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient: get_auth_request con 0x7f821013d9c0 auth_method 0 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f82080b3030 con 0x7f821013a2d0 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f81f003df40 0x7f81f0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(363..363 src has 1..363) ==== 6202+0+0 (secure 0 0 0) 0x7f82080e9270 con 0x7f821013a2d0 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 client.25417.objecter ms_dispatch 0x7f8210000ba0 osd_map(363..363 src has 1..363) 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 3 client.25417.objecter handle_osd_map got epochs [363,363] > 0 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 3 client.25417.objecter handle_osd_map decoding full epoch 363 2024-09-11T02:58:54.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 10 monclient(hunting): handle_auth_done global_id 20102 payload 824 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 1 -- 172.21.15.98:0/1749378595 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f821006e680 con 0x7f821013d9c0 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 20 client.25417.objecter dump_active .. 0 homeless 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820ffff640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f81f003df40 0x7f81f0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820ffff640 10 monclient: get_auth_request con 0x7f81f003df40 auth_method 0 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 client.25417.objecter ms_handle_connect 0x7f821013d9c0 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 client.25417.objecter resend_mon_ops 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f8214bdb640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821013d9c0 0x7f821013b3b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f8200011990 tx=0x7f8200011e50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8200019020 con 0x7f821013d9c0 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 monclient: dump: 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 client.25417.objecter ms_handle_connect 0x7f81f003df40 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820ffff640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f81f003df40 0x7f81f0040400 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7f82100a5db0 tx=0x7f81f800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8200004300 con 0x7f821013d9c0 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821013d9c0 msgr2=0x7f821013b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821013d9c0 0x7f821013b3b0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f8200011990 tx=0x7f8200011e50 comp rx=0 tx=0).stop 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821013d9c0 msgr2=0x7f821013b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.570+0000 7f820d7fa640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821013d9c0 0x7f821013b3b0 secure :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f8200011990 tx=0x7f8200011e50 comp rx=0 tx=0).stop 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821002d450 0x7f821002d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f821006e870 con 0x7f821002d450 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821002d450 0x7f821002d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient: get_auth_request con 0x7f821002d450 auth_method 0 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 10 monclient(hunting): handle_auth_done global_id 20111 payload 824 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 1 -- 172.21.15.98:0/1749378595 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8210072600 con 0x7f821002d450 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8214bdb640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821002d450 0x7f821002d860 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f820001a000 tx=0x7f82000046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 10 client.25417.objecter ms_handle_connect 0x7f821002d450 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 10 client.25417.objecter resend_mon_ops 2024-09-11T02:58:54.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8200019070 con 0x7f821002d450 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 10 monclient: dump: 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f820000fe90 con 0x7f821002d450 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821002d450 msgr2=0x7f821002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821002d450 0x7f821002d860 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f820001a000 tx=0x7f82000046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821002d450 msgr2=0x7f821002d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f820d7fa640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f821002d450 0x7f821002d860 secure :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f820001a000 tx=0x7f82000046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 550332, 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:54.602 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.603 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.604 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:54.605 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:54.606 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.607 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:54.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:54.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.614 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:54.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.619 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.621 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.623 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.624 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:58:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.625 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 10 librados: watch_flush enter 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 10 librados: watch_flush exit 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 20 client.25417.objecter shutdown clearing up homeless session... 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 10 client.25417.objecter successfully canceled tick 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f81f003df40 msgr2=0x7f81f0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f81f003df40 0x7f81f0040400 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7f82100a5db0 tx=0x7f81f800b040 comp rx=0 tx=0).stop 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 10 monclient: shutdown 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f821013a2d0 msgr2=0x7f821013a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 --2- 172.21.15.98:0/1749378595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f821013a2d0 0x7f821013a6b0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f820801c270 tx=0x7f82080191e0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f82153dc640 1 -- 172.21.15.98:0/1749378595 reap_dead start 2024-09-11T02:58:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f82153dc640 1 -- 172.21.15.98:0/1749378595 reap_dead start 2024-09-11T02:58:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 shutdown_connections 2024-09-11T02:58:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 >> 172.21.15.98:0/1749378595 conn(0x7f82100ac790 msgr2=0x7f8210138320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 shutdown_connections 2024-09-11T02:58:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 -- 172.21.15.98:0/1749378595 wait complete. 2024-09-11T02:58:54.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.578+0000 7f8216e3f640 1 librados: shutdown 2024-09-11T02:58:54.628 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:54.628 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T02:58:54.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:58:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient: get_monmap_and_config 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient: monmap: 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:54.749796+0000 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:54.749796+0000 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:54.749 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 Processor -- start 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 -- start start 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient: init 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient: _add_conns ranks=[5,6,4,10,1,9,8,3,2,7,0] 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f13c0139b80 0x7f13c0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): picked mon.noname-h con 0x7f13c0139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): picked mon.noname-d con 0x7f13c006fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c00a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): picked mon.noname-c con 0x7f13c0072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f13c00719c0 con 0x7f13c0072640 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f13c0071c10 con 0x7f13c0139b80 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f13c00702d0 con 0x7f13c006fe50 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13c56f0640 10 monclient(hunting): authenticate will time out at 1488.629895s 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c00a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c00a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44620/0 (socket says 172.21.15.98:44620) 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c00a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59848/0 (socket says 172.21.15.98:59848) 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 -- 172.21.15.98:0/1807023901 learned_addr learned my addr 172.21.15.98:0/1807023901 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): get_auth_request con 0x7f13c006fe50 auth_method 0 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 10 monclient(hunting): get_auth_request con 0x7f13c0072640 auth_method 0 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13beffd640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f13c0139b80 0x7f13c0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient(hunting): handle_auth_done global_id 44430 payload 274 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 -- 172.21.15.98:0/1807023901 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f13c0139b80 msgr2=0x7f13c0139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f13c0139b80 0x7f13c0139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 -- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 msgr2=0x7f13c00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c00a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 monclient: found mon.noname-d 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13be7fc640 1 -- 172.21.15.98:0/1807023901 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c0071370 con 0x7f13c006fe50 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13beffd640 10 monclient: get_auth_request con 0x7f13c0139b80 auth_method 0 2024-09-11T02:58:54.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13beffd640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f13c0139b80 0x7f13c0139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.746+0000 7f13bf7fe640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13be7fc640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c00a7a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f13b40098d0 tx=0x7f13b401c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ea94ecde0445d4ee server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 1 -- 172.21.15.98:0/1807023901 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13b40b0070 con 0x7f13c006fe50 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: dump: 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.757 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 1 -- 172.21.15.98:0/1807023901 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b40aa810 con 0x7f13c006fe50 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 1 -- 172.21.15.98:0/1807023901 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13b40a9300 con 0x7f13c006fe50 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 5 monclient: authenticate success, global_id 44430 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 4 set_mon_vals no callback set 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bdffb640 10 monclient: dump: 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.759 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient: shutdown 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 msgr2=0x7f13c00a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c00a7a10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f13b40098d0 tx=0x7f13b401c440 comp rx=0 tx=0).stop 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- 172.21.15.98:0/1807023901 shutdown_connections 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c00a7a10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 --2- 172.21.15.98:0/1807023901 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f13c0139b80 0x7f13c0139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- 172.21.15.98:0/1807023901 >> 172.21.15.98:0/1807023901 conn(0x7f13c00ac730 msgr2=0x7f13c00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- 172.21.15.98:0/1807023901 shutdown_connections 2024-09-11T02:58:54.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- 172.21.15.98:0/1807023901 wait complete. 2024-09-11T02:58:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient: monmap: 2024-09-11T02:58:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:54.766 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 librados: starting msgr at 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 librados: starting objecter 2024-09-11T02:58:54.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 Processor -- start 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- start start 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 librados: setting wanted keys 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 librados: calling monclient init 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient: init 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient: _add_conns ranks=[6,4,3,1,7,8,5,2,0] 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c013fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): picked mon.noname-d con 0x7f13c006fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c01d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): picked mon.noname-c con 0x7f13c0072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13c0139b80 0x7f13c01d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): picked mon.noname-g con 0x7f13c0139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f13c01435c0 con 0x7f13c0139b80 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f13c0143440 con 0x7f13c0072640 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f13c0143740 con 0x7f13c006fe50 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13c56f0640 10 monclient(hunting): authenticate will time out at 1488.635351s 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c013fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c013fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:44622/0 (socket says 172.21.15.98:44622) 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 -- 172.21.15.98:0/897691512 learned_addr learned my addr 172.21.15.98:0/897691512 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): get_auth_request con 0x7f13c006fe50 auth_method 0 2024-09-11T02:58:54.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bf7fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13c0139b80 0x7f13c01d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bf7fe640 10 monclient(hunting): get_auth_request con 0x7f13c0139b80 auth_method 0 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient(hunting): handle_auth_done global_id 44439 payload 824 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 msgr2=0x7f13c01d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13c0072640 0x7f13c01d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13c0139b80 msgr2=0x7f13c01d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f13c0139b80 0x7f13c01d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 monclient: found mon.noname-d 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.750+0000 7f13beffd640 1 -- 172.21.15.98:0/897691512 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13c00a5840 con 0x7f13c006fe50 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13beffd640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c013fab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c7f0 tx=0x7f13b000ccb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 client.?.objecter ms_handle_connect 0x7f13c006fe50 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13b0015070 con 0x7f13c006fe50 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: dump: 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.772 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13b0007a90 con 0x7f13c006fe50 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 5 monclient: authenticate success, global_id 44439 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 10 monclient: _renew_subs 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:54.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13c013caa0 con 0x7f13c006fe50 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13b0007c80 con 0x7f13c006fe50 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 10 monclient: dump: 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13bd7fa640 4 set_mon_vals no callback set 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 10 client.44439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 10 monclient: _renew_subs 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f13c00a51f0 con 0x7f13c006fe50 2024-09-11T02:58:54.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13c56f0640 1 librados: init done 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.754+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f13b00113f0 con 0x7f13c006fe50 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13c56f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f13ac03dfa0 0x7f13ac040460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13c56f0640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013ab70 0x7f13c01d2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f13c0140640 con 0x7f13c013ab70 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(363..363 src has 1..363) ==== 6202+0+0 (secure 0 0 0) 0x7f13b004fb50 con 0x7f13c006fe50 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 client.44439.objecter ms_dispatch 0x7f13c0000ba0 osd_map(363..363 src has 1..363) 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 3 client.44439.objecter handle_osd_map got epochs [363,363] > 0 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 3 client.44439.objecter handle_osd_map decoding full epoch 363 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 20 client.44439.objecter dump_active .. 0 homeless 2024-09-11T02:58:54.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13be7fc640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f13ac03dfa0 0x7f13ac040460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013ab70 0x7f13c01d2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient: get_auth_request con 0x7f13c013ab70 auth_method 0 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13be7fc640 10 monclient: get_auth_request con 0x7f13ac03dfa0 auth_method 0 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_done global_id 19735 payload 824 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 1 -- 172.21.15.98:0/897691512 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13c013b6d0 con 0x7f13c013ab70 2024-09-11T02:58:54.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 client.44439.objecter ms_handle_connect 0x7f13c013ab70 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 client.44439.objecter resend_mon_ops 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13bf7fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013ab70 0x7f13c01d2150 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fb0 tx=0x7f13b80115a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 client.44439.objecter ms_handle_connect 0x7f13ac03dfa0 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13be7fc640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f13ac03dfa0 0x7f13ac040460 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7f13b40ad000 tx=0x7f13b4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13b801a020 con 0x7f13c013ab70 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 monclient: dump: 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.778 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f13b8016470 con 0x7f13c013ab70 2024-09-11T02:58:54.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013ab70 msgr2=0x7f13c01d2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013ab70 0x7f13c01d2150 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fb0 tx=0x7f13b80115a0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013ab70 msgr2=0x7f13c01d2150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.758+0000 7f13a77fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013ab70 0x7f13c01d2150 secure :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fb0 tx=0x7f13b80115a0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13c56f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13c56f0640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013fff0 0x7f13c00a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13c56f0640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f13c006ca10 con 0x7f13c013fff0 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013fff0 0x7f13c00a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 10 monclient: get_auth_request con 0x7f13c013fff0 auth_method 0 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.766+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13bf7fe640 10 monclient(hunting): handle_auth_done global_id 19744 payload 824 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13bf7fe640 1 -- 172.21.15.98:0/897691512 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13c00a8180 con 0x7f13c013fff0 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13bf7fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013fff0 0x7f13c00a0e60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fb0 tx=0x7f13b8004a50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 10 client.44439.objecter ms_handle_connect 0x7f13c013fff0 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 10 client.44439.objecter resend_mon_ops 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f13b801a050 con 0x7f13c013fff0 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 10 monclient: dump: 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.783 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.784 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f13b80210d0 con 0x7f13c013fff0 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:54.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013fff0 msgr2=0x7f13c00a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013fff0 0x7f13c00a0e60 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fb0 tx=0x7f13b8004a50 comp rx=0 tx=0).stop 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013fff0 msgr2=0x7f13c00a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13a77fe640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13c013fff0 0x7f13c00a0e60 secure :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f13b8009fb0 tx=0x7f13b8004a50 comp rx=0 tx=0).stop 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 68, 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 550540, 2024-09-11T02:58:54.786 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.787 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.788 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.789 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:58:54.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:58:54.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:58:54.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:58:54.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.799 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:58:54.802 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.804 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:58:54.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:58:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.808 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:58:54.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:58:54.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:58:54.810 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:58:54.810 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 10 librados: watch_flush enter 2024-09-11T02:58:54.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 10 librados: watch_flush exit 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 20 client.44439.objecter shutdown clearing up homeless session... 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 10 client.44439.objecter successfully canceled tick 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f13ac03dfa0 msgr2=0x7f13ac040460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f13ac03dfa0 0x7f13ac040460 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7f13b40ad000 tx=0x7f13b4004740 comp rx=0 tx=0).stop 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 10 monclient: shutdown 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 msgr2=0x7f13c013fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13bf7fe640 1 -- 172.21.15.98:0/897691512 reap_dead start 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c013fab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c7f0 tx=0x7f13b000ccb0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 shutdown_connections 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 --2- 172.21.15.98:0/897691512 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13c006fe50 0x7f13c013fab0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 >> 172.21.15.98:0/897691512 conn(0x7f13c00ac730 msgr2=0x7f13c00aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 shutdown_connections 2024-09-11T02:58:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 -- 172.21.15.98:0/897691512 wait complete. 2024-09-11T02:58:54.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.770+0000 7f13c56f0640 1 librados: shutdown 2024-09-11T02:58:54.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T02:58:54.813 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['g', 'h'] 2024-09-11T02:58:54.813 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T02:58:54.813 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-09-11T02:58:54.813 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g store 2024-09-11T02:58:54.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g sync_force --yes-i-really-mean-it 2024-09-11T02:58:54.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient: get_monmap_and_config 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient: monmap: 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:54.938637+0000 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:54.938637+0000 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.937 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:54.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 Processor -- start 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 -- start start 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient: init 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient: _add_conns ranks=[5,10,4,8,7,0,1,3,9,6,2] 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67d8139b10 0x7f67d80725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): picked mon.noname-h con 0x7f67d8139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f67d806fe30 0x7f67d8139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): picked mon.noname-f con 0x7f67d806fe30 addr v2:172.21.15.190:3300/0 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d80a7a10 0x7f67d80a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): picked mon.noname-c con 0x7f67d80a7a10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f67d80a9650 con 0x7f67d806fe30 2024-09-11T02:58:54.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f67d8071350 con 0x7f67d80a7a10 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f67d8071630 con 0x7f67d8139b10 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67de548640 10 monclient(hunting): authenticate will time out at 1488.818899s 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d80a7a10 0x7f67d80a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d80a7a10 0x7f67d80a7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59876/0 (socket says 172.21.15.98:59876) 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 -- 172.21.15.98:0/2086810627 learned_addr learned my addr 172.21.15.98:0/2086810627 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): get_auth_request con 0x7f67d80a7a10 auth_method 0 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d7fff640 1 --2- 172.21.15.98:0/2086810627 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67d8139b10 0x7f67d80725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d77fe640 1 --2- 172.21.15.98:0/2086810627 >> v2:172.21.15.190:3300/0 conn(0x7f67d806fe30 0x7f67d8139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d7fff640 10 monclient(hunting): get_auth_request con 0x7f67d8139b10 auth_method 0 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d77fe640 10 monclient(hunting): get_auth_request con 0x7f67d806fe30 auth_method 0 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient(hunting): handle_auth_done global_id 24070 payload 274 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 -- 172.21.15.98:0/2086810627 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67d8139b10 msgr2=0x7f67d80725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 --2- 172.21.15.98:0/2086810627 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67d8139b10 0x7f67d80725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 -- 172.21.15.98:0/2086810627 >> v2:172.21.15.190:3300/0 conn(0x7f67d806fe30 msgr2=0x7f67d8139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 --2- 172.21.15.98:0/2086810627 >> v2:172.21.15.190:3300/0 conn(0x7f67d806fe30 0x7f67d8139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 monclient: found mon.noname-c 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.934+0000 7f67dcae5640 1 -- 172.21.15.98:0/2086810627 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d80702b0 con 0x7f67d80a7a10 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67dcae5640 1 --2- 172.21.15.98:0/2086810627 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d80a7a10 0x7f67d80a7ec0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009e50 tx=0x7f67d001c8f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=172fad36196b0d0b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 1 -- 172.21.15.98:0/2086810627 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d00090d0 con 0x7f67d80a7a10 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: dump: 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 1 -- 172.21.15.98:0/2086810627 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d00a59a0 con 0x7f67d80a7a10 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 1 -- 172.21.15.98:0/2086810627 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d00ad3f0 con 0x7f67d80a7a10 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 5 monclient: authenticate success, global_id 24070 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 4 set_mon_vals no callback set 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67d6ffd640 10 monclient: dump: 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 10 monclient: shutdown 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 -- 172.21.15.98:0/2086810627 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d80a7a10 msgr2=0x7f67d80a7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 --2- 172.21.15.98:0/2086810627 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d80a7a10 0x7f67d80a7ec0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009e50 tx=0x7f67d001c8f0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 -- 172.21.15.98:0/2086810627 shutdown_connections 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 --2- 172.21.15.98:0/2086810627 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d80a7a10 0x7f67d80a7ec0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 --2- 172.21.15.98:0/2086810627 >> v2:172.21.15.190:3300/0 conn(0x7f67d806fe30 0x7f67d8139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 --2- 172.21.15.98:0/2086810627 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67d8139b10 0x7f67d80725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 -- 172.21.15.98:0/2086810627 >> 172.21.15.98:0/2086810627 conn(0x7f67d80ac770 msgr2=0x7f67d80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 -- 172.21.15.98:0/2086810627 shutdown_connections 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 1 -- 172.21.15.98:0/2086810627 wait complete. 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 10 monclient: build_initial_monmap 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.938+0000 7f67de548640 10 monclient: monmap: 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:54.956 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 librados: starting msgr at 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 librados: starting objecter 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 Processor -- start 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 -- start start 2024-09-11T02:58:54.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 librados: setting wanted keys 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 librados: calling monclient init 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient: init 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient: _add_conns ranks=[2,0,4,7,3,5,6,8,1] 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67d80a7a10 0x7f67d813a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): picked mon.noname-b con 0x7f67d80a7a10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8139b10 0x7f67d813acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): picked mon.noname-a con 0x7f67d8139b10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d813b230 0x7f67d8133860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): picked mon.noname-c con 0x7f67d813b230 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81435c0 con 0x7f67d8139b10 2024-09-11T02:58:54.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f67d8143440 con 0x7f67d80a7a10 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f67d8143740 con 0x7f67d813b230 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): authenticate will time out at 1488.824751s 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8139b10 0x7f67d813acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d813b230 0x7f67d8133860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67d80a7a10 0x7f67d813a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8139b10 0x7f67d813acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38686/0 (socket says 172.21.15.98:38686) 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d813b230 0x7f67d8133860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59886/0 (socket says 172.21.15.98:59886) 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 1 -- 172.21.15.98:0/905109246 learned_addr learned my addr 172.21.15.98:0/905109246 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient(hunting): get_auth_request con 0x7f67d8139b10 auth_method 0 2024-09-11T02:58:54.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): get_auth_request con 0x7f67d813b230 auth_method 0 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): get_auth_request con 0x7f67d80a7a10 auth_method 0 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient(hunting): handle_auth_done global_id 24079 payload 824 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67d80a7a10 msgr2=0x7f67d813a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67d80a7a10 0x7f67d813a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8139b10 msgr2=0x7f67d813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8139b10 0x7f67d813acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient: _finish_hunting 0 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 monclient: found mon.noname-c 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 -- 172.21.15.98:0/905109246 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d813d480 con 0x7f67d813b230 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f67d8139b10 0x7f67d813acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67dcae5640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d813b230 0x7f67d8133860 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f67d00ab000 tx=0x7f67d0019690 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 client.?.objecter ms_handle_connect 0x7f67d813b230 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d0003e10 con 0x7f67d813b230 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:54.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: dump: 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.963 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: _finish_auth 0 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: _check_auth_tickets 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d00bc070 con 0x7f67d813b230 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 5 monclient: authenticate success, global_id 24079 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient: _renew_subs 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d8073820 con 0x7f67d813b230 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:54.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d00a5d60 con 0x7f67d813b230 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 monclient: dump: 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.965 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 client.24079.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d67fc640 4 set_mon_vals no callback set 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient: _renew_subs 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d81407f0 con 0x7f67d813b230 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 librados: init done 2024-09-11T02:58:54.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f67d00039a0 con 0x7f67d813b230 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d806fe30 0x7f67d8133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f67d806ca10 con 0x7f67d806fe30 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(363..363 src has 1..363) ==== 6202+0+0 (secure 0 0 0) 0x7f67d00ea0f0 con 0x7f67d813b230 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 10 client.24079.objecter ms_dispatch 0x7f67d8000ba0 osd_map(363..363 src has 1..363) 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 3 client.24079.objecter handle_osd_map got epochs [363,363] > 0 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 3 client.24079.objecter handle_osd_map decoding full epoch 363 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d806fe30 0x7f67d8133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d4ff9640 20 client.24079.objecter dump_active .. 0 homeless 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient: get_auth_request con 0x7f67d806fe30 auth_method 0 2024-09-11T02:58:54.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.942+0000 7f67d77fe640 10 monclient: get_auth_request con 0x7f67c403df40 auth_method 0 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d7fff640 10 monclient(hunting): handle_auth_done global_id 64290 payload 824 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d7fff640 1 -- 172.21.15.98:0/905109246 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67d813cbe0 con 0x7f67d806fe30 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 client.24079.objecter ms_handle_connect 0x7f67d806fe30 2024-09-11T02:58:54.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 client.24079.objecter resend_mon_ops 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d7fff640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d806fe30 0x7f67d8133da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f67c8010820 tx=0x7f67c8010ce0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 client.24079.objecter ms_handle_connect 0x7f67c403df40 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d77fe640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 crypto rx=0x7f67d80a6010 tx=0x7f67cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67c800eea0 con 0x7f67d806fe30 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 monclient: dump: 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.969 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f67c800ea30 con 0x7f67d806fe30 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d806fe30 msgr2=0x7f67d8133da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d806fe30 0x7f67d8133da0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f67c8010820 tx=0x7f67c8010ce0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d806fe30 msgr2=0x7f67d8133da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.946+0000 7f67d4ff9640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d806fe30 0x7f67d8133da0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f67c8010820 tx=0x7f67c8010ce0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:58:54.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d81342e0 0x7f67d80a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:54.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f67d806e870 con 0x7f67d81342e0 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d81342e0 0x7f67d80a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient: get_auth_request con 0x7f67d81342e0 auth_method 0 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 10 monclient(hunting): handle_auth_done global_id 64299 payload 824 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 1 -- 172.21.15.98:0/905109246 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f67d81346c0 con 0x7f67d81342e0 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d7fff640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d81342e0 0x7f67d80a0e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67c8010820 tx=0x7f67c80046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:54.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 10 client.24079.objecter ms_handle_connect 0x7f67d81342e0 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 10 client.24079.objecter resend_mon_ops 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67c802c080 con 0x7f67d81342e0 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 10 monclient: dump: 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:54.974 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f67c8003990 con 0x7f67d81342e0 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d81342e0 msgr2=0x7f67d80a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d81342e0 0x7f67d80a0e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67c8010820 tx=0x7f67c80046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d81342e0 msgr2=0x7f67d80a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:58:54.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67d4ff9640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f67d81342e0 0x7f67d80a0e60 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f67c8010820 tx=0x7f67c80046d0 comp rx=0 tx=0).stop 2024-09-11T02:58:54.976 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:58:54.976 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-09-11T02:58:54.976 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T02:58:54.976 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 10 librados: watch_flush enter 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 10 librados: watch_flush exit 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 20 client.24079.objecter shutdown clearing up homeless session... 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 10 client.24079.objecter successfully canceled tick 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 msgr2=0x7f67c4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 crypto rx=0x7f67d80a6010 tx=0x7f67cc008040 comp rx=0 tx=0).stop 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 10 monclient: shutdown 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:54.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d813b230 msgr2=0x7f67d8133860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 --2- 172.21.15.98:0/905109246 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67d813b230 0x7f67d8133860 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f67d00ab000 tx=0x7f67d0019690 comp rx=0 tx=0).stop 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67dcae5640 1 -- 172.21.15.98:0/905109246 reap_dead start 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67dcae5640 1 -- 172.21.15.98:0/905109246 reap_dead start 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 shutdown_connections 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 >> 172.21.15.98:0/905109246 conn(0x7f67d80ac770 msgr2=0x7f67d8137cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 shutdown_connections 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 -- 172.21.15.98:0/905109246 wait complete. 2024-09-11T02:58:54.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:54.954+0000 7f67de548640 1 librados: shutdown 2024-09-11T02:58:54.979 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-09-11T02:58:54.980 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-11T02:58:54.980 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:58:54.995 INFO:tasks.ceph.mon.g:Stopped 2024-09-11T02:58:54.995 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-09-11T02:58:54.995 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-09-11T02:58:54.996 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-11T02:58:54.996 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T02:58:55.057 INFO:tasks.ceph.mon.h:Stopped 2024-09-11T02:58:55.057 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-09-11T02:58:55.057 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:58:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.174+0000 7fa28693d640 10 monclient: get_monmap_and_config 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.174+0000 7fa28693d640 10 monclient: build_initial_monmap 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.174+0000 7fa28693d640 10 monclient: monmap: 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:55.179776+0000 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:55.179776+0000 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:55.179 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:55.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.174+0000 7fa28693d640 1 Processor -- start 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.174+0000 7fa28693d640 1 -- start start 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.174+0000 7fa28693d640 10 monclient: init 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient: _add_conns ranks=[1,4,7,2,6,9,5,0,8,10,3] 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa28006f650 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2800a8380 0x7fa280072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient(hunting): picked mon.noname-c con 0x7fa2800a8380 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa280070140 0x7fa280140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient(hunting): picked mon.noname-i con 0x7fa280070140 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa280071a80 con 0x7fa28006f650 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa2800a8800 con 0x7fa2800a8380 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa280071d60 con 0x7fa280070140 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient(hunting): authenticate will time out at 1489.060280s 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2800a8380 0x7fa280072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2800a8380 0x7fa280072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59898/0 (socket says 172.21.15.98:59898) 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2161221787 learned_addr learned my addr 172.21.15.98:0/2161221787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): get_auth_request con 0x7fa2800a8380 auth_method 0 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa284eda640 1 --2- 172.21.15.98:0/2161221787 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa280070140 0x7fa280140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_done global_id 24088 payload 274 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2161221787 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa280070140 msgr2=0x7fa280140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 --2- 172.21.15.98:0/2161221787 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa280070140 0x7fa280140c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2161221787 >> v1:172.21.15.190:6789/0 conn(0x7fa28006f650 legacy=0x7fa28006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 monclient: found mon.noname-c 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2161221787 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2800714c0 con 0x7fa2800a8380 2024-09-11T02:58:55.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa284eda640 10 monclient: get_auth_request con 0x7fa280070140 auth_method 0 2024-09-11T02:58:55.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa284eda640 1 --2- 172.21.15.98:0/2161221787 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa280070140 0x7fa280140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27f7fe640 1 --2- 172.21.15.98:0/2161221787 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2800a8380 0x7fa280072640 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa268009e80 tx=0x7fa26801c910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d4edebed4c84260e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 1 -- 172.21.15.98:0/2161221787 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2680090d0 con 0x7fa2800a8380 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: dump: 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:55.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: _finish_auth 0 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: _check_auth_tickets 2024-09-11T02:58:55.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 1 -- 172.21.15.98:0/2161221787 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2680a5b80 con 0x7fa2800a8380 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 1 -- 172.21.15.98:0/2161221787 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2680a5d70 con 0x7fa2800a8380 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa27effd640 10 monclient: dump: 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 5 monclient: authenticate success, global_id 24088 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.178+0000 7fa28693d640 4 set_mon_vals no callback set 2024-09-11T02:58:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient: shutdown 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- 172.21.15.98:0/2161221787 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2800a8380 msgr2=0x7fa280072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 --2- 172.21.15.98:0/2161221787 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2800a8380 0x7fa280072640 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fa268009e80 tx=0x7fa26801c910 comp rx=0 tx=0).stop 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- 172.21.15.98:0/2161221787 shutdown_connections 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 --2- 172.21.15.98:0/2161221787 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa280070140 0x7fa280140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 --2- 172.21.15.98:0/2161221787 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa2800a8380 0x7fa280072640 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- 172.21.15.98:0/2161221787 >> 172.21.15.98:0/2161221787 conn(0x7fa2800ac8e0 msgr2=0x7fa2800a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- 172.21.15.98:0/2161221787 shutdown_connections 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- 172.21.15.98:0/2161221787 wait complete. 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient: build_initial_monmap 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient: monmap: 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 librados: starting msgr at 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 librados: starting objecter 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 Processor -- start 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- start start 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 librados: setting wanted keys 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 librados: calling monclient init 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient: init 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient: _add_conns ranks=[7,8,1,4,2,0,3,6,5] 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa28006f650 0x7fa280128c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): picked mon.noname-i con 0x7fa28006f650 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa280070140 0x7fa2801d1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): picked mon.noname-e con 0x7fa280070140 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2800a8380 0x7fa280126650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): picked mon.noname-f con 0x7fa2800a8380 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa280143d10 con 0x7fa2800a8380 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa280143b90 con 0x7fa28006f650 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa280143e90 con 0x7fa280070140 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa28693d640 10 monclient(hunting): authenticate will time out at 1489.066360s 2024-09-11T02:58:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa280070140 0x7fa2801d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa280070140 0x7fa2801d1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:39784/0 (socket says 172.21.15.98:39784) 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2983717938 learned_addr learned my addr 172.21.15.98:0/2983717938 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): get_auth_request con 0x7fa280070140 auth_method 0 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27ffff640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa28006f650 0x7fa280128c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27ffff640 10 monclient(hunting): get_auth_request con 0x7fa28006f650 auth_method 0 2024-09-11T02:58:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa284eda640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2800a8380 0x7fa280126650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa284eda640 10 monclient(hunting): get_auth_request con 0x7fa2800a8380 auth_method 0 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa284eda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa284eda640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa284eda640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient(hunting): handle_auth_done global_id 24605 payload 824 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa28006f650 msgr2=0x7fa280128c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa28006f650 0x7fa280128c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2800a8380 msgr2=0x7fa280126650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2800a8380 0x7fa280126650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 monclient: found mon.noname-e 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 -- 172.21.15.98:0/2983717938 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2800a75a0 con 0x7fa280070140 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27ffff640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa28006f650 0x7fa280128c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27f7fe640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa280070140 0x7fa2801d1480 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fa268009040 tx=0x7fa2800a7f40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 client.?.objecter ms_handle_connect 0x7fa280070140 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 1 -- 172.21.15.98:0/2983717938 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2680bc020 con 0x7fa280070140 2024-09-11T02:58:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: dump: 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: _finish_auth 0 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: _check_auth_tickets 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 1 -- 172.21.15.98:0/2983717938 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa268003df0 con 0x7fa280070140 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 1 -- 172.21.15.98:0/2983717938 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2680b8470 con 0x7fa280070140 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27cff9640 10 monclient: dump: 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.182+0000 7fa27e7fc640 4 set_mon_vals no callback set 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 5 monclient: authenticate success, global_id 24605 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 monclient: _renew_subs 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2800a7150 con 0x7fa280070140 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 client.24605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 monclient: _renew_subs 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28012a170 con 0x7fa280070140 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 1 librados: init done 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2801c8d20 con 0x7fa280070140 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 1 -- 172.21.15.98:0/2983717938 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa2680b8630 con 0x7fa280070140 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa26003df40 0x7fa260040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 1 -- 172.21.15.98:0/2983717938 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(363..363 src has 1..363) ==== 6202+0+0 (secure 0 0 0) 0x7fa2680eaf90 con 0x7fa280070140 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 10 client.24605.objecter ms_dispatch 0x7fa280000ba0 osd_map(363..363 src has 1..363) 2024-09-11T02:58:55.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 3 client.24605.objecter handle_osd_map got epochs [363,363] > 0 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 3 client.24605.objecter handle_osd_map decoding full epoch 363 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 20 client.24605.objecter dump_active .. 0 homeless 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27ffff640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa26003df40 0x7fa260040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27ffff640 10 monclient: get_auth_request con 0x7fa26003df40 auth_method 0 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27cff9640 10 client.24605.objecter ms_handle_connect 0x7fa26003df40 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.186+0000 7fa27ffff640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa26003df40 0x7fa260040400 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7fa270003ec0 tx=0x7fa270009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.190+0000 7fa27cff9640 1 -- 172.21.15.98:0/2983717938 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fa2680c6030 con 0x7fa280070140 2024-09-11T02:58:55.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.190+0000 7fa27cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:55.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.190+0000 7fa27cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:55.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa28693d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:58:55.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa28693d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:55.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa28693d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:58:55.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa2801280f0 con 0x7fa280070140 2024-09-11T02:58:55.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa27cff9640 1 -- 172.21.15.98:0/2983717938 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fa2800a7150 con 0x7fa280070140 2024-09-11T02:58:55.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa27cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:55.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa27cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:55.422 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":68,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:58:55.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.418+0000 7fa28693d640 10 librados: watch_flush enter 2024-09-11T02:58:55.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 10 librados: watch_flush exit 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 20 client.24605.objecter shutdown clearing up homeless session... 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 10 client.24605.objecter successfully canceled tick 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa26003df40 msgr2=0x7fa260040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa26003df40 0x7fa260040400 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7fa270003ec0 tx=0x7fa270009040 comp rx=0 tx=0).stop 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 10 monclient: shutdown 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa280070140 msgr2=0x7fa2801d1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa280070140 0x7fa2801d1480 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fa268009040 tx=0x7fa2800a7f40 comp rx=0 tx=0).stop 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 shutdown_connections 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa26003df40 0x7fa260040400 unknown :-1 s=CLOSED pgs=3555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2800a8380 0x7fa280126650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa280070140 0x7fa2801d1480 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 --2- 172.21.15.98:0/2983717938 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa28006f650 0x7fa280128c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:55.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 >> 172.21.15.98:0/2983717938 conn(0x7fa2800ac8e0 msgr2=0x7fa2800a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:55.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 shutdown_connections 2024-09-11T02:58:55.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 -- 172.21.15.98:0/2983717938 wait complete. 2024-09-11T02:58:55.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:55.422+0000 7fa28693d640 1 librados: shutdown 2024-09-11T02:58:58.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient: get_monmap_and_config 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient: build_initial_monmap 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient: monmap: 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:58:58.571061+0000 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:58:58.571061+0000 2024-09-11T02:58:58.569 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:58:58.570 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 1 Processor -- start 2024-09-11T02:58:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 1 -- start start 2024-09-11T02:58:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient: init 2024-09-11T02:58:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient: _add_conns ranks=[9,6,1,4,8,2,3,10,5,7,0] 2024-09-11T02:58:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7701399e0 0x7fe770139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient(hunting): picked mon.noname-a con 0x7fe7701399e0 addr v2:172.21.15.98:3300/0 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe77013a300 0x7fe7700a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient(hunting): picked mon.noname-d con 0x7fe77013a300 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe770071b60 addr v1:172.21.15.190:6789/0 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe77006ed40 con 0x7fe770071b60 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe7700a7870 con 0x7fe7701399e0 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe7700704a0 con 0x7fe77013a300 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe7773ed640 10 monclient(hunting): authenticate will time out at 1492.451385s 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7701399e0 0x7fe770139dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:58.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe77013a300 0x7fe7700a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe7701399e0 0x7fe770139dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47386/0 (socket says 172.21.15.98:47386) 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 1 -- 172.21.15.98:0/830662147 learned_addr learned my addr 172.21.15.98:0/830662147 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 10 monclient(hunting): get_auth_request con 0x7fe7701399e0 auth_method 0 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): get_auth_request con 0x7fe77013a300 auth_method 0 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient(hunting): handle_auth_done global_id 44448 payload 274 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 1 -- 172.21.15.98:0/830662147 >> v2:172.21.15.98:3300/0 conn(0x7fe7701399e0 msgr2=0x7fe770139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 1 --2- 172.21.15.98:0/830662147 >> v2:172.21.15.98:3300/0 conn(0x7fe7701399e0 0x7fe770139dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 1 -- 172.21.15.98:0/830662147 >> v1:172.21.15.190:6789/0 conn(0x7fe770071b60 legacy=0x7fe7700a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient: _finish_hunting 0 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 1 monclient: found mon.noname-d 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:58:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe774988640 1 -- 172.21.15.98:0/830662147 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77006f080 con 0x7fe77013a300 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.566+0000 7fe775189640 1 --2- 172.21.15.98:0/830662147 >> v2:172.21.15.98:3300/0 conn(0x7fe7701399e0 0x7fe770139dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe774988640 1 --2- 172.21.15.98:0/830662147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe77013a300 0x7fe7700a58c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe7600097c0 tx=0x7fe76001c360 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e7c66904871b68ae server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 1 -- 172.21.15.98:0/830662147 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7600b0070 con 0x7fe77013a300 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: dump: 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:58.578 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: _finish_auth 0 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: _check_auth_tickets 2024-09-11T02:58:58.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 1 -- 172.21.15.98:0/830662147 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7700a6720 con 0x7fe77013a300 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 1 -- 172.21.15.98:0/830662147 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7600aac80 con 0x7fe77013a300 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 5 monclient: authenticate success, global_id 44448 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 10 monclient: get_monmap_and_config success 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 4 set_mon_vals no callback set 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe767fff640 10 monclient: dump: 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:58.580 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 10 monclient: shutdown 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 -- 172.21.15.98:0/830662147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe77013a300 msgr2=0x7fe7700a58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 --2- 172.21.15.98:0/830662147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe77013a300 0x7fe7700a58c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe7600097c0 tx=0x7fe76001c360 comp rx=0 tx=0).stop 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 -- 172.21.15.98:0/830662147 shutdown_connections 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 --2- 172.21.15.98:0/830662147 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe77013a300 0x7fe7700a58c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 --2- 172.21.15.98:0/830662147 >> v2:172.21.15.98:3300/0 conn(0x7fe7701399e0 0x7fe770139dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 -- 172.21.15.98:0/830662147 >> 172.21.15.98:0/830662147 conn(0x7fe7700aa4b0 msgr2=0x7fe7700a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 -- 172.21.15.98:0/830662147 shutdown_connections 2024-09-11T02:58:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 1 -- 172.21.15.98:0/830662147 wait complete. 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 10 monclient: build_initial_monmap 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.570+0000 7fe7773ed640 10 monclient: monmap: 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:58:58.587 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 librados: starting msgr at 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 librados: starting objecter 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 Processor -- start 2024-09-11T02:58:58.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 -- start start 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 librados: setting wanted keys 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 librados: calling monclient init 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: init 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _reopen_session rank -1 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _add_conns ranks=[7,4,1,3,6,5,2,8,0] 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe770071b60 0x7fe7701409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): picked mon.noname-i con 0x7fe770071b60 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe7701399e0 0x7fe770140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): picked mon.noname-c con 0x7fe7701399e0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe77013a300 0x7fe77013a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): picked mon.noname-f con 0x7fe77013a300 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:58.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe770144510 con 0x7fe77013a300 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe770144390 con 0x7fe7701399e0 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): start opening mon connection 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe770144690 con 0x7fe770071b60 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): _renew_subs 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient(hunting): authenticate will time out at 1492.456983s 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe7701399e0 0x7fe770140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe7701399e0 0x7fe770140ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:51222/0 (socket says 172.21.15.98:51222) 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 -- 172.21.15.98:0/3142944822 learned_addr learned my addr 172.21.15.98:0/3142944822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): get_auth_request con 0x7fe7701399e0 auth_method 0 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:58.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe775189640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe770071b60 0x7fe7701409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe77598a640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe77013a300 0x7fe77013a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe77598a640 10 monclient(hunting): get_auth_request con 0x7fe77013a300 auth_method 0 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe77598a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe77598a640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe77598a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe775189640 10 monclient(hunting): get_auth_request con 0x7fe770071b60 auth_method 0 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe775189640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe775189640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:58:58.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe775189640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient(hunting): handle_auth_done global_id 24097 payload 824 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 -- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe770071b60 msgr2=0x7fe7701409a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe770071b60 0x7fe7701409a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 -- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe77013a300 msgr2=0x7fe77013a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe77013a300 0x7fe77013a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient: _finish_hunting 0 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 monclient: found mon.noname-c 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 -- 172.21.15.98:0/3142944822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77013e460 con 0x7fe7701399e0 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe77598a640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe77013a300 0x7fe77013a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe774988640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe7701399e0 0x7fe770140ee0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fe7600097c0 tx=0x7fe760019100 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:58.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 client.?.objecter ms_handle_connect 0x7fe7701399e0 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 1 -- 172.21.15.98:0/3142944822 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7600b0040 con 0x7fe7701399e0 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: dump: 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:58.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: _finish_auth 0 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: _check_auth_tickets 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 5 monclient: authenticate success, global_id 24097 2024-09-11T02:58:58.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _renew_subs 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 1 -- 172.21.15.98:0/3142944822 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe760003760 con 0x7fe7701399e0 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7700a8520 con 0x7fe7701399e0 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 1 -- 172.21.15.98:0/3142944822 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe760003d80 con 0x7fe7701399e0 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 monclient: dump: 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.595 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:58:58.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7677fe640 4 set_mon_vals no callback set 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 client.24097.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _renew_subs 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7700a7fb0 con 0x7fe7701399e0 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 librados: init done 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 1 -- 172.21.15.98:0/3142944822 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe7600b3070 con 0x7fe7701399e0 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe770139dc0 con 0x7fe7701399e0 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe75403df40 0x7fe754040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 1 -- 172.21.15.98:0/3142944822 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(363..363 src has 1..363) ==== 6202+0+0 (secure 0 0 0) 0x7fe7600e9020 con 0x7fe7701399e0 2024-09-11T02:58:58.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 10 client.24097.objecter ms_dispatch 0x7fe770000ba0 osd_map(363..363 src has 1..363) 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 3 client.24097.objecter handle_osd_map got epochs [363,363] > 0 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 3 client.24097.objecter handle_osd_map decoding full epoch 363 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe765ffb640 20 client.24097.objecter dump_active .. 0 homeless 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe775189640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe75403df40 0x7fe754040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.574+0000 7fe775189640 10 monclient: get_auth_request con 0x7fe75403df40 auth_method 0 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.578+0000 7fe765ffb640 10 client.24097.objecter ms_handle_connect 0x7fe75403df40 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.578+0000 7fe775189640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe75403df40 0x7fe754040400 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7fe768009240 tx=0x7fe76800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.578+0000 7fe765ffb640 1 -- 172.21.15.98:0/3142944822 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe760016070 con 0x7fe7701399e0 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.578+0000 7fe765ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:58:58.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.578+0000 7fe765ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:58:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:58:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:58:58.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe77013c100 con 0x7fe7701399e0 2024-09-11T02:58:58.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe765ffb640 1 -- 172.21.15.98:0/3142944822 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fe7600aa3a0 con 0x7fe7701399e0 2024-09-11T02:58:58.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe765ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:58:58.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe765ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:58:58.808 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":68,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:58:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 10 librados: watch_flush enter 2024-09-11T02:58:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 10 librados: watch_flush exit 2024-09-11T02:58:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 20 client.24097.objecter shutdown clearing up homeless session... 2024-09-11T02:58:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 10 client.24097.objecter successfully canceled tick 2024-09-11T02:58:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe75403df40 msgr2=0x7fe754040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:58.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe75403df40 0x7fe754040400 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7fe768009240 tx=0x7fe76800b040 comp rx=0 tx=0).stop 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 10 monclient: shutdown 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe7701399e0 msgr2=0x7fe770140ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe7701399e0 0x7fe770140ee0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fe7600097c0 tx=0x7fe760019100 comp rx=0 tx=0).stop 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 shutdown_connections 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe75403df40 0x7fe754040400 unknown :-1 s=CLOSED pgs=3556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe77013a300 0x7fe77013a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe7701399e0 0x7fe770140ee0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 --2- 172.21.15.98:0/3142944822 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe770071b60 0x7fe7701409a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 >> 172.21.15.98:0/3142944822 conn(0x7fe7700aa4b0 msgr2=0x7fe7700a4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 shutdown_connections 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 -- 172.21.15.98:0/3142944822 wait complete. 2024-09-11T02:58:58.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:58:58.806+0000 7fe7773ed640 1 librados: shutdown 2024-09-11T02:59:01.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:59:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient: get_monmap_and_config 2024-09-11T02:59:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient: build_initial_monmap 2024-09-11T02:59:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient: monmap: 2024-09-11T02:59:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:01.955219+0000 2024-09-11T02:59:01.953 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:01.955219+0000 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:01.954 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:01.955 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 Processor -- start 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 -- start start 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient: init 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient: _add_conns ranks=[10,4,5,2,9,6,3,0,7,1,8] 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f7f5006f650 0x7f7f50072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): picked mon.noname-f con 0x7f7f5006f650 addr v2:172.21.15.190:3300/0 2024-09-11T02:59:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f500a8380 0x7f7f5006fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): picked mon.noname-c con 0x7f7f500a8380 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7f5006ff70 0x7f7f501397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): picked mon.noname-h con 0x7f7f5006ff70 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f7f500a8800 con 0x7f7f5006f650 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7f50071a80 con 0x7f7f500a8380 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f7f50071d60 con 0x7f7f5006ff70 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f56e23640 10 monclient(hunting): authenticate will time out at 1495.835583s 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f553c0640 1 -- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7f5006ff70 msgr2=0x7f7f501397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3302/0 2024-09-11T02:59:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f553c0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7f5006ff70 0x7f7f501397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f500a8380 0x7f7f5006fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f500a8380 0x7f7f5006fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:51236/0 (socket says 172.21.15.98:51236) 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 1 -- 172.21.15.98:0/2022651218 learned_addr learned my addr 172.21.15.98:0/2022651218 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 10 monclient(hunting): get_auth_request con 0x7f7f500a8380 auth_method 0 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f54bbf640 1 --2- 172.21.15.98:0/2022651218 >> v2:172.21.15.190:3300/0 conn(0x7f7f5006f650 0x7f7f50072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f54bbf640 10 monclient(hunting): get_auth_request con 0x7f7f5006f650 auth_method 0 2024-09-11T02:59:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f54bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f54bbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.950+0000 7f7f54bbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_done global_id 24106 payload 274 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 1 -- 172.21.15.98:0/2022651218 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7f5006ff70 msgr2=0x7f7f501397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 1 --2- 172.21.15.98:0/2022651218 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7f5006ff70 0x7f7f501397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 1 -- 172.21.15.98:0/2022651218 >> v2:172.21.15.190:3300/0 conn(0x7f7f5006f650 msgr2=0x7f7f50072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 1 --2- 172.21.15.98:0/2022651218 >> v2:172.21.15.190:3300/0 conn(0x7f7f5006f650 0x7f7f50072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 1 monclient: found mon.noname-c 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 1 -- 172.21.15.98:0/2022651218 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f500714c0 con 0x7f7f500a8380 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f54bbf640 1 --2- 172.21.15.98:0/2022651218 >> v2:172.21.15.190:3300/0 conn(0x7f7f5006f650 0x7f7f50072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:59:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4ffff640 1 --2- 172.21.15.98:0/2022651218 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f500a8380 0x7f7f5006fa30 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009e80 tx=0x7f7f4001d060 comp rx=0 tx=0).ready entity=mon.4 client_cookie=130be72498ba5d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 1 -- 172.21.15.98:0/2022651218 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f40009110 con 0x7f7f500a8380 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: dump: 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: _finish_auth 0 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:59:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 1 -- 172.21.15.98:0/2022651218 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f400a68e0 con 0x7f7f500a8380 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 1 -- 172.21.15.98:0/2022651218 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f400a7450 con 0x7f7f500a8380 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 5 monclient: authenticate success, global_id 24106 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 4 set_mon_vals no callback set 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f4f7fe640 10 monclient: dump: 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 10 monclient: shutdown 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 -- 172.21.15.98:0/2022651218 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f500a8380 msgr2=0x7f7f5006fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 --2- 172.21.15.98:0/2022651218 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f500a8380 0x7f7f5006fa30 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009e80 tx=0x7f7f4001d060 comp rx=0 tx=0).stop 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 -- 172.21.15.98:0/2022651218 shutdown_connections 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 --2- 172.21.15.98:0/2022651218 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7f5006ff70 0x7f7f501397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 --2- 172.21.15.98:0/2022651218 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f500a8380 0x7f7f5006fa30 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 --2- 172.21.15.98:0/2022651218 >> v2:172.21.15.190:3300/0 conn(0x7f7f5006f650 0x7f7f50072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 -- 172.21.15.98:0/2022651218 >> 172.21.15.98:0/2022651218 conn(0x7f7f500ac8e0 msgr2=0x7f7f500a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 -- 172.21.15.98:0/2022651218 shutdown_connections 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.954+0000 7f7f56e23640 1 -- 172.21.15.98:0/2022651218 wait complete. 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: build_initial_monmap 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: monmap: 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 librados: starting msgr at 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 librados: starting objecter 2024-09-11T02:59:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 Processor -- start 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 -- start start 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 librados: setting wanted keys 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 librados: calling monclient init 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: init 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _add_conns ranks=[0,4,7,3,1,2,8,6,5] 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f5006f650 0x7f7f501d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): picked mon.noname-a con 0x7f7f5006f650 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f5006ff70 0x7f7f50136820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): picked mon.noname-c con 0x7f7f5006ff70 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7f500a8380 0x7f7f5013bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): picked mon.noname-i con 0x7f7f500a8380 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50142fc0 con 0x7f7f5006f650 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7f50142e40 con 0x7f7f5006ff70 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f7f50143140 con 0x7f7f500a8380 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient(hunting): authenticate will time out at 1495.841103s 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f5006ff70 0x7f7f50136820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f5006f650 0x7f7f501d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f5006f650 0x7f7f501d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47394/0 (socket says 172.21.15.98:47394) 2024-09-11T02:59:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f5006ff70 0x7f7f50136820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:51252/0 (socket says 172.21.15.98:51252) 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 1 -- 172.21.15.98:0/3766376537 learned_addr learned my addr 172.21.15.98:0/3766376537 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient(hunting): get_auth_request con 0x7f7f5006f650 auth_method 0 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): get_auth_request con 0x7f7f5006ff70 auth_method 0 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient(hunting): handle_auth_done global_id 24115 payload 824 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 -- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7f500a8380 msgr2=0x7f7f5013bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7f500a8380 0x7f7f5013bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 -- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f5006f650 msgr2=0x7f7f501d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f5006f650 0x7f7f501d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 monclient: found mon.noname-c 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 -- 172.21.15.98:0/3766376537 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5013d580 con 0x7f7f5006ff70 2024-09-11T02:59:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f5006f650 0x7f7f501d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4ffff640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f5006ff70 0x7f7f50136820 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f7f50071c00 tx=0x7f7f4001ae40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 client.?.objecter ms_handle_connect 0x7f7f5006ff70 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 1 -- 172.21.15.98:0/3766376537 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f400b6070 con 0x7f7f5006ff70 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: dump: 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: _finish_auth 0 2024-09-11T02:59:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 5 monclient: authenticate success, global_id 24115 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 1 -- 172.21.15.98:0/3766376537 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f40003e10 con 0x7f7f5006ff70 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _renew_subs 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f50072910 con 0x7f7f5006ff70 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 1 -- 172.21.15.98:0/3766376537 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f400a6ca0 con 0x7f7f5006ff70 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4effd640 4 set_mon_vals no callback set 2024-09-11T02:59:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 client.24115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 monclient: dump: 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _renew_subs 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f50137580 con 0x7f7f5006ff70 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 librados: init done 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f501397c0 con 0x7f7f5006ff70 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 1 -- 172.21.15.98:0/3766376537 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f7f400039a0 con 0x7f7f5006ff70 2024-09-11T02:59:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f3003df30 0x7f7f300403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 1 -- 172.21.15.98:0/3766376537 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(363..363 src has 1..363) ==== 6202+0+0 (secure 0 0 0) 0x7f7f400e9de0 con 0x7f7f5006ff70 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 10 client.24115.objecter ms_dispatch 0x7f7f50000ba0 osd_map(363..363 src has 1..363) 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 3 client.24115.objecter handle_osd_map got epochs [363,363] > 0 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 3 client.24115.objecter handle_osd_map decoding full epoch 363 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f4d7fa640 20 client.24115.objecter dump_active .. 0 homeless 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f3003df30 0x7f7f300403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.958+0000 7f7f54bbf640 10 monclient: get_auth_request con 0x7f7f3003df30 auth_method 0 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.962+0000 7f7f4d7fa640 10 client.24115.objecter ms_handle_connect 0x7f7f3003df30 2024-09-11T02:59:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.962+0000 7f7f54bbf640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f3003df30 0x7f7f300403f0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009e00 tx=0x7f7f38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.962+0000 7f7f4d7fa640 1 -- 172.21.15.98:0/3766376537 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f7f40016600 con 0x7f7f5006ff70 2024-09-11T02:59:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.962+0000 7f7f4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:01.962+0000 7f7f4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.190+0000 7f7f56e23640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:59:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.190+0000 7f7f56e23640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:59:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.190+0000 7f7f56e23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.190+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7f501d1f80 con 0x7f7f5006ff70 2024-09-11T02:59:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f4d7fa640 1 -- 172.21.15.98:0/3766376537 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f7f400a68b0 con 0x7f7f5006ff70 2024-09-11T02:59:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:59:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:02.196 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":68,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:59:02.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 10 librados: watch_flush enter 2024-09-11T02:59:02.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 10 librados: watch_flush exit 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 20 client.24115.objecter shutdown clearing up homeless session... 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 10 client.24115.objecter successfully canceled tick 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f3003df30 msgr2=0x7f7f300403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f3003df30 0x7f7f300403f0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009e00 tx=0x7f7f38008040 comp rx=0 tx=0).stop 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 10 monclient: shutdown 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f5006ff70 msgr2=0x7f7f50136820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f5006ff70 0x7f7f50136820 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f7f50071c00 tx=0x7f7f4001ae40 comp rx=0 tx=0).stop 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 shutdown_connections 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7f3003df30 0x7f7f300403f0 unknown :-1 s=CLOSED pgs=3557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7f500a8380 0x7f7f5013bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7f5006ff70 0x7f7f50136820 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 --2- 172.21.15.98:0/3766376537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f5006f650 0x7f7f501d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 >> 172.21.15.98:0/3766376537 conn(0x7f7f500ac8e0 msgr2=0x7f7f500a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 shutdown_connections 2024-09-11T02:59:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 -- 172.21.15.98:0/3766376537 wait complete. 2024-09-11T02:59:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:02.194+0000 7f7f56e23640 1 librados: shutdown 2024-09-11T02:59:05.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T02:59:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient: get_monmap_and_config 2024-09-11T02:59:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient: build_initial_monmap 2024-09-11T02:59:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient: monmap: 2024-09-11T02:59:05.341 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:05.343320+0000 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:05.343320+0000 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:05.342 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:05.343 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:05.343 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:05.343 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:05.343 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:05.343 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:05.343 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 1 Processor -- start 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 1 -- start start 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient: init 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient: _add_conns ranks=[1,10,9,8,7,2,4,5,6,3,0] 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbb5c139670 addr v1:172.21.15.190:6789/0 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient(hunting): picked mon.noname-f con 0x7fbb5c09ec20 addr v2:172.21.15.190:3300/0 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:05.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient(hunting): picked mon.noname-a con 0x7fbb5c13a160 addr v2:172.21.15.98:3300/0 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5c0a19d0 con 0x7fbb5c139670 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5c09def0 con 0x7fbb5c13a160 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5c09e1d0 con 0x7fbb5c09ec20 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb61b87640 10 monclient(hunting): authenticate will time out at 1499.223626s 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb5bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb5bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47404/0 (socket says 172.21.15.98:47404) 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 learned_addr learned my addr 172.21.15.98:0/4247647401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request con 0x7fbb5c13a160 auth_method 0 2024-09-11T02:59:05.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.338+0000 7fbb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb5c09ec20 auth_method 0 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:05.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:05.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:05.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.342+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:05.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:05.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb5c09ec20 auth_method 0 2024-09-11T02:59:05.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:05.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:05.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:05.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:05.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:05.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:05.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:59:05.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:05.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request con 0x7fbb5c13a160 auth_method 0 2024-09-11T02:59:05.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:05.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:05.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:05.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:59:05.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:05.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:05.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.546+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:59:05.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request con 0x7fbb5c13a160 auth_method 0 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb5c09ec20 auth_method 0 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:05.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:59:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:05.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:05.950+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:59:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request con 0x7fbb5c13a160 auth_method 0 2024-09-11T02:59:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:06.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb5c09ec20 auth_method 0 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:06.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:06.754+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:59:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): tick 2024-09-11T02:59:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:59:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 monclient(hunting): continuing hunt 2024-09-11T02:59:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:59:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 msgr2=0x7fbb5c13ec00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:08.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.190:3300/0 conn(0x7fbb5c09ec20 0x7fbb5c13ec00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 msgr2=0x7fbb5c141150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> v2:172.21.15.98:3300/0 conn(0x7fbb5c13a160 0x7fbb5c141150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 >> v1:172.21.15.190:6789/0 conn(0x7fbb5c139670 legacy=0x7fbb5c139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient: _add_conns ranks=[2,7,3,0,4,6,5,8] 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): picked mon.noname-b con 0x7fbb480037b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): picked mon.noname-i con 0x7fbb480062f0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 0x7fbb4800b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): picked mon.noname-g con 0x7fbb48008e50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:08.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5c09def0 con 0x7fbb480037b0 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5c09e1d0 con 0x7fbb48008e50 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbb4800baa0 con 0x7fbb480062f0 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb597fa640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 msgr2=0x7fbb4800b2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 0x7fbb4800b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb480037b0 auth_method 0 2024-09-11T02:59:08.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb480062f0 auth_method 0 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:08.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:08.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.342+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb480037b0 auth_method 0 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb480062f0 auth_method 0 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:08.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 msgr2=0x7fbb4800b2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 0x7fbb4800b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:08.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.546+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T02:59:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 msgr2=0x7fbb4800b2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 0x7fbb4800b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:59:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:08.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb480037b0 auth_method 0 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb480062f0 auth_method 0 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T02:59:08.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:59:08.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:08.950+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T02:59:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb480037b0 auth_method 0 2024-09-11T02:59:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:09.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb480062f0 auth_method 0 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 msgr2=0x7fbb4800b2e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 0x7fbb4800b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T02:59:09.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T02:59:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:09.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:09.754+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T02:59:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): tick 2024-09-11T02:59:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): _check_auth_tickets 2024-09-11T02:59:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 monclient(hunting): continuing hunt 2024-09-11T02:59:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T02:59:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:11.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbb480062f0 0x7fbb48008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 msgr2=0x7fbb4800b2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbb48008e50 0x7fbb4800b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 msgr2=0x7fbb48005c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb480037b0 0x7fbb48005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 reap_dead start 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient: _add_conns ranks=[0,8,6,4,5] 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 reap_dead start 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb48003bf0 addr v1:172.21.15.98:6789/0 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbb4800a9f0 0x7fbb48008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): picked mon.noname-e con 0x7fbb4800a9f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb4800b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): picked mon.noname-d con 0x7fbb4800add0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:59:11.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5c09e4b0 con 0x7fbb48003bf0 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbb5c09def0 con 0x7fbb4800add0 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 1 -- 172.21.15.98:0/4247647401 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbb5c09e1d0 con 0x7fbb4800a9f0 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb597fa640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb4800b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbb4800a9f0 0x7fbb48008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request con 0x7fbb4800add0 auth_method 0 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb4800a9f0 auth_method 0 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient(hunting): handle_auth_done global_id 44457 payload 274 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbb4800a9f0 msgr2=0x7fbb48008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbb4800a9f0 0x7fbb48008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 >> v1:172.21.15.98:6789/0 conn(0x7fbb48003bf0 legacy=0x7fbb480092d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 monclient: found mon.noname-d 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5bfff640 1 -- 172.21.15.98:0/4247647401 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4800f2f0 con 0x7fbb4800add0 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5a7fc640 1 -- 172.21.15.98:0/4247647401 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617361478 0 0) 0x7fbb5c09e4b0 con 0x7fbb48003bf0 2024-09-11T02:59:11.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.342+0000 7fbb5a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:59:11.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.346+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb4800b260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c09cae0 tx=0x7fbb5401d080 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7d1fa708dda1ac31 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.346+0000 7fbb5a7fc640 1 -- 172.21.15.98:0/4247647401 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbb540ab070 con 0x7fbb4800add0 2024-09-11T02:59:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.346+0000 7fbb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T02:59:11.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: dump: 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.352 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 1 -- 172.21.15.98:0/4247647401 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb540a5e00 con 0x7fbb4800add0 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 1 -- 172.21.15.98:0/4247647401 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbb540af870 con 0x7fbb4800add0 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 5 monclient: authenticate success, global_id 44457 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 4 set_mon_vals no callback set 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb5a7fc640 10 monclient: dump: 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.354 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.355 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.355 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient: shutdown 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 msgr2=0x7fbb4800b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb4800b260 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c09cae0 tx=0x7fbb5401d080 comp rx=0 tx=0).stop 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- 172.21.15.98:0/4247647401 shutdown_connections 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb4800b260 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 --2- 172.21.15.98:0/4247647401 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbb4800a9f0 0x7fbb48008780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- 172.21.15.98:0/4247647401 >> 172.21.15.98:0/4247647401 conn(0x7fbb5c0a4890 msgr2=0x7fbb5c0a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- 172.21.15.98:0/4247647401 shutdown_connections 2024-09-11T02:59:11.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- 172.21.15.98:0/4247647401 wait complete. 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient: build_initial_monmap 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient: monmap: 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:11.362 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 librados: starting msgr at 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 librados: starting objecter 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 Processor -- start 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- start start 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 librados: setting wanted keys 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 librados: calling monclient init 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient: init 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:11.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient: _add_conns ranks=[4,2,6,3,7,5,8,1,0] 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbb48003bf0 0x7fbb5c129380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): picked mon.noname-c con 0x7fbb48003bf0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb4800a9f0 0x7fbb5c1298c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): picked mon.noname-b con 0x7fbb4800a9f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb5c129e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): picked mon.noname-d con 0x7fbb4800add0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5c139120 con 0x7fbb4800a9f0 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbb5c138fa0 con 0x7fbb48003bf0 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fbb5c1392a0 con 0x7fbb4800add0 2024-09-11T02:59:11.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.350+0000 7fbb61b87640 10 monclient(hunting): authenticate will time out at 1505.235838s 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbb48003bf0 0x7fbb5c129380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb4800a9f0 0x7fbb5c1298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb5c129e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb5c129e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:50068/0 (socket says 172.21.15.98:50068) 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 1 -- 172.21.15.98:0/1485719297 learned_addr learned my addr 172.21.15.98:0/1485719297 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbb48003bf0 0x7fbb5c129380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34890/0 (socket says 172.21.15.98:34890) 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request con 0x7fbb4800add0 auth_method 0 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb4800a9f0 0x7fbb5c1298c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57154/0 (socket says 172.21.15.98:57154) 2024-09-11T02:59:11.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request con 0x7fbb48003bf0 auth_method 0 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 10 monclient(hunting): get_auth_request con 0x7fbb4800a9f0 auth_method 0 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient(hunting): handle_auth_done global_id 24124 payload 824 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 msgr2=0x7fbb5c129e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb5c129e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb4800a9f0 msgr2=0x7fbb5c1298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb4800a9f0 0x7fbb5c1298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 monclient: found mon.noname-c 2024-09-11T02:59:11.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 -- 172.21.15.98:0/1485719297 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5c12d1a0 con 0x7fbb48003bf0 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5affd640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb4800a9f0 0x7fbb5c1298c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5bfff640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb5c129e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb5b7fe640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbb48003bf0 0x7fbb5c129380 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c012570 tx=0x7fbb4c012970 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 client.?.objecter ms_handle_connect 0x7fbb48003bf0 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 1 -- 172.21.15.98:0/1485719297 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbb4c019070 con 0x7fbb48003bf0 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:59:11.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: dump: 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.369 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: _finish_auth 0 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: _check_auth_tickets 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 1 -- 172.21.15.98:0/1485719297 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4c0052e0 con 0x7fbb48003bf0 2024-09-11T02:59:11.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 1 -- 172.21.15.98:0/1485719297 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbb4c015440 con 0x7fbb48003bf0 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 monclient: dump: 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.371 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb597fa640 4 set_mon_vals no callback set 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 5 monclient: authenticate success, global_id 24124 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 monclient: _renew_subs 2024-09-11T02:59:11.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5c12d650 con 0x7fbb48003bf0 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 client.24124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 monclient: _renew_subs 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5c0a30e0 con 0x7fbb48003bf0 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 1 librados: init done 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 1 -- 172.21.15.98:0/1485719297 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbb4c003600 con 0x7fbb48003bf0 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5c13cbf0 con 0x7fbb48003bf0 2024-09-11T02:59:11.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbb48040100 0x7fbb480425c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 1 -- 172.21.15.98:0/1485719297 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(364..364 src has 1..364) ==== 5817+0+0 (secure 0 0 0) 0x7fbb4c010040 con 0x7fbb48003bf0 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 10 client.24124.objecter ms_dispatch 0x7fbb5c000ba0 osd_map(364..364 src has 1..364) 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 3 client.24124.objecter handle_osd_map got epochs [364,364] > 0 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 3 client.24124.objecter handle_osd_map decoding full epoch 364 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.354+0000 7fbb43fff640 20 client.24124.objecter dump_active .. 0 homeless 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.358+0000 7fbb5affd640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbb48040100 0x7fbb480425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.358+0000 7fbb5affd640 10 monclient: get_auth_request con 0x7fbb48040100 auth_method 0 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.358+0000 7fbb5affd640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbb48040100 0x7fbb480425c0 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c12b4b0 tx=0x7fbb50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.358+0000 7fbb43fff640 10 client.24124.objecter ms_handle_connect 0x7fbb48040100 2024-09-11T02:59:11.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.362+0000 7fbb43fff640 1 -- 172.21.15.98:0/1485719297 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbb4c023030 con 0x7fbb48003bf0 2024-09-11T02:59:11.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.362+0000 7fbb43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:11.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.362+0000 7fbb43fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T02:59:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T02:59:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbb5c12c330 con 0x7fbb48003bf0 2024-09-11T02:59:11.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb43fff640 1 -- 172.21.15.98:0/1485719297 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7fbb4c01ce30 con 0x7fbb48003bf0 2024-09-11T02:59:11.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T02:59:11.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb43fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:11.601 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":72,"quorum":[0,1,2,4,6,7,8],"quorum_names":["a","f","b","c","d","i","e"],"quorum_leader_name":"e","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T02:59:11.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 10 librados: watch_flush enter 2024-09-11T02:59:11.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 10 librados: watch_flush exit 2024-09-11T02:59:11.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 20 client.24124.objecter shutdown clearing up homeless session... 2024-09-11T02:59:11.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.598+0000 7fbb61b87640 10 client.24124.objecter successfully canceled tick 2024-09-11T02:59:11.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbb48040100 msgr2=0x7fbb480425c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbb48040100 0x7fbb480425c0 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c12b4b0 tx=0x7fbb50008040 comp rx=0 tx=0).stop 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 10 monclient: shutdown 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbb48003bf0 msgr2=0x7fbb5c129380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbb48003bf0 0x7fbb5c129380 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c012570 tx=0x7fbb4c012970 comp rx=0 tx=0).stop 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 shutdown_connections 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fbb4800add0 0x7fbb5c129e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbb48040100 0x7fbb480425c0 unknown :-1 s=CLOSED pgs=3558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fbb4800a9f0 0x7fbb5c1298c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 --2- 172.21.15.98:0/1485719297 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbb48003bf0 0x7fbb5c129380 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 >> 172.21.15.98:0/1485719297 conn(0x7fbb5c0a4890 msgr2=0x7fbb5c0a64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:11.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 shutdown_connections 2024-09-11T02:59:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 -- 172.21.15.98:0/1485719297 wait complete. 2024-09-11T02:59:11.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.602+0000 7fbb61b87640 1 librados: shutdown 2024-09-11T02:59:11.620 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-09-11T02:59:11.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-09-11T02:59:11.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T02:59:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.734+0000 7fc8b6e26640 10 monclient: get_monmap_and_config 2024-09-11T02:59:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.734+0000 7fc8b6e26640 10 monclient: build_initial_monmap 2024-09-11T02:59:11.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.734+0000 7fc8b6e26640 10 monclient: monmap: 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:11.740637+0000 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:11.740637+0000 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:11.739 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:11.740 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:11.740 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:11.740 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:11.740 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:11.740 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:11.740 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 1 Processor -- start 2024-09-11T02:59:11.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 1 -- start start 2024-09-11T02:59:11.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient: init 2024-09-11T02:59:11.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:11.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient: _add_conns ranks=[1,9,3,6,10,7,8,4,5,2,0] 2024-09-11T02:59:11.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc8b00a58c0 addr v1:172.21.15.190:6789/0 2024-09-11T02:59:11.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc8b0071db0 0x7fc8b00a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b0071db0 addr v2:172.21.15.98:3300/0 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8b013c920 0x7fc8b013ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient(hunting): picked mon.noname-g con 0x7fc8b013c920 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8b0073b70 con 0x7fc8b00a58c0 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc8b0073640 con 0x7fc8b0071db0 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b0073890 con 0x7fc8b013c920 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient(hunting): authenticate will time out at 1505.620638s 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b53c3640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8b013c920 msgr2=0x7fc8b013ed10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc8b0071db0 0x7fc8b00a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b53c3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8b013c920 0x7fc8b013ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc8b0071db0 0x7fc8b00a63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55202/0 (socket says 172.21.15.98:55202) 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 -- 172.21.15.98:0/1225422592 learned_addr learned my addr 172.21.15.98:0/1225422592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): get_auth_request con 0x7fc8b0071db0 auth_method 0 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient(hunting): handle_auth_done global_id 44586 payload 274 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 -- 172.21.15.98:0/1225422592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8b013c920 msgr2=0x7fc8b013ed10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 --2- 172.21.15.98:0/1225422592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8b013c920 0x7fc8b013ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 -- 172.21.15.98:0/1225422592 >> v1:172.21.15.190:6789/0 conn(0x7fc8b00a58c0 legacy=0x7fc8b00a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 monclient: found mon.noname-a 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 -- 172.21.15.98:0/1225422592 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b00a7410 con 0x7fc8b0071db0 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8affff640 1 --2- 172.21.15.98:0/1225422592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b0071db0 0x7fc8b00a63b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098a0 tx=0x7fc89c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d3e9b501dadaf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 1 -- 172.21.15.98:0/1225422592 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc89c0b0070 con 0x7fc8b0071db0 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: dump: 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.747 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.748 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: _finish_auth 0 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: _check_auth_tickets 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 5 monclient: authenticate success, global_id 44586 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 1 -- 172.21.15.98:0/1225422592 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc89c0aa810 con 0x7fc8b0071db0 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 1 -- 172.21.15.98:0/1225422592 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc89c0a9300 con 0x7fc8b0071db0 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 4 set_mon_vals no callback set 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8af7fe640 10 monclient: dump: 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.738+0000 7fc8b6e26640 10 monclient: shutdown 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/1225422592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b0071db0 msgr2=0x7fc8b00a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 --2- 172.21.15.98:0/1225422592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b0071db0 0x7fc8b00a63b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098a0 tx=0x7fc89c01c420 comp rx=0 tx=0).stop 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/1225422592 shutdown_connections 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 --2- 172.21.15.98:0/1225422592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc8b013c920 0x7fc8b013ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 --2- 172.21.15.98:0/1225422592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b0071db0 0x7fc8b00a63b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/1225422592 >> 172.21.15.98:0/1225422592 conn(0x7fc8b00aa4b0 msgr2=0x7fc8b00a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/1225422592 shutdown_connections 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/1225422592 wait complete. 2024-09-11T02:59:11.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: build_initial_monmap 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: monmap: 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:11.752 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 librados: starting msgr at 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 librados: starting objecter 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 Processor -- start 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- start start 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 librados: setting wanted keys 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 librados: calling monclient init 2024-09-11T02:59:11.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: init 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: _add_conns ranks=[7,4,6,1,5,0,3,2,8] 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc8b0071db0 0x7fc8b0122d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): picked mon.noname-i con 0x7fc8b0071db0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc8b00a58c0 0x7fc8b0124c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b00a58c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8b013c920 0x7fc8b013baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): picked mon.noname-d con 0x7fc8b013c920 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc8b0141c80 con 0x7fc8b00a58c0 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc8b0072230 con 0x7fc8b013c920 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc8b0141e00 con 0x7fc8b0071db0 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): authenticate will time out at 1505.625808s 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc8b00a58c0 0x7fc8b0124c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc8b00a58c0 0x7fc8b0124c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34906/0 (socket says 172.21.15.98:34906) 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 -- 172.21.15.98:0/4170460710 learned_addr learned my addr 172.21.15.98:0/4170460710 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8b013c920 0x7fc8b013baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): get_auth_request con 0x7fc8b00a58c0 auth_method 0 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 10 monclient(hunting): get_auth_request con 0x7fc8b013c920 auth_method 0 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b53c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient(hunting): handle_auth_done global_id 24133 payload 824 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc8b0071db0 msgr2=0x7fc8b0122d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc8b0071db0 0x7fc8b0122d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8b013c920 msgr2=0x7fc8b013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc8b013c920 0x7fc8b013baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 monclient: found mon.noname-c 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 -- 172.21.15.98:0/4170460710 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b0124120 con 0x7fc8b00a58c0 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8affff640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc8b00a58c0 0x7fc8b0124c80 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fc89c01a050 tx=0x7fc89c0191c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 client.?.objecter ms_handle_connect 0x7fc8b00a58c0 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc89c0b9070 con 0x7fc8b00a58c0 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: dump: 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: _finish_auth 0 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: _check_auth_tickets 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc89c003620 con 0x7fc8b00a58c0 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 5 monclient: authenticate success, global_id 24133 2024-09-11T02:59:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: _renew_subs 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b0123c80 con 0x7fc8b00a58c0 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc89c0b0040 con 0x7fc8b00a58c0 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8aeffd640 4 set_mon_vals no callback set 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 monclient: dump: 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 client.24133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: _renew_subs 2024-09-11T02:59:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b013c030 con 0x7fc8b00a58c0 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 librados: init done 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01ceef0 0x7fc8b01cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b00a3720 con 0x7fc8b01ceef0 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01ceef0 0x7fc8b01cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient: get_auth_request con 0x7fc8b01ceef0 auth_method 0 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc89c004180 con 0x7fc8b00a58c0 2024-09-11T02:59:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_done global_id 44595 payload 824 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8b4bc2640 1 -- 172.21.15.98:0/4170460710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8b00a31f0 con 0x7fc8b01ceef0 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc89403df40 0x7fc894040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(364..364 src has 1..364) ==== 5817+0+0 (secure 0 0 0) 0x7fc89c0e8a20 con 0x7fc8b00a58c0 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 10 client.24133.objecter ms_dispatch 0x7fc8b0000ba0 osd_map(364..364 src has 1..364) 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 3 client.24133.objecter handle_osd_map got epochs [364,364] > 0 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.742+0000 7fc8ad7fa640 3 client.24133.objecter handle_osd_map decoding full epoch 364 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8b4bc2640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01ceef0 0x7fc8b01cf2d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400c310 tx=0x7fc8a400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 20 client.24133.objecter dump_active .. 0 homeless 2024-09-11T02:59:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 client.24133.objecter ms_handle_connect 0x7fc8b01ceef0 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 client.24133.objecter resend_mon_ops 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8a4017070 con 0x7fc8b01ceef0 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 monclient: dump: 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8b53c3640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc89403df40 0x7fc894040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8b53c3640 10 monclient: get_auth_request con 0x7fc89403df40 auth_method 0 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc8a40050d0 con 0x7fc8b01ceef0 2024-09-11T02:59:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01ceef0 msgr2=0x7fc8b01cf2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01ceef0 0x7fc8b01cf2d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400c310 tx=0x7fc8a400c7d0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01ceef0 msgr2=0x7fc8b01cf2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01ceef0 0x7fc8b01cf2d0 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400c310 tx=0x7fc8a400c7d0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8b53c3640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc89403df40 0x7fc894040400 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8003cc0 tx=0x7fc8a80095f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.746+0000 7fc8ad7fa640 10 client.24133.objecter ms_handle_connect 0x7fc89403df40 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01cf810 0x7fc8b009ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b006ca10 con 0x7fc8b01cf810 2024-09-11T02:59:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01cf810 0x7fc8b009ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient: get_auth_request con 0x7fc8b01cf810 auth_method 0 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 10 monclient(hunting): handle_auth_done global_id 44604 payload 824 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 1 -- 172.21.15.98:0/4170460710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc8b013c260 con 0x7fc8b01cf810 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 10 client.24133.objecter ms_handle_connect 0x7fc8b01cf810 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 10 client.24133.objecter resend_mon_ops 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b4bc2640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01cf810 0x7fc8b009ea60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4014000 tx=0x7fc8a4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8a4017030 con 0x7fc8b01cf810 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 10 monclient: dump: 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.775 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7fc8a4011be0 con 0x7fc8b01cf810 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01cf810 msgr2=0x7fc8b009ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01cf810 0x7fc8b009ea60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4014000 tx=0x7fc8a4000ee0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01cf810 msgr2=0x7fc8b009ea60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8ad7fa640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8b01cf810 0x7fc8b009ea60 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4014000 tx=0x7fc8a4000ee0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 72, 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 178189, 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:59:11.777 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:11.778 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:59:11.779 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.780 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:59:11.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:11.782 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:59:11.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:59:11.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:59:11.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:59:11.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.791 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.793 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:59:11.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:59:11.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.798 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.799 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.800 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:59:11.801 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:59:11.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 10 librados: watch_flush enter 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 10 librados: watch_flush exit 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 20 client.24133.objecter shutdown clearing up homeless session... 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 10 client.24133.objecter successfully canceled tick 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc89403df40 msgr2=0x7fc894040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.750+0000 7fc8b6e26640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc89403df40 0x7fc894040400 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fc8a8003cc0 tx=0x7fc8a80095f0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 10 monclient: shutdown 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc8b00a58c0 msgr2=0x7fc8b0124c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 1 --2- 172.21.15.98:0/4170460710 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc8b00a58c0 0x7fc8b0124c80 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fc89c01a050 tx=0x7fc89c0191c0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b53c3640 1 -- 172.21.15.98:0/4170460710 reap_dead start 2024-09-11T02:59:11.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b53c3640 1 -- 172.21.15.98:0/4170460710 reap_dead start 2024-09-11T02:59:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 shutdown_connections 2024-09-11T02:59:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 >> 172.21.15.98:0/4170460710 conn(0x7fc8b00aa4b0 msgr2=0x7fc8b00a1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 shutdown_connections 2024-09-11T02:59:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 1 -- 172.21.15.98:0/4170460710 wait complete. 2024-09-11T02:59:11.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.754+0000 7fc8b6e26640 1 librados: shutdown 2024-09-11T02:59:11.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:59:11.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-09-11T02:59:11.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T02:59:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient: get_monmap_and_config 2024-09-11T02:59:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient: build_initial_monmap 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient: monmap: 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:11.925572+0000 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:11.925572+0000 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:11.924 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:11.925 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 Processor -- start 2024-09-11T02:59:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 -- start start 2024-09-11T02:59:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient: init 2024-09-11T02:59:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient: _add_conns ranks=[5,2,7,4,9,8,0,10,1,6,3] 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa504139b80 0x7fa504139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): picked mon.noname-h con 0x7fa504139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50406fe50 0x7fa5040a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): picked mon.noname-b con 0x7fa50406fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa504072640 0x7fa5040a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): picked mon.noname-i con 0x7fa504072640 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa5040719c0 con 0x7fa50406fe50 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa504071c10 con 0x7fa504139b80 2024-09-11T02:59:11.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa5040702d0 con 0x7fa504072640 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa50b209640 10 monclient(hunting): authenticate will time out at 1505.805929s 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa508fa5640 1 -- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa504139b80 msgr2=0x7fa504139f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3302/0 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50406fe50 0x7fa5040a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa508fa5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa504139b80 0x7fa504139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50406fe50 0x7fa5040a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57168/0 (socket says 172.21.15.98:57168) 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 -- 172.21.15.98:0/1804216142 learned_addr learned my addr 172.21.15.98:0/1804216142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): get_auth_request con 0x7fa50406fe50 auth_method 0 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa5097a6640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa504072640 0x7fa5040a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa5097a6640 10 monclient(hunting): get_auth_request con 0x7fa504072640 auth_method 0 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa5097a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa5097a6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa5097a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient(hunting): handle_auth_done global_id 64253 payload 274 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 -- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa504072640 msgr2=0x7fa5040a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa504072640 0x7fa5040a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 -- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa504139b80 msgr2=0x7fa504139f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa504139b80 0x7fa504139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 monclient: found mon.noname-b 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa503fff640 1 -- 172.21.15.98:0/1804216142 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa504071370 con 0x7fa50406fe50 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.922+0000 7fa5097a6640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa504072640 0x7fa5040a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa503fff640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50406fe50 0x7fa5040a7a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0098d0 tx=0x7fa4ec01c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aad6d1a24f613b41 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 1 -- 172.21.15.98:0/1804216142 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4ec0b0070 con 0x7fa50406fe50 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: dump: 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: _finish_auth 0 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: _check_auth_tickets 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 5 monclient: authenticate success, global_id 64253 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 1 -- 172.21.15.98:0/1804216142 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4ec0aa810 con 0x7fa50406fe50 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 4 set_mon_vals no callback set 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 1 -- 172.21.15.98:0/1804216142 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4ec0a9300 con 0x7fa50406fe50 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa5037fe640 10 monclient: dump: 2024-09-11T02:59:11.941 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.942 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient: shutdown 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- 172.21.15.98:0/1804216142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50406fe50 msgr2=0x7fa5040a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50406fe50 0x7fa5040a7a10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0098d0 tx=0x7fa4ec01c460 comp rx=0 tx=0).stop 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- 172.21.15.98:0/1804216142 shutdown_connections 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa504072640 0x7fa5040a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50406fe50 0x7fa5040a7a10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 --2- 172.21.15.98:0/1804216142 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa504139b80 0x7fa504139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- 172.21.15.98:0/1804216142 >> 172.21.15.98:0/1804216142 conn(0x7fa5040ac730 msgr2=0x7fa5040a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- 172.21.15.98:0/1804216142 shutdown_connections 2024-09-11T02:59:11.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- 172.21.15.98:0/1804216142 wait complete. 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient: build_initial_monmap 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient: monmap: 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:11.944 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 librados: starting msgr at 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 librados: starting objecter 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 Processor -- start 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- start start 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 librados: setting wanted keys 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 librados: calling monclient init 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient: init 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:11.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient: _add_conns ranks=[0,8,2,6,7,3,1,5,4] 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50406fe50 0x7fa5041337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): picked mon.noname-a con 0x7fa50406fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa504072640 0x7fa504133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): picked mon.noname-e con 0x7fa504072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa504139b80 0x7fa504134270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): picked mon.noname-b con 0x7fa504139b80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5041435c0 con 0x7fa50406fe50 2024-09-11T02:59:11.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa504143440 con 0x7fa504139b80 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa504143740 con 0x7fa504072640 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.926+0000 7fa50b209640 10 monclient(hunting): authenticate will time out at 1505.811966s 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa504072640 0x7fa504133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50406fe50 0x7fa5041337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa504139b80 0x7fa504134270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa504072640 0x7fa504133d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:48688/0 (socket says 172.21.15.98:48688) 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient(hunting): get_auth_request con 0x7fa504139b80 auth_method 0 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 1 -- 172.21.15.98:0/1614387783 learned_addr learned my addr 172.21.15.98:0/1614387783 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50406fe50 0x7fa5041337f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55234/0 (socket says 172.21.15.98:55234) 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): get_auth_request con 0x7fa504072640 auth_method 0 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): get_auth_request con 0x7fa50406fe50 auth_method 0 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient(hunting): handle_auth_done global_id 44613 payload 824 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa504072640 msgr2=0x7fa504133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa504072640 0x7fa504133d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa504139b80 msgr2=0x7fa504134270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa504139b80 0x7fa504134270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient: _finish_hunting 0 2024-09-11T02:59:11.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 monclient: found mon.noname-a 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 -- 172.21.15.98:0/1614387783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50413d140 con 0x7fa50406fe50 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa504139b80 0x7fa504134270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa504072640 0x7fa504133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa508fa5640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50406fe50 0x7fa5041337f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00be00 tx=0x7fa4fc009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 client.?.objecter ms_handle_connect 0x7fa50406fe50 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4fc00ccf0 con 0x7fa50406fe50 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: dump: 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.950 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.951 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: _finish_auth 0 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: _check_auth_tickets 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4fc010070 con 0x7fa50406fe50 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 5 monclient: authenticate success, global_id 44613 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 10 monclient: _renew_subs 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5040aa100 con 0x7fa50406fe50 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4fc015d50 con 0x7fa50406fe50 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: dump: 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.953 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa502ffd640 4 set_mon_vals no callback set 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 10 client.44613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 10 monclient: _renew_subs 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50413ce60 con 0x7fa50406fe50 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 1 librados: init done 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413a6b0 0x7fa50413aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa50406ca10 con 0x7fa50413a6b0 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413a6b0 0x7fa50413aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient: get_auth_request con 0x7fa50413a6b0 auth_method 0 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa4fc01d030 con 0x7fa50406fe50 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 10 monclient(hunting): handle_auth_done global_id 64271 payload 824 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 1 -- 172.21.15.98:0/1614387783 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5040aa630 con 0x7fa50413a6b0 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa4e403df40 0x7fa4e4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa503fff640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413a6b0 0x7fa50413aa90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01c290 tx=0x7fa4ec019200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 client.44613.objecter ms_handle_connect 0x7fa50413a6b0 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 client.44613.objecter resend_mon_ops 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 6189+0+0 (secure 0 0 0) 0x7fa4fc04f730 con 0x7fa50406fe50 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 client.44613.objecter ms_dispatch 0x7fa504000ba0 osd_map(365..365 src has 1..365) 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 3 client.44613.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 3 client.44613.objecter handle_osd_map decoding full epoch 365 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 20 client.44613.objecter dump_active .. 0 homeless 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4ec0b0040 con 0x7fa50413a6b0 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: dump: 2024-09-11T02:59:11.956 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.957 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa4e403df40 0x7fa4e4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 10 monclient: get_auth_request con 0x7fa4e403df40 auth_method 0 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa4ec0aa810 con 0x7fa50413a6b0 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413a6b0 msgr2=0x7fa50413aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413a6b0 0x7fa50413aa90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01c290 tx=0x7fa4ec019200 comp rx=0 tx=0).stop 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413a6b0 msgr2=0x7fa50413aa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413a6b0 0x7fa50413aa90 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01c290 tx=0x7fa4ec019200 comp rx=0 tx=0).stop 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5097a6640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa4e403df40 0x7fa4e4040400 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7fa50413db90 tx=0x7fa4f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.930+0000 7fa5017fa640 10 client.44613.objecter ms_handle_connect 0x7fa4e403df40 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413afd0 0x7fa5040a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa50406e870 con 0x7fa50413afd0 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413afd0 0x7fa5040a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient: get_auth_request con 0x7fa50413afd0 auth_method 0 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:11.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 10 monclient(hunting): handle_auth_done global_id 64280 payload 824 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 1 -- 172.21.15.98:0/1614387783 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa504139f60 con 0x7fa50413afd0 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 10 client.44613.objecter ms_handle_connect 0x7fa50413afd0 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 10 client.44613.objecter resend_mon_ops 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa503fff640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413afd0 0x7fa5040a0e60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01c4a0 tx=0x7fa4ec0a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4ec0c6030 con 0x7fa50413afd0 2024-09-11T02:59:11.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 10 monclient: dump: 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:11.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9244 (secure 0 0 0) 0x7fa4ec0c2290 con 0x7fa50413afd0 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:59:11.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413afd0 msgr2=0x7fa5040a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413afd0 0x7fa5040a0e60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01c4a0 tx=0x7fa4ec0a8000 comp rx=0 tx=0).stop 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413afd0 msgr2=0x7fa5040a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5017fa640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa50413afd0 0x7fa5040a0e60 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec01c4a0 tx=0x7fa4ec0a8000 comp rx=0 tx=0).stop 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:59:11.965 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 72, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 44131, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:59:11.966 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.967 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:59:11.969 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:11.970 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.972 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.974 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:59:11.977 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:59:11.980 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:59:11.983 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:59:11.985 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.986 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:11.987 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:11.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:11.988 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:59:11.988 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:59:11.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 10 librados: watch_flush enter 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 10 librados: watch_flush exit 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 20 client.44613.objecter shutdown clearing up homeless session... 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 10 client.44613.objecter successfully canceled tick 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa4e403df40 msgr2=0x7fa4e4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa4e403df40 0x7fa4e4040400 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7fa50413db90 tx=0x7fa4f4008040 comp rx=0 tx=0).stop 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 10 monclient: shutdown 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50406fe50 msgr2=0x7fa5041337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 --2- 172.21.15.98:0/1614387783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa50406fe50 0x7fa5041337f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00be00 tx=0x7fa4fc009f70 comp rx=0 tx=0).stop 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5097a6640 1 -- 172.21.15.98:0/1614387783 reap_dead start 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa5097a6640 1 -- 172.21.15.98:0/1614387783 reap_dead start 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 shutdown_connections 2024-09-11T02:59:11.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 >> 172.21.15.98:0/1614387783 conn(0x7fa5040ac730 msgr2=0x7fa5040aa860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:11.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 shutdown_connections 2024-09-11T02:59:11.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 -- 172.21.15.98:0/1614387783 wait complete. 2024-09-11T02:59:11.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:11.938+0000 7fa50b209640 1 librados: shutdown 2024-09-11T02:59:11.991 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:59:11.991 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-09-11T02:59:11.991 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T02:59:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient: get_monmap_and_config 2024-09-11T02:59:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient: monmap: 2024-09-11T02:59:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.116 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:12.117692+0000 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:12.117692+0000 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.117 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:12.118 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 1 Processor -- start 2024-09-11T02:59:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 1 -- start start 2024-09-11T02:59:12.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient: init 2024-09-11T02:59:12.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient: _add_conns ranks=[5,1,10,6,9,8,3,4,2,7,0] 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a9c139b80 0x7f3a9c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient(hunting): picked mon.noname-h con 0x7f3a9c139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3a9c06fe50 addr v1:172.21.15.190:6789/0 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a9c072640 0x7f3a9c0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient(hunting): picked mon.noname-f con 0x7f3a9c072640 addr v2:172.21.15.190:3300/0 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a9c071370 con 0x7f3a9c06fe50 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3a9c0719c0 con 0x7f3a9c072640 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3a9c071c10 con 0x7f3a9c139b80 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3aa20df640 10 monclient(hunting): authenticate will time out at 1505.997988s 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9b7fe640 1 -- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a9c139b80 msgr2=0x7f3a9c139f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3302/0 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9b7fe640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a9c139b80 0x7f3a9c139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a9c072640 0x7f3a9c0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3a9c072640 0x7f3a9c0a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:41928/0 (socket says 172.21.15.98:41928) 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9affd640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f3a9c06fe50 0x7f3a9c0a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:38352/0 (socket says 172.21.15.98:38352) 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 -- 172.21.15.98:0/1301443343 learned_addr learned my addr 172.21.15.98:0/1301443343 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): get_auth_request con 0x7f3a9c072640 auth_method 0 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient(hunting): handle_auth_done global_id 25462 payload 274 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 -- 172.21.15.98:0/1301443343 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a9c139b80 msgr2=0x7f3a9c139f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 --2- 172.21.15.98:0/1301443343 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a9c139b80 0x7f3a9c139f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 -- 172.21.15.98:0/1301443343 >> v1:172.21.15.190:6789/0 conn(0x7f3a9c06fe50 legacy=0x7f3a9c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 monclient: found mon.noname-f 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.114+0000 7f3a9bfff640 1 -- 172.21.15.98:0/1301443343 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c0a9600 con 0x7f3a9c072640 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9bfff640 1 --2- 172.21.15.98:0/1301443343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a9c072640 0x7f3a9c0a8120 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009e50 tx=0x7f3a9401c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7338482d1b35a270 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 1 -- 172.21.15.98:0/1301443343 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a940090d0 con 0x7f3a9c072640 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T02:59:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: dump: 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 1 -- 172.21.15.98:0/1301443343 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a940a5b50 con 0x7f3a9c072640 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 5 monclient: authenticate success, global_id 25462 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 1 -- 172.21.15.98:0/1301443343 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a940a5d40 con 0x7f3a9c072640 2024-09-11T02:59:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 4 set_mon_vals no callback set 2024-09-11T02:59:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:59:12.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3a9a7fc640 10 monclient: dump: 2024-09-11T02:59:12.128 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.129 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.130 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.130 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient: shutdown 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- 172.21.15.98:0/1301443343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a9c072640 msgr2=0x7f3a9c0a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1301443343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a9c072640 0x7f3a9c0a8120 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f3a94009e50 tx=0x7f3a9401c910 comp rx=0 tx=0).stop 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- 172.21.15.98:0/1301443343 shutdown_connections 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1301443343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a9c072640 0x7f3a9c0a8120 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1301443343 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3a9c139b80 0x7f3a9c139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- 172.21.15.98:0/1301443343 >> 172.21.15.98:0/1301443343 conn(0x7f3a9c0ac730 msgr2=0x7f3a9c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- 172.21.15.98:0/1301443343 shutdown_connections 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- 172.21.15.98:0/1301443343 wait complete. 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient: monmap: 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.138 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.139 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 librados: starting msgr at 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 librados: starting objecter 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 Processor -- start 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- start start 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 librados: setting wanted keys 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 librados: calling monclient init 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient: init 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient: _add_conns ranks=[0,2,8,6,5,4,7,1,3] 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a9c06fe50 0x7f3a9c133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): picked mon.noname-a con 0x7f3a9c06fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T02:59:12.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9c072640 0x7f3a9c133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): picked mon.noname-b con 0x7f3a9c072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a9c139b80 0x7f3a9c1341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): picked mon.noname-e con 0x7f3a9c139b80 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3a9c144450 con 0x7f3a9c06fe50 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9c0a8870 con 0x7f3a9c072640 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3a9c1445d0 con 0x7f3a9c139b80 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.118+0000 7f3aa20df640 10 monclient(hunting): authenticate will time out at 1506.003935s 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a9c06fe50 0x7f3a9c133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9c072640 0x7f3a9c133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a9c06fe50 0x7f3a9c133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:55236/0 (socket says 172.21.15.98:55236) 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 -- 172.21.15.98:0/1563517416 learned_addr learned my addr 172.21.15.98:0/1563517416 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): get_auth_request con 0x7f3a9c06fe50 auth_method 0 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): get_auth_request con 0x7f3a9c072640 auth_method 0 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a9c139b80 0x7f3a9c1341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient(hunting): get_auth_request con 0x7f3a9c139b80 auth_method 0 2024-09-11T02:59:12.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient(hunting): handle_auth_done global_id 44622 payload 824 2024-09-11T02:59:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a9c139b80 msgr2=0x7f3a9c1341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a9c139b80 0x7f3a9c1341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9c072640 msgr2=0x7f3a9c133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3a9c072640 0x7f3a9c133ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 monclient: found mon.noname-a 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 -- 172.21.15.98:0/1563517416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9c13cfc0 con 0x7f3a9c06fe50 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3a9c139b80 0x7f3a9c1341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9b7fe640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a9c06fe50 0x7f3a9c133760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00b530 tx=0x7f3a8c00b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 client.?.objecter ms_handle_connect 0x7f3a9c06fe50 2024-09-11T02:59:12.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a8c013020 con 0x7f3a9c06fe50 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: dump: 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.146 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a8c003bd0 con 0x7f3a9c06fe50 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 5 monclient: authenticate success, global_id 44622 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 10 monclient: _renew_subs 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a9c0a8020 con 0x7f3a9c06fe50 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a8c003d80 con 0x7f3a9c06fe50 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T02:59:12.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 10 client.44622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a99ffb640 4 set_mon_vals no callback set 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: dump: 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.149 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 10 monclient: _renew_subs 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9c13cce0 con 0x7f3a9c06fe50 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 1 librados: init done 2024-09-11T02:59:12.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3a8c005070 con 0x7f3a9c06fe50 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c063e00 0x7f3a9c0641e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3a9c13c3f0 con 0x7f3a9c063e00 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c063e00 0x7f3a9c0641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient: get_auth_request con 0x7f3a9c063e00 auth_method 0 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a8803df40 0x7f3a88040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(365..365 src has 1..365) ==== 6189+0+0 (secure 0 0 0) 0x7f3a8c04f630 con 0x7f3a9c06fe50 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 client.44622.objecter ms_dispatch 0x7f3a9c000ba0 osd_map(365..365 src has 1..365) 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 3 client.44622.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 3 client.44622.objecter handle_osd_map decoding full epoch 365 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 10 monclient(hunting): handle_auth_done global_id 24142 payload 824 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 1 -- 172.21.15.98:0/1563517416 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a9c0ab9e0 con 0x7f3a9c063e00 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 20 client.44622.objecter dump_active .. 0 homeless 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 client.44622.objecter ms_handle_connect 0x7f3a9c063e00 2024-09-11T02:59:12.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 client.44622.objecter resend_mon_ops 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9affd640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c063e00 0x7f3a9c0641e0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0a50c0 tx=0x7f3a90001970 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a8803df40 0x7f3a88040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 10 monclient: get_auth_request con 0x7f3a8803df40 auth_method 0 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a900119f0 con 0x7f3a9c063e00 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: dump: 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.153 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.154 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a90018070 con 0x7f3a9c063e00 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c063e00 msgr2=0x7f3a9c0641e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c063e00 0x7f3a9c0641e0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0a50c0 tx=0x7f3a90001970 comp rx=0 tx=0).stop 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c063e00 msgr2=0x7f3a9c0641e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c063e00 0x7f3a9c0641e0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a9bfff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a8803df40 0x7f3a88040400 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7f3a940ae000 tx=0x7f3a940a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.122+0000 7f3a83fff640 10 client.44622.objecter ms_handle_connect 0x7f3a8803df40 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:59:12.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c02d450 0x7f3a9c02d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3a9c06ca10 con 0x7f3a9c02d450 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c02d450 0x7f3a9c02d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient: get_auth_request con 0x7f3a9c02d450 auth_method 0 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 10 monclient(hunting): handle_auth_done global_id 24151 payload 824 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 1 -- 172.21.15.98:0/1563517416 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a9c13c8b0 con 0x7f3a9c02d450 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9affd640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c02d450 0x7f3a9c02d860 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f3a90001ec0 tx=0x7f3a90000ee0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 10 client.44622.objecter ms_handle_connect 0x7f3a9c02d450 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 10 client.44622.objecter resend_mon_ops 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a90013860 con 0x7f3a9c02d450 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 10 monclient: dump: 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.157 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.158 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7f3a900133f0 con 0x7f3a9c02d450 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c02d450 msgr2=0x7f3a9c02d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c02d450 0x7f3a9c02d860 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f3a90001ec0 tx=0x7f3a90000ee0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c02d450 msgr2=0x7f3a9c02d860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a83fff640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3a9c02d450 0x7f3a9c02d860 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 72, 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 567914, 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:59:12.160 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:59:12.161 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.162 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.163 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.164 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.165 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.166 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.168 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.170 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.172 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:59:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:59:12.173 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:59:12.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.177 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:59:12.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:59:12.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:59:12.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 5 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:59:12.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:59:12.186 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 10 librados: watch_flush enter 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 10 librados: watch_flush exit 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 20 client.44622.objecter shutdown clearing up homeless session... 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 10 client.44622.objecter successfully canceled tick 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a8803df40 msgr2=0x7f3a88040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3a8803df40 0x7f3a88040400 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7f3a940ae000 tx=0x7f3a940a7070 comp rx=0 tx=0).stop 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 10 monclient: shutdown 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a9c06fe50 msgr2=0x7f3a9c133760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3a9bfff640 1 -- 172.21.15.98:0/1563517416 reap_dead start 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a9c06fe50 0x7f3a9c133760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00b530 tx=0x7f3a8c00b9f0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 shutdown_connections 2024-09-11T02:59:12.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 --2- 172.21.15.98:0/1563517416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3a9c06fe50 0x7f3a9c133760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 >> 172.21.15.98:0/1563517416 conn(0x7f3a9c0ac730 msgr2=0x7f3a9c0aa170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 shutdown_connections 2024-09-11T02:59:12.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 -- 172.21.15.98:0/1563517416 wait complete. 2024-09-11T02:59:12.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.130+0000 7f3aa20df640 1 librados: shutdown 2024-09-11T02:59:12.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:59:12.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-09-11T02:59:12.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T02:59:12.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient: get_monmap_and_config 2024-09-11T02:59:12.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient: monmap: 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:12.305549+0000 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:12.305549+0000 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:12.304 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:12.305 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 Processor -- start 2024-09-11T02:59:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 -- start start 2024-09-11T02:59:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient: init 2024-09-11T02:59:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient: _add_conns ranks=[8,10,3,6,5,0,9,4,7,2,1] 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f300a3de0 0x7f0f300a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): picked mon.noname-e con 0x7f0f300a3de0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f0f300a41c0 0x7f0f300a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): picked mon.noname-f con 0x7f0f300a41c0 addr v2:172.21.15.190:3300/0 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f3009b240 0x7f0f3009b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): picked mon.noname-g con 0x7f0f3009b240 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f0f300a0170 con 0x7f0f300a41c0 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0f30068fa0 con 0x7f0f3009b240 2024-09-11T02:59:12.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0f30068b40 con 0x7f0f300a3de0 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f36634640 10 monclient(hunting): authenticate will time out at 1506.185770s 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f300a3de0 0x7f0f300a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f34bd1640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f3009b240 msgr2=0x7f0f3009b6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f300a3de0 0x7f0f300a0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:48692/0 (socket says 172.21.15.98:48692) 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 -- 172.21.15.98:0/1844481190 learned_addr learned my addr 172.21.15.98:0/1844481190 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request con 0x7f0f300a3de0 auth_method 0 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f34bd1640 1 --2- 172.21.15.98:0/1844481190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f3009b240 0x7f0f3009b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/1844481190 >> v2:172.21.15.190:3300/0 conn(0x7f0f300a41c0 0x7f0f300a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request con 0x7f0f300a41c0 auth_method 0 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_done global_id 24641 payload 274 2024-09-11T02:59:12.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 -- 172.21.15.98:0/1844481190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f3009b240 msgr2=0x7f0f3009b6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/1844481190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f3009b240 0x7f0f3009b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 -- 172.21.15.98:0/1844481190 >> v2:172.21.15.190:3300/0 conn(0x7f0f300a41c0 msgr2=0x7f0f300a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/1844481190 >> v2:172.21.15.190:3300/0 conn(0x7f0f300a41c0 0x7f0f300a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 monclient: found mon.noname-e 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2ffff640 1 -- 172.21.15.98:0/1844481190 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3009eaf0 con 0x7f0f300a3de0 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.302+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/1844481190 >> v2:172.21.15.190:3300/0 conn(0x7f0f300a41c0 0x7f0f300a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/1844481190 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f300a3de0 0x7f0f300a0570 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009870 tx=0x7f0f2001c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=24c1b922c1bcd479 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 1 -- 172.21.15.98:0/1844481190 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f200b0070 con 0x7f0f300a3de0 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: dump: 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.320 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 1 -- 172.21.15.98:0/1844481190 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f200aa810 con 0x7f0f300a3de0 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 5 monclient: authenticate success, global_id 24641 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 1 -- 172.21.15.98:0/1844481190 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f200a9300 con 0x7f0f300a3de0 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 4 set_mon_vals no callback set 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2effd640 10 monclient: dump: 2024-09-11T02:59:12.322 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.323 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient: shutdown 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- 172.21.15.98:0/1844481190 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f300a3de0 msgr2=0x7f0f300a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 --2- 172.21.15.98:0/1844481190 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f300a3de0 0x7f0f300a0570 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f0f20009870 tx=0x7f0f2001c440 comp rx=0 tx=0).stop 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- 172.21.15.98:0/1844481190 shutdown_connections 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 --2- 172.21.15.98:0/1844481190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0f3009b240 0x7f0f3009b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 --2- 172.21.15.98:0/1844481190 >> v2:172.21.15.190:3300/0 conn(0x7f0f300a41c0 0x7f0f300a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 --2- 172.21.15.98:0/1844481190 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f300a3de0 0x7f0f300a0570 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- 172.21.15.98:0/1844481190 >> 172.21.15.98:0/1844481190 conn(0x7f0f300a4ed0 msgr2=0x7f0f300a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- 172.21.15.98:0/1844481190 shutdown_connections 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- 172.21.15.98:0/1844481190 wait complete. 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient: monmap: 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.325 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 librados: starting msgr at 2024-09-11T02:59:12.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 librados: starting objecter 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 Processor -- start 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- start start 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 librados: setting wanted keys 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 librados: calling monclient init 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient: init 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient: _add_conns ranks=[8,5,1,7,4,3,6,0,2] 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f3009b240 0x7f0f3009a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): picked mon.noname-e con 0x7f0f3009b240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0f300a3de0 0x7f0f3009a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): picked mon.noname-h con 0x7f0f300a3de0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f300a41c0 0x7f0f3012e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): picked mon.noname-f con 0x7f0f300a41c0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301413a0 con 0x7f0f300a41c0 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f0f30141220 con 0x7f0f300a3de0 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0f30141520 con 0x7f0f3009b240 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 10 monclient(hunting): authenticate will time out at 1506.191068s 2024-09-11T02:59:12.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f3009b240 0x7f0f3009a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f3009b240 0x7f0f3009a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:48700/0 (socket says 172.21.15.98:48700) 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 -- 172.21.15.98:0/417837602 learned_addr learned my addr 172.21.15.98:0/417837602 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request con 0x7f0f3009b240 auth_method 0 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2f7fe640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0f300a3de0 msgr2=0x7f0f3009a8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3302/0 2024-09-11T02:59:12.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0f300a3de0 0x7f0f3009a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient(hunting): handle_auth_done global_id 24650 payload 824 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0f300a3de0 msgr2=0x7f0f3009a8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0f300a3de0 0x7f0f3009a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f34bd1640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f300a41c0 0x7f0f3012e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f300a41c0 msgr2=0x7f0f3012e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f300a41c0 0x7f0f3012e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 monclient: found mon.noname-e 2024-09-11T02:59:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:59:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 -- 172.21.15.98:0/417837602 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f300a3520 con 0x7f0f3009b240 2024-09-11T02:59:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f34bd1640 10 monclient: get_auth_request con 0x7f0f300a41c0 auth_method 0 2024-09-11T02:59:12.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f34bd1640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f300a41c0 0x7f0f3012e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2ffff640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f3009b240 0x7f0f3009a390 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f0f200ae040 tx=0x7f0f200191e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 client.?.objecter ms_handle_connect 0x7f0f3009b240 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f200b0040 con 0x7f0f3009b240 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.332 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.333 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f36634640 5 monclient: authenticate success, global_id 24650 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f200044b0 con 0x7f0f3009b240 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f200aada0 con 0x7f0f3009b240 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:59:12.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2e7fc640 4 set_mon_vals no callback set 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.306+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.335 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 10 monclient: _renew_subs 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f30131760 con 0x7f0f3009b240 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 10 client.24650.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 10 monclient: _renew_subs 2024-09-11T02:59:12.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f3009ae10 con 0x7f0f3009b240 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 1 librados: init done 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012ea20 0x7f0f3012ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0f3006ca10 con 0x7f0f3012ea20 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012ea20 0x7f0f3012ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient: get_auth_request con 0x7f0f3012ea20 auth_method 0 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0f200b3070 con 0x7f0f3009b240 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_done global_id 44484 payload 824 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 1 -- 172.21.15.98:0/417837602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f300008e0 con 0x7f0f3012ea20 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03df70 0x7f0f1c040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012ea20 0x7f0f3012ee40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0f3012fac0 tx=0x7f0f2400e950 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 client.24650.objecter ms_handle_connect 0x7f0f3012ea20 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 client.24650.objecter resend_mon_ops 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(365..365 src has 1..365) ==== 6189+0+0 (secure 0 0 0) 0x7f0f200e90a0 con 0x7f0f3009b240 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 client.24650.objecter ms_dispatch 0x7f0f30000ba0 osd_map(365..365 src has 1..365) 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 3 client.24650.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 3 client.24650.objecter handle_osd_map decoding full epoch 365 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 20 client.24650.objecter dump_active .. 0 homeless 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f24017020 con 0x7f0f3012ea20 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.339 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.340 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f34bd1640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03df70 0x7f0f1c040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0f24013430 con 0x7f0f3012ea20 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012ea20 msgr2=0x7f0f3012ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012ea20 0x7f0f3012ee40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0f3012fac0 tx=0x7f0f2400e950 comp rx=0 tx=0).stop 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012ea20 msgr2=0x7f0f3012ee40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012ea20 0x7f0f3012ee40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f34bd1640 10 monclient: get_auth_request con 0x7f0f1c03df70 auth_method 0 2024-09-11T02:59:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f2cff9640 10 client.24650.objecter ms_handle_connect 0x7f0f1c03df70 2024-09-11T02:59:12.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.310+0000 7f0f34bd1640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03df70 0x7f0f1c040430 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f0f28003cc0 tx=0x7f0f280072e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.314+0000 7f0f36634640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.314+0000 7f0f36634640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012f380 0x7f0f300717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.314+0000 7f0f36634640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.314+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0f3006e870 con 0x7f0f3012f380 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012f380 0x7f0f300717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient: get_auth_request con 0x7f0f3012f380 auth_method 0 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 10 monclient(hunting): handle_auth_done global_id 44493 payload 824 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 1 -- 172.21.15.98:0/417837602 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f30131e40 con 0x7f0f3012f380 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 10 client.24650.objecter ms_handle_connect 0x7f0f3012f380 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 10 client.24650.objecter resend_mon_ops 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2f7fe640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012f380 0x7f0f300717c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0f24018000 tx=0x7f0f24004740 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f24017040 con 0x7f0f3012f380 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T02:59:12.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 10 monclient: dump: 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.346 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9245 (secure 0 0 0) 0x7f0f2401eb30 con 0x7f0f3012f380 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012f380 msgr2=0x7f0f300717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012f380 0x7f0f300717c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0f24018000 tx=0x7f0f24004740 comp rx=0 tx=0).stop 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012f380 msgr2=0x7f0f300717c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f2cff9640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f3012f380 0x7f0f300717c0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0f24018000 tx=0x7f0f24004740 comp rx=0 tx=0).stop 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 72, 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 178745, 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:59:12.348 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:59:12.349 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.350 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.351 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.352 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.353 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:59:12.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.356 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.358 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.360 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.373 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.375 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:59:12.375 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:59:12.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 10 librados: watch_flush enter 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 10 librados: watch_flush exit 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 20 client.24650.objecter shutdown clearing up homeless session... 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 10 client.24650.objecter successfully canceled tick 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03df70 msgr2=0x7f0f1c040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f1c03df70 0x7f0f1c040430 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f0f28003cc0 tx=0x7f0f280072e0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 10 monclient: shutdown 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f3009b240 msgr2=0x7f0f3009a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 --2- 172.21.15.98:0/417837602 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f3009b240 0x7f0f3009a390 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f0f200ae040 tx=0x7f0f200191e0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f34bd1640 1 -- 172.21.15.98:0/417837602 reap_dead start 2024-09-11T02:59:12.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f34bd1640 1 -- 172.21.15.98:0/417837602 reap_dead start 2024-09-11T02:59:12.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 shutdown_connections 2024-09-11T02:59:12.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 >> 172.21.15.98:0/417837602 conn(0x7f0f300a4ed0 msgr2=0x7f0f3013cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 shutdown_connections 2024-09-11T02:59:12.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 -- 172.21.15.98:0/417837602 wait complete. 2024-09-11T02:59:12.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.318+0000 7f0f36634640 1 librados: shutdown 2024-09-11T02:59:12.378 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:59:12.378 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-09-11T02:59:12.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T02:59:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.494+0000 7f2a6dae5640 10 monclient: get_monmap_and_config 2024-09-11T02:59:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.494+0000 7f2a6dae5640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient: monmap: 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:12.501312+0000 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:12.501312+0000 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.500 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:12.501 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 1 Processor -- start 2024-09-11T02:59:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 1 -- start start 2024-09-11T02:59:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient: init 2024-09-11T02:59:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient: _add_conns ranks=[1,0,4,2,6,8,5,7,9,10,3] 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2a68139b80 addr v1:172.21.15.190:6789/0 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a6806fe50 addr v1:172.21.15.98:6789/0 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a68072640 0x7f2a68142130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a68072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a68071370 con 0x7f2a6806fe50 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a680a9600 con 0x7f2a68139b80 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2a680719c0 con 0x7f2a68072640 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a6dae5640 10 monclient(hunting): authenticate will time out at 1506.381808s 2024-09-11T02:59:12.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a66ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2a6806fe50 0x7f2a6813ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:37254/0 (socket says 172.21.15.98:37254) 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a66ffd640 1 -- 172.21.15.98:0/408969004 learned_addr learned my addr 172.21.15.98:0/408969004 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 1 --2- 172.21.15.98:0/408969004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a68072640 0x7f2a68142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): get_auth_request con 0x7f2a68072640 auth_method 0 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient(hunting): handle_auth_done global_id 24160 payload 274 2024-09-11T02:59:12.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 1 -- 172.21.15.98:0/408969004 >> v1:172.21.15.190:6789/0 conn(0x7f2a68139b80 legacy=0x7f2a68139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a667fc640 1 -- 172.21.15.98:0/408969004 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461004657 0 0) 0x7f2a68071370 con 0x7f2a6806fe50 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 1 -- 172.21.15.98:0/408969004 >> v1:172.21.15.98:6789/0 conn(0x7f2a6806fe50 legacy=0x7f2a6813ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 1 monclient: found mon.noname-c 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a67fff640 1 -- 172.21.15.98:0/408969004 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a68144690 con 0x7f2a68072640 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.498+0000 7f2a667fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 1 --2- 172.21.15.98:0/408969004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a68072640 0x7f2a68142130 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009dc0 tx=0x7f2a6001c850 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3fd43d468eb5d04c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 1 -- 172.21.15.98:0/408969004 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a600b2070 con 0x7f2a68072640 2024-09-11T02:59:12.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: dump: 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.509 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 1 -- 172.21.15.98:0/408969004 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a600a5980 con 0x7f2a68072640 2024-09-11T02:59:12.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 5 monclient: authenticate success, global_id 24160 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 1 -- 172.21.15.98:0/408969004 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a600ab340 con 0x7f2a68072640 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a667fc640 10 monclient: dump: 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.511 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 4 set_mon_vals no callback set 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: shutdown 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- 172.21.15.98:0/408969004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a68072640 msgr2=0x7f2a68142130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 --2- 172.21.15.98:0/408969004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a68072640 0x7f2a68142130 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009dc0 tx=0x7f2a6001c850 comp rx=0 tx=0).stop 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- 172.21.15.98:0/408969004 shutdown_connections 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 --2- 172.21.15.98:0/408969004 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a68072640 0x7f2a68142130 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- 172.21.15.98:0/408969004 >> 172.21.15.98:0/408969004 conn(0x7f2a680ac730 msgr2=0x7f2a680a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- 172.21.15.98:0/408969004 shutdown_connections 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- 172.21.15.98:0/408969004 wait complete. 2024-09-11T02:59:12.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: monmap: 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.514 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 librados: starting msgr at 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 librados: starting objecter 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 Processor -- start 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- start start 2024-09-11T02:59:12.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 librados: setting wanted keys 2024-09-11T02:59:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 librados: calling monclient init 2024-09-11T02:59:12.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: init 2024-09-11T02:59:12.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: _add_conns ranks=[4,7,6,2,3,0,5,1,8] 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a6806fe50 0x7f2a68137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): picked mon.noname-c con 0x7f2a6806fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2a68072640 0x7f2a68138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): picked mon.noname-i con 0x7f2a68072640 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2a68139b80 0x7f2a68138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): picked mon.noname-d con 0x7f2a68139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2a680702d0 con 0x7f2a6806fe50 2024-09-11T02:59:12.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2a68145690 con 0x7f2a68139b80 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f2a68145810 con 0x7f2a68072640 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient(hunting): authenticate will time out at 1506.386973s 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a6806fe50 0x7f2a68137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a6806fe50 0x7f2a68137bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34942/0 (socket says 172.21.15.98:34942) 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 -- 172.21.15.98:0/3224596807 learned_addr learned my addr 172.21.15.98:0/3224596807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): get_auth_request con 0x7f2a6806fe50 auth_method 0 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2a68139b80 0x7f2a68138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 10 monclient(hunting): get_auth_request con 0x7f2a68139b80 auth_method 0 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient(hunting): handle_auth_done global_id 24169 payload 824 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a66ffd640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2a68072640 0x7f2a68138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2a68072640 msgr2=0x7f2a68138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2a68072640 0x7f2a68138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2a68139b80 msgr2=0x7f2a68138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2a68139b80 0x7f2a68138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 monclient: found mon.noname-c 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:59:12.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 -- 172.21.15.98:0/3224596807 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a68140a90 con 0x7f2a6806fe50 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a677fe640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a6806fe50 0x7f2a68137bf0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800c420 tx=0x7f2a5800c8e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 client.?.objecter ms_handle_connect 0x7f2a6806fe50 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a58017070 con 0x7f2a6806fe50 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: dump: 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.527 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.528 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a58003600 con 0x7f2a6806fe50 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a58003c60 con 0x7f2a6806fe50 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a4ffff640 10 monclient: dump: 2024-09-11T02:59:12.529 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.530 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a65ffb640 4 set_mon_vals no callback set 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 5 monclient: authenticate success, global_id 24169 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: _renew_subs 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a680a80b0 con 0x7f2a6806fe50 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 client.24169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: _renew_subs 2024-09-11T02:59:12.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T02:59:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a681407b0 con 0x7f2a6806fe50 2024-09-11T02:59:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.502+0000 7f2a6dae5640 1 librados: init done 2024-09-11T02:59:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a6dae5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a6dae5640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a6813e580 0x7f2a68063e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a6dae5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2a6806ca10 con 0x7f2a6813e580 2024-09-11T02:59:12.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a6813e580 0x7f2a68063e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient: get_auth_request con 0x7f2a6813e580 auth_method 0 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2a580037f0 con 0x7f2a6806fe50 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_done global_id 24659 payload 824 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 1 -- 172.21.15.98:0/3224596807 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a6813f3d0 con 0x7f2a6813e580 2024-09-11T02:59:12.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2a5403df40 0x7f2a54040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(365..365 src has 1..365) ==== 6189+0+0 (secure 0 0 0) 0x7f2a5804ec30 con 0x7f2a6806fe50 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 client.24169.objecter ms_dispatch 0x7f2a68000ba0 osd_map(365..365 src has 1..365) 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 3 client.24169.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 3 client.24169.objecter handle_osd_map decoding full epoch 365 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a66ffd640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a6813e580 0x7f2a68063e00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f2a6813f360 tx=0x7f2a5c00d950 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 20 client.24169.objecter dump_active .. 0 homeless 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 client.24169.objecter ms_handle_connect 0x7f2a6813e580 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 client.24169.objecter resend_mon_ops 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a5c0070f0 con 0x7f2a6813e580 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:59:12.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 monclient: dump: 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.535 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a67fff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2a5403df40 0x7f2a54040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a67fff640 10 monclient: get_auth_request con 0x7f2a5403df40 auth_method 0 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2a5c017070 con 0x7f2a6813e580 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a6813e580 msgr2=0x7f2a68063e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a6813e580 0x7f2a68063e00 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f2a6813f360 tx=0x7f2a5c00d950 comp rx=0 tx=0).stop 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a6813e580 msgr2=0x7f2a68063e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a6813e580 0x7f2a68063e00 secure :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f2a6813f360 tx=0x7f2a5c00d950 comp rx=0 tx=0).stop 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a4ffff640 10 client.24169.objecter ms_handle_connect 0x7f2a5403df40 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.506+0000 7f2a67fff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2a5403df40 0x7f2a54040400 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7f2a600af000 tx=0x7f2a600a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.510+0000 7f2a6dae5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.510+0000 7f2a6dae5640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a68064340 0x7f2a6802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.510+0000 7f2a6dae5640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.510+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2a6806e870 con 0x7f2a68064340 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a68064340 0x7f2a6802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient: get_auth_request con 0x7f2a68064340 auth_method 0 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 10 monclient(hunting): handle_auth_done global_id 24668 payload 824 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 1 -- 172.21.15.98:0/3224596807 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2a680a85c0 con 0x7f2a68064340 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a66ffd640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a68064340 0x7f2a6802d450 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c001230 tx=0x7f2a5c000ee0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 10 client.24169.objecter ms_handle_connect 0x7f2a68064340 2024-09-11T02:59:12.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 10 client.24169.objecter resend_mon_ops 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2a5c01ce40 con 0x7f2a68064340 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 10 monclient: dump: 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.539 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9247 (secure 0 0 0) 0x7f2a5c01c9d0 con 0x7f2a68064340 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:12.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a68064340 msgr2=0x7f2a6802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a68064340 0x7f2a6802d450 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c001230 tx=0x7f2a5c000ee0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a68064340 msgr2=0x7f2a6802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a4ffff640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2a68064340 0x7f2a6802d450 secure :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c001230 tx=0x7f2a5c000ee0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.541 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:59:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:12.541 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 72, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 568306, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:59:12.542 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.543 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:59:12.545 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:59:12.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.547 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.549 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:12.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:59:12.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.552 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.557 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.559 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.560 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.562 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.563 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.565 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.566 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:59:12.567 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:59:12.568 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:59:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 10 librados: watch_flush enter 2024-09-11T02:59:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 10 librados: watch_flush exit 2024-09-11T02:59:12.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 20 client.24169.objecter shutdown clearing up homeless session... 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 10 client.24169.objecter successfully canceled tick 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2a5403df40 msgr2=0x7f2a54040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2a5403df40 0x7f2a54040400 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7f2a600af000 tx=0x7f2a600a7070 comp rx=0 tx=0).stop 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 10 monclient: shutdown 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a6806fe50 msgr2=0x7f2a68137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 --2- 172.21.15.98:0/3224596807 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2a6806fe50 0x7f2a68137bf0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800c420 tx=0x7f2a5800c8e0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a67fff640 1 -- 172.21.15.98:0/3224596807 reap_dead start 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a67fff640 1 -- 172.21.15.98:0/3224596807 reap_dead start 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 shutdown_connections 2024-09-11T02:59:12.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 >> 172.21.15.98:0/3224596807 conn(0x7f2a680ac730 msgr2=0x7f2a680aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 shutdown_connections 2024-09-11T02:59:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 -- 172.21.15.98:0/3224596807 wait complete. 2024-09-11T02:59:12.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.514+0000 7f2a6dae5640 1 librados: shutdown 2024-09-11T02:59:12.572 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:59:12.572 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-09-11T02:59:12.572 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T02:59:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient: get_monmap_and_config 2024-09-11T02:59:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient: monmap: 2024-09-11T02:59:12.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:12.689790+0000 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:12.689790+0000 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:12.690 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 Processor -- start 2024-09-11T02:59:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 -- start start 2024-09-11T02:59:12.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient: init 2024-09-11T02:59:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient: _add_conns ranks=[7,8,4,10,9,2,0,6,5,3,1] 2024-09-11T02:59:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5e4139b10 0x7fc5e4072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): picked mon.noname-i con 0x7fc5e4139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5e4139ef0 0x7fc5e40a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): picked mon.noname-e con 0x7fc5e4139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:59:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc5e413a2d0 0x7fc5e40a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e413a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc5e40702b0 con 0x7fc5e413a2d0 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc5e4071bf0 con 0x7fc5e4139b10 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc5e4071350 con 0x7fc5e4139ef0 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e9afe640 10 monclient(hunting): authenticate will time out at 1506.570084s 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5e4139ef0 0x7fc5e40a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc5e413a2d0 0x7fc5e40a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5e4139ef0 0x7fc5e40a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:48720/0 (socket says 172.21.15.98:48720) 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 -- 172.21.15.98:0/465454123 learned_addr learned my addr 172.21.15.98:0/465454123 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc5e4139ef0 auth_method 0 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 10 monclient(hunting): get_auth_request con 0x7fc5e413a2d0 auth_method 0 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e37fe640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5e4139b10 0x7fc5e4072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e37fe640 10 monclient(hunting): get_auth_request con 0x7fc5e4139b10 auth_method 0 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_done global_id 24677 payload 274 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 -- 172.21.15.98:0/465454123 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5e4139b10 msgr2=0x7fc5e4072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5e4139b10 0x7fc5e4072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 -- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc5e413a2d0 msgr2=0x7fc5e40a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc5e413a2d0 0x7fc5e40a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 monclient: found mon.noname-e 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e2ffd640 1 -- 172.21.15.98:0/465454123 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e40a9620 con 0x7fc5e4139ef0 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.686+0000 7fc5e3fff640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc5e413a2d0 0x7fc5e40a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5e4139ef0 0x7fc5e40a7a10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009e80 tx=0x7fc5d801c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=476bab0f3da96f31 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 1 -- 172.21.15.98:0/465454123 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5d80090d0 con 0x7fc5e4139ef0 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: dump: 2024-09-11T02:59:12.706 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.707 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 1 -- 172.21.15.98:0/465454123 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d80a5a70 con 0x7fc5e4139ef0 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 1 -- 172.21.15.98:0/465454123 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5d80a5c60 con 0x7fc5e4139ef0 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 5 monclient: authenticate success, global_id 24677 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 4 set_mon_vals no callback set 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e27fc640 10 monclient: dump: 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.709 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.710 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient: shutdown 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5e4139ef0 msgr2=0x7fc5e40a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5e4139ef0 0x7fc5e40a7a10 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009e80 tx=0x7fc5d801c910 comp rx=0 tx=0).stop 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- 172.21.15.98:0/465454123 shutdown_connections 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc5e413a2d0 0x7fc5e40a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5e4139ef0 0x7fc5e40a7a10 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/465454123 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5e4139b10 0x7fc5e4072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- 172.21.15.98:0/465454123 >> 172.21.15.98:0/465454123 conn(0x7fc5e40ac790 msgr2=0x7fc5e40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- 172.21.15.98:0/465454123 shutdown_connections 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- 172.21.15.98:0/465454123 wait complete. 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient: monmap: 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.712 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 librados: starting msgr at 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 librados: starting objecter 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 Processor -- start 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- start start 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 librados: setting wanted keys 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 librados: calling monclient init 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient: init 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient: _add_conns ranks=[2,3,1,4,0,5,8,7,6] 2024-09-11T02:59:12.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5e4139b10 0x7fc5e41d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e4139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4139ef0 0x7fc5e413fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): picked mon.noname-g con 0x7fc5e4139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a2d0 0x7fc5e413ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): picked mon.noname-f con 0x7fc5e413a2d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e4143460 con 0x7fc5e413a2d0 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e40a88a0 con 0x7fc5e4139b10 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e41435e0 con 0x7fc5e4139ef0 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.690+0000 7fc5e9afe640 10 monclient(hunting): authenticate will time out at 1506.575937s 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5e4139b10 0x7fc5e41d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5e4139b10 0x7fc5e41d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:57202/0 (socket says 172.21.15.98:57202) 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 -- 172.21.15.98:0/138500771 learned_addr learned my addr 172.21.15.98:0/138500771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): get_auth_request con 0x7fc5e4139b10 auth_method 0 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4139ef0 msgr2=0x7fc5e413fa70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4139ef0 0x7fc5e413fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e3fff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a2d0 0x7fc5e413ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient(hunting): handle_auth_done global_id 64298 payload 824 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4139ef0 msgr2=0x7fc5e413fa70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc5e4139ef0 0x7fc5e413fa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a2d0 msgr2=0x7fc5e413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a2d0 0x7fc5e413ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 monclient: found mon.noname-b 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 -- 172.21.15.98:0/138500771 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e413c1e0 con 0x7fc5e4139b10 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e3fff640 10 monclient: get_auth_request con 0x7fc5e413a2d0 auth_method 0 2024-09-11T02:59:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e3fff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a2d0 0x7fc5e413ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e37fe640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5e4139b10 0x7fc5e41d15e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400d800 tx=0x7fc5d400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 client.?.objecter ms_handle_connect 0x7fc5e4139b10 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5d400bb30 con 0x7fc5e4139b10 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: dump: 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 5 monclient: authenticate success, global_id 64298 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d4019070 con 0x7fc5e4139b10 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5d400bcf0 con 0x7fc5e4139b10 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 10 monclient: _renew_subs 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e40aa200 con 0x7fc5e4139b10 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e17fa640 4 set_mon_vals no callback set 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: dump: 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 10 client.64298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 10 monclient: _renew_subs 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e413e260 con 0x7fc5e4139b10 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 1 librados: init done 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a7d0 0x7fc5e413abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e406ca10 con 0x7fc5e413a7d0 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc5d4013070 con 0x7fc5e4139b10 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a7d0 0x7fc5e413abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient: get_auth_request con 0x7fc5e413a7d0 auth_method 0 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc5d003de70 0x7fc5d0040330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(365..365 src has 1..365) ==== 6189+0+0 (secure 0 0 0) 0x7fc5d400f040 con 0x7fc5e4139b10 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 client.64298.objecter ms_dispatch 0x7fc5e4000ba0 osd_map(365..365 src has 1..365) 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 3 client.64298.objecter handle_osd_map got epochs [365,365] > 0 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 3 client.64298.objecter handle_osd_map decoding full epoch 365 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 20 client.64298.objecter dump_active .. 0 homeless 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e3fff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc5d003de70 0x7fc5d0040330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e3fff640 10 monclient: get_auth_request con 0x7fc5d003de70 auth_method 0 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_done global_id 25489 payload 824 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 1 -- 172.21.15.98:0/138500771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5e406e680 con 0x7fc5e413a7d0 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 client.64298.objecter ms_handle_connect 0x7fc5e413a7d0 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 client.64298.objecter resend_mon_ops 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a7d0 0x7fc5e413abb0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009e50 tx=0x7fc5d80196b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5d80bb070 con 0x7fc5e413a7d0 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 monclient: dump: 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.724 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5e3fff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc5d003de70 0x7fc5d0040330 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40a9a70 tx=0x7fc5dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.694+0000 7fc5cbfff640 10 client.64298.objecter ms_handle_connect 0x7fc5d003de70 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.698+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc5d8009af0 con 0x7fc5e413a7d0 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.698+0000 7fc5cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.698+0000 7fc5cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.698+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a7d0 msgr2=0x7fc5e413abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.698+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a7d0 0x7fc5e413abb0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009e50 tx=0x7fc5d80196b0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.698+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a7d0 msgr2=0x7fc5e413abb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.698+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413a7d0 0x7fc5e413abb0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e9afe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:59:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413b0f0 0x7fc5e40a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e9afe640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e406e870 con 0x7fc5e413b0f0 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413b0f0 0x7fc5e40a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient: get_auth_request con 0x7fc5e413b0f0 auth_method 0 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 10 monclient(hunting): handle_auth_done global_id 25498 payload 824 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 1 -- 172.21.15.98:0/138500771 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5e413dee0 con 0x7fc5e413b0f0 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5e2ffd640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413b0f0 0x7fc5e40a0e60 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fc5d801c740 tx=0x7fc5d8009810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5cbfff640 10 client.64298.objecter ms_handle_connect 0x7fc5e413b0f0 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5cbfff640 10 client.64298.objecter resend_mon_ops 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5d80b7e40 con 0x7fc5e413b0f0 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5cbfff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.702+0000 7fc5cbfff640 10 monclient: dump: 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.729 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9245 (secure 0 0 0) 0x7fc5d80b79d0 con 0x7fc5e413b0f0 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413b0f0 msgr2=0x7fc5e40a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413b0f0 0x7fc5e40a0e60 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fc5d801c740 tx=0x7fc5d8009810 comp rx=0 tx=0).stop 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5cbfff640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413b0f0 msgr2=0x7fc5e40a0e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5cbfff640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e413b0f0 0x7fc5e40a0e60 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 72, 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 568476, 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:59:12.731 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.732 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:59:12.733 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.734 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:59:12.735 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.736 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.738 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:59:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.745 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:59:12.746 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.747 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.749 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.753 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:59:12.754 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:59:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 10 librados: watch_flush enter 2024-09-11T02:59:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 10 librados: watch_flush exit 2024-09-11T02:59:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 20 client.64298.objecter shutdown clearing up homeless session... 2024-09-11T02:59:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 10 client.64298.objecter successfully canceled tick 2024-09-11T02:59:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc5d003de70 msgr2=0x7fc5d0040330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc5d003de70 0x7fc5d0040330 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40a9a70 tx=0x7fc5dc008040 comp rx=0 tx=0).stop 2024-09-11T02:59:12.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 10 monclient: shutdown 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5e4139b10 msgr2=0x7fc5e41d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 --2- 172.21.15.98:0/138500771 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5e4139b10 0x7fc5e41d15e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400d800 tx=0x7fc5d400dcc0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e3fff640 1 -- 172.21.15.98:0/138500771 reap_dead start 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e3fff640 1 -- 172.21.15.98:0/138500771 reap_dead start 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 shutdown_connections 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 >> 172.21.15.98:0/138500771 conn(0x7fc5e40ac790 msgr2=0x7fc5e4135a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 shutdown_connections 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 -- 172.21.15.98:0/138500771 wait complete. 2024-09-11T02:59:12.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.706+0000 7fc5e9afe640 1 librados: shutdown 2024-09-11T02:59:12.758 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:59:12.758 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-09-11T02:59:12.758 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T02:59:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.882+0000 7fc4dc000640 10 monclient: get_monmap_and_config 2024-09-11T02:59:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.882+0000 7fc4dc000640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.882+0000 7fc4dc000640 10 monclient: monmap: 2024-09-11T02:59:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:59:12.888175+0000 2024-09-11T02:59:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:59:12.888175+0000 2024-09-11T02:59:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T02:59:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T02:59:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.889 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T02:59:12.890 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T02:59:12.891 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 Processor -- start 2024-09-11T02:59:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 -- start start 2024-09-11T02:59:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient: init 2024-09-11T02:59:12.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient: _add_conns ranks=[8,10,7,2,4,3,6,1,0,9,5] 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc4d40a3690 0x7fc4d40a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): picked mon.noname-e con 0x7fc4d40a3690 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc4d40a1860 0x7fc4d40a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): picked mon.noname-f con 0x7fc4d40a1860 addr v2:172.21.15.190:3300/0 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d409ec20 0x7fc4d4137460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): picked mon.noname-i con 0x7fc4d409ec20 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc4d40a44f0 con 0x7fc4d40a1860 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc4d409d460 con 0x7fc4d409ec20 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc4d409d740 con 0x7fc4d40a3690 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4dc000640 10 monclient(hunting): authenticate will time out at 1506.771069s 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc4d40a3690 0x7fc4d40a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc4d40a3690 0x7fc4d40a3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:48726/0 (socket says 172.21.15.98:48726) 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 -- 172.21.15.98:0/3844067184 learned_addr learned my addr 172.21.15.98:0/3844067184 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 1 --2- 172.21.15.98:0/3844067184 >> v2:172.21.15.190:3300/0 conn(0x7fc4d40a1860 0x7fc4d40a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): get_auth_request con 0x7fc4d40a3690 auth_method 0 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 10 monclient(hunting): get_auth_request con 0x7fc4d40a1860 auth_method 0 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d959b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4da59d640 1 --2- 172.21.15.98:0/3844067184 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d409ec20 0x7fc4d4137460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_done global_id 24686 payload 274 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 -- 172.21.15.98:0/3844067184 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d409ec20 msgr2=0x7fc4d4137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/3844067184 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d409ec20 0x7fc4d4137460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 -- 172.21.15.98:0/3844067184 >> v2:172.21.15.190:3300/0 conn(0x7fc4d40a1860 msgr2=0x7fc4d40a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/3844067184 >> v2:172.21.15.190:3300/0 conn(0x7fc4d40a1860 0x7fc4d40a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 monclient: found mon.noname-e 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.886+0000 7fc4d9d9c640 1 -- 172.21.15.98:0/3844067184 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d40a5410 con 0x7fc4d40a3690 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4da59d640 10 monclient: get_auth_request con 0x7fc4d409ec20 auth_method 0 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4da59d640 1 --2- 172.21.15.98:0/3844067184 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d409ec20 0x7fc4d4137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/3844067184 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc4d40a3690 0x7fc4d40a3a70 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009870 tx=0x7fc4d0020510 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c79f97b9dec9255f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 1 -- 172.21.15.98:0/3844067184 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4d00b5070 con 0x7fc4d40a3690 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: dump: 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.910 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.911 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 5 monclient: authenticate success, global_id 24686 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 1 -- 172.21.15.98:0/3844067184 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4d00a8be0 con 0x7fc4d40a3690 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 10 monclient: get_monmap_and_config success 2024-09-11T02:59:12.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 4 set_mon_vals no callback set 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 1 -- 172.21.15.98:0/3844067184 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4d00a8e10 con 0x7fc4d40a3690 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4d8d9a640 10 monclient: dump: 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 10 monclient: shutdown 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 -- 172.21.15.98:0/3844067184 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc4d40a3690 msgr2=0x7fc4d40a3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 --2- 172.21.15.98:0/3844067184 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc4d40a3690 0x7fc4d40a3a70 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fc4d0009870 tx=0x7fc4d0020510 comp rx=0 tx=0).stop 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 -- 172.21.15.98:0/3844067184 shutdown_connections 2024-09-11T02:59:12.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 --2- 172.21.15.98:0/3844067184 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d409ec20 0x7fc4d4137460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 --2- 172.21.15.98:0/3844067184 >> v2:172.21.15.190:3300/0 conn(0x7fc4d40a1860 0x7fc4d40a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 --2- 172.21.15.98:0/3844067184 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc4d40a3690 0x7fc4d40a3a70 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 -- 172.21.15.98:0/3844067184 >> 172.21.15.98:0/3844067184 conn(0x7fc4d40a8280 msgr2=0x7fc4d40a4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 -- 172.21.15.98:0/3844067184 shutdown_connections 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.890+0000 7fc4dc000640 1 -- 172.21.15.98:0/3844067184 wait complete. 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient: build_initial_monmap 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient: monmap: 2024-09-11T02:59:12.916 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T02:59:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 librados: starting msgr at 2024-09-11T02:59:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 librados: starting objecter 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 Processor -- start 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 -- start start 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 librados: setting wanted keys 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 librados: calling monclient init 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient: init 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient: _add_conns ranks=[3,4,5,8,1,6,0,2,7] 2024-09-11T02:59:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d409ec20 0x7fc4d41cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): picked mon.noname-g con 0x7fc4d409ec20 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 0x7fc4d41346b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): picked mon.noname-c con 0x7fc4d40a1860 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc4d40a3690 0x7fc4d413b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): picked mon.noname-h con 0x7fc4d40a3690 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d413f8e0 con 0x7fc4d409ec20 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc4d413f760 con 0x7fc4d40a1860 2024-09-11T02:59:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc4d413fa60 con 0x7fc4d40a3690 2024-09-11T02:59:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4dc000640 10 monclient(hunting): authenticate will time out at 1506.777843s 2024-09-11T02:59:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 -- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc4d40a3690 msgr2=0x7fc4d413b8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3302/0 2024-09-11T02:59:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc4d40a3690 0x7fc4d413b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:12.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 0x7fc4d41346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 0x7fc4d41346b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34958/0 (socket says 172.21.15.98:34958) 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 -- 172.21.15.98:0/130634662 learned_addr learned my addr 172.21.15.98:0/130634662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): get_auth_request con 0x7fc4d40a1860 auth_method 0 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): handle_auth_done global_id 24187 payload 824 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc4d40a3690 msgr2=0x7fc4d413b8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc4d40a3690 0x7fc4d413b8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d9d9c640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d409ec20 msgr2=0x7fc4d41cdc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d409ec20 0x7fc4d41cdc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d409ec20 msgr2=0x7fc4d41cdc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T02:59:12.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc4d409ec20 0x7fc4d41cdc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 monclient: found mon.noname-c 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d411fd70 con 0x7fc4d40a1860 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 0x7fc4d41346b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4006d50 tx=0x7fc4c400ed60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 client.?.objecter ms_handle_connect 0x7fc4d40a1860 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:12.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 msgr2=0x7fc4d41346b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 msgr2=0x7fc4d41346b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 0x7fc4d41346b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4006d50 tx=0x7fc4c400ed60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 0x7fc4d41346b0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4006d50 tx=0x7fc4c400ed60 comp rx=0 tx=0).stop 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient: _reopen_session rank -1 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 msgr2=0x7fc4d41346b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc4d40a1860 0x7fc4d41346b0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient: _add_conns ranks=[1,7,2,8,6,0] 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8004f70 0x7fc4b80073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient(hunting): picked mon.noname-f con 0x7fc4b8004f70 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4b8007e70 0x7fc4b800a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient(hunting): picked mon.noname-i con 0x7fc4b8007e70 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4b800aea0 0x7fc4b800d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4b800aea0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b800df10 con 0x7fc4b8004f70 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc4b800e090 con 0x7fc4b800aea0 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc4b800e290 con 0x7fc4b8007e70 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 monclient(hunting): _renew_subs 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4b800aea0 0x7fc4b800d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): get_auth_request con 0x7fc4b800aea0 auth_method 0 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4b8007e70 0x7fc4b800a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8004f70 0x7fc4b80073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): get_auth_request con 0x7fc4b8007e70 auth_method 0 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d959b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d9d9c640 10 monclient(hunting): get_auth_request con 0x7fc4b8004f70 auth_method 0 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d9d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient(hunting): handle_auth_done global_id 24187 payload 293 2024-09-11T02:59:12.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4b8007e70 msgr2=0x7fc4b800a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4b8007e70 0x7fc4b800a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8004f70 msgr2=0x7fc4b80073c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc4b8004f70 0x7fc4b80073c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient: _finish_hunting 0 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 monclient: found mon.noname-b 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b800e550 con 0x7fc4b800aea0 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 -- 172.21.15.98:0/130634662 reap_dead start 2024-09-11T02:59:12.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4da59d640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4b800aea0 0x7fc4b800d360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc008e40 tx=0x7fc4c4009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 client.?.objecter ms_handle_connect 0x7fc4b800aea0 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.894+0000 7fc4ca7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4cc008af0 con 0x7fc4b800aea0 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: dump: 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.930 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: _finish_auth 0 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: _check_auth_tickets 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4cc00b8e0 con 0x7fc4b800aea0 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 5 monclient: authenticate success, global_id 24187 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 10 monclient: _renew_subs 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d4120050 con 0x7fc4b800aea0 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4cc00e440 con 0x7fc4b800aea0 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T02:59:12.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 monclient: dump: 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.933 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4cbfff640 4 set_mon_vals no callback set 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 10 client.24187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 10 monclient: _renew_subs 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T02:59:12.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d413d370 con 0x7fc4b800aea0 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 1 librados: init done 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 0x7fc4d41cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc4d406ca10 con 0x7fc4d411f7b0 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc4cc00b470 con 0x7fc4b800aea0 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4b80428c0 0x7fc4b8044d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 0x7fc4d41cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(366..366 src has 1..366) ==== 6202+0+0 (secure 0 0 0) 0x7fc4cc04eaa0 con 0x7fc4b800aea0 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 client.24187.objecter ms_dispatch 0x7fc4d4000ba0 osd_map(366..366 src has 1..366) 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 3 client.24187.objecter handle_osd_map got epochs [366,366] > 0 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 3 client.24187.objecter handle_osd_map decoding full epoch 366 2024-09-11T02:59:12.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient: get_auth_request con 0x7fc4d411f7b0 auth_method 0 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 20 client.24187.objecter dump_active .. 0 homeless 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4b80428c0 0x7fc4b8044d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d959b640 10 monclient: get_auth_request con 0x7fc4b80428c0 auth_method 0 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_done global_id 19789 payload 824 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4d4068b40 con 0x7fc4d411f7b0 2024-09-11T02:59:12.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 0x7fc4d41cdc70 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40a5380 tx=0x7fc4d001d5d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 client.24187.objecter ms_handle_connect 0x7fc4d411f7b0 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 client.24187.objecter resend_mon_ops 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4ca7fc640 10 client.24187.objecter ms_handle_connect 0x7fc4b80428c0 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.898+0000 7fc4d959b640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4b80428c0 0x7fc4b8044d80 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400e820 tx=0x7fc4c401e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4d00c0030 con 0x7fc4d411f7b0 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 10 monclient: dump: 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.937 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.939 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.939 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc4d00b6040 con 0x7fc4d411f7b0 2024-09-11T02:59:12.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T02:59:12.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T02:59:12.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 msgr2=0x7fc4d41cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 0x7fc4d41cdc70 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40a5380 tx=0x7fc4d001d5d0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 msgr2=0x7fc4d41cdc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.902+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 0x7fc4d41cdc70 secure :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40a5380 tx=0x7fc4d001d5d0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 0x7fc4d413b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 10 monclient(hunting): start opening mon connection 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc4d406e870 con 0x7fc4d411e850 2024-09-11T02:59:12.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 0x7fc4d413b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient: get_auth_request con 0x7fc4d411e850 auth_method 0 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 10 monclient(hunting): handle_auth_done global_id 19798 payload 824 2024-09-11T02:59:12.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 1 -- 172.21.15.98:0/130634662 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc4d4069020 con 0x7fc4d411e850 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4d9d9c640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 0x7fc4d413b8b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00097c0 tx=0x7fc4d00aa000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 10 client.24187.objecter ms_handle_connect 0x7fc4d411e850 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 10 client.24187.objecter resend_mon_ops 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4d00c0040 con 0x7fc4d411e850 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 10 monclient: dump: 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T02:59:12.944 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T02:59:12.945 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9412 (secure 0 0 0) 0x7fc4d00c6a20 con 0x7fc4d411e850 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T02:59:12.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T02:59:12.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 msgr2=0x7fc4d413b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 0x7fc4d413b8b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00097c0 tx=0x7fc4d00aa000 comp rx=0 tx=0).stop 2024-09-11T02:59:12.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 msgr2=0x7fc4d413b8b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T02:59:12.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4ca7fc640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 0x7fc4d413b8b0 secure :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00097c0 tx=0x7fc4d00aa000 comp rx=0 tx=0).stop 2024-09-11T02:59:12.947 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T02:59:12.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:12.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 72, 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 568684, 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T02:59:12.948 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.949 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.951 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.952 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.952 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.952 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T02:59:12.952 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T02:59:12.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T02:59:12.954 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.955 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T02:59:12.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.961 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T02:59:12.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.963 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.965 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.967 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T02:59:12.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.974 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.976 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.976 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T02:59:12.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.976 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.976 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.976 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T02:59:12.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.977 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T02:59:12.977 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T02:59:12.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T02:59:12.977 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T02:59:12.977 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T02:59:12.977 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T02:59:12.978 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T02:59:12.978 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T02:59:12.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T02:59:12.978 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T02:59:12.978 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T02:59:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 10 librados: watch_flush enter 2024-09-11T02:59:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 10 librados: watch_flush exit 2024-09-11T02:59:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 20 client.24187.objecter shutdown clearing up homeless session... 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 10 client.24187.objecter successfully canceled tick 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4b80428c0 msgr2=0x7fc4b8044d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4b80428c0 0x7fc4b8044d80 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7fc4c400e820 tx=0x7fc4c401e040 comp rx=0 tx=0).stop 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 10 monclient: shutdown 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4b800aea0 msgr2=0x7fc4b800d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4b800aea0 0x7fc4b800d360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc008e40 tx=0x7fc4c4009190 comp rx=0 tx=0).stop 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 shutdown_connections 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4b80428c0 0x7fc4b8044d80 unknown :-1 s=CLOSED pgs=3565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc4b800aea0 0x7fc4b800d360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411e850 0x7fc4d413b8b0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 --2- 172.21.15.98:0/130634662 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc4d411f7b0 0x7fc4d41cdc70 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T02:59:12.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 >> 172.21.15.98:0/130634662 conn(0x7fc4d40a8280 msgr2=0x7fc4d4072500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T02:59:12.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 shutdown_connections 2024-09-11T02:59:12.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 -- 172.21.15.98:0/130634662 wait complete. 2024-09-11T02:59:12.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T02:59:12.910+0000 7fc4dc000640 1 librados: shutdown 2024-09-11T02:59:12.984 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-09-11T02:59:12.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T02:59:43.475 INFO:tasks.ceph.osd.1.smithi098.stderr:2024-09-11T02:59:43.470+0000 7f4c01f17640 -1 snap_mapper.add_oid found existing snaps mapped on 300:74d77409:test-rados-api-smithi190-28350-80::foo:2, removing 2024-09-11T02:59:43.476 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-11T02:59:43.470+0000 7f2bc1659640 -1 snap_mapper.add_oid found existing snaps mapped on 300:74d77409:test-rados-api-smithi190-28350-80::foo:2, removing 2024-09-11T03:00:14.452 INFO:tasks.workunit.client.0.smithi190.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (27689 ms) 2024-09-11T03:00:14.452 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-11T03:00:14.452 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (36298 ms) 2024-09-11T03:00:14.452 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-11T03:00:14.452 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3019 ms) 2024-09-11T03:00:14.452 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3047 ms) 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2916 ms) 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3022 ms) 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4039 ms) 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-11T03:00:14.453 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39977 ms) 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17012 ms) 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3979 ms) 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3034 ms) 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25224 ms) 2024-09-11T03:00:14.454 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (23251 ms) 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5045 ms) 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3459 ms) 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: waiting for scrubs... 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: done waiting 2024-09-11T03:00:14.455 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25170 ms) 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9911 ms) 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9080 ms) 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8910 ms) 2024-09-11T03:00:14.456 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-11T03:00:42.984 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-09-11T03:00:42.985 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-11T03:00:42.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-11T03:00:42.989 INFO:tasks.ceph.mon.g:Started 2024-09-11T03:00:42.989 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-09-11T03:00:42.989 INFO:tasks.ceph.mon.h:Restarting daemon 2024-09-11T03:00:42.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-09-11T03:00:42.991 INFO:tasks.ceph.mon.h:Started 2024-09-11T03:00:42.992 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:00:42.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:00:43.094 INFO:tasks.ceph.mon.g.smithi190.stderr:2024-09-11T03:00:43.098+0000 7fcf3c1a9e00 -1 mon.g@-1(???).log v0 log_external_backlog rewinding external_log_to from 375 -> 0 (sync_force? mon rebuild?) 2024-09-11T03:00:43.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient: get_monmap_and_config 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient: build_initial_monmap 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient: monmap: 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:43.122939+0000 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:43.122939+0000 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:43.121 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:43.122 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:43.122 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:43.122 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:43.122 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:43.122 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:43.122 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:43.122 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 1 Processor -- start 2024-09-11T03:00:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 1 -- start start 2024-09-11T03:00:43.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient: init 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient: _add_conns ranks=[2,0,8,3,10,9,4,5,1,7,6] 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab40a28c0 0x7f7ab40a2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient(hunting): picked mon.noname-b con 0x7f7ab40a28c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ab40a2180 addr v1:172.21.15.98:6789/0 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7ab4077760 0x7f7ab40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient(hunting): picked mon.noname-e con 0x7f7ab4077760 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ab40a0f00 con 0x7f7ab40a2180 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab40a1550 con 0x7f7ab40a28c0 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:43.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7ab40a11e0 con 0x7f7ab4077760 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7abc61b640 10 monclient(hunting): authenticate will time out at 1597.003297s 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab9bb6640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f7ab40a2180 0x7f7ab40a47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:53914/0 (socket says 172.21.15.98:53914) 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab9bb6640 1 -- 172.21.15.98:0/734353864 learned_addr learned my addr 172.21.15.98:0/734353864 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab40a28c0 0x7f7ab40a2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7ab4077760 0x7f7ab40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 10 monclient(hunting): get_auth_request con 0x7f7ab40a28c0 auth_method 0 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): get_auth_request con 0x7f7ab4077760 auth_method 0 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:43.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7aba3b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab93b5640 1 -- 172.21.15.98:0/734353864 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065666843 0 0) 0x7f7ab40a0f00 con 0x7f7ab40a2180 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab93b5640 10 monclient(hunting): _negotiate 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab93b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab93b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab93b5640 10 monclient(hunting): my global_id is 44640 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ab93b5640 1 -- 172.21.15.98:0/734353864 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa0003530 con 0x7f7ab40a2180 2024-09-11T03:00:43.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient(hunting): handle_auth_done global_id 24695 payload 274 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 1 -- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab40a28c0 msgr2=0x7f7ab40a2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab40a28c0 0x7f7ab40a2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 1 -- 172.21.15.98:0/734353864 >> v1:172.21.15.98:6789/0 conn(0x7f7ab40a2180 legacy=0x7f7ab40a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 10 monclient: _finish_hunting 0 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 1 monclient: found mon.noname-e 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.121+0000 7f7ababb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ababb8640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ababb8640 1 -- 172.21.15.98:0/734353864 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab40a31e0 con 0x7f7ab4077760 2024-09-11T03:00:43.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7aba3b7640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab40a28c0 0x7f7ab40a2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ababb8640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7ab4077760 0x7f7ab40a4ec0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009dc0 tx=0x7f7aac01c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=567d62eb29dfc23 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 1 -- 172.21.15.98:0/734353864 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7aac0b2070 con 0x7f7ab4077760 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: dump: 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.134 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:43.135 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: _finish_auth 0 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: _check_auth_tickets 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 1 -- 172.21.15.98:0/734353864 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aac0a5b60 con 0x7f7ab4077760 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 1 -- 172.21.15.98:0/734353864 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7aac0a5d50 con 0x7f7ab4077760 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 5 monclient: authenticate success, global_id 24695 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 4 set_mon_vals no callback set 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7ab93b5640 10 monclient: dump: 2024-09-11T03:00:43.136 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:43.137 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 10 monclient: shutdown 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 -- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7ab4077760 msgr2=0x7f7ab40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7ab4077760 0x7f7ab40a4ec0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f7aac009dc0 tx=0x7f7aac01c850 comp rx=0 tx=0).stop 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 -- 172.21.15.98:0/734353864 shutdown_connections 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7ab4077760 0x7f7ab40a4ec0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 --2- 172.21.15.98:0/734353864 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7ab40a28c0 0x7f7ab40a2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 -- 172.21.15.98:0/734353864 >> 172.21.15.98:0/734353864 conn(0x7f7ab406eed0 msgr2=0x7f7ab406f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 -- 172.21.15.98:0/734353864 shutdown_connections 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.125+0000 7f7abc61b640 1 -- 172.21.15.98:0/734353864 wait complete. 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: build_initial_monmap 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: monmap: 2024-09-11T03:00:43.138 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:43.139 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 librados: starting msgr at 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 librados: starting objecter 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 Processor -- start 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 -- start start 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 librados: setting wanted keys 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 librados: calling monclient init 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: init 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _add_conns ranks=[0,3,4,7,6,5,2,1,8] 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4077760 0x7f7ab4123800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): picked mon.noname-a con 0x7f7ab4077760 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 0x7f7ab4123d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): picked mon.noname-g con 0x7f7ab40a2180 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:43.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7ab40a28c0 0x7f7ab4124280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): picked mon.noname-c con 0x7f7ab40a28c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab41441d0 con 0x7f7ab4077760 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab40a5610 con 0x7f7ab40a2180 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7ab4144350 con 0x7f7ab40a28c0 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient(hunting): authenticate will time out at 1597.009476s 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4077760 0x7f7ab4123800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4077760 0x7f7ab4123800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38206/0 (socket says 172.21.15.98:38206) 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 -- 172.21.15.98:0/1505877971 learned_addr learned my addr 172.21.15.98:0/1505877971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): get_auth_request con 0x7f7ab4077760 auth_method 0 2024-09-11T03:00:43.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7ab40a28c0 0x7f7ab4124280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 10 monclient(hunting): get_auth_request con 0x7f7ab40a28c0 auth_method 0 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 0x7f7ab4123d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:43.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ababb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 10 monclient(hunting): get_auth_request con 0x7f7ab40a2180 auth_method 0 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient(hunting): handle_auth_done global_id 44649 payload 824 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 -- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7ab40a28c0 msgr2=0x7f7ab4124280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 1 -- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 msgr2=0x7f7ab4123d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7ab40a28c0 0x7f7ab4124280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 1 -- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 msgr2=0x7f7ab4123d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 0x7f7ab4123d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 -- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 msgr2=0x7f7ab4123d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab9bb6640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 0x7f7ab4123d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 0x7f7ab4123d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient: _finish_hunting 0 2024-09-11T03:00:43.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 monclient: found mon.noname-a 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 -- 172.21.15.98:0/1505877971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab4126130 con 0x7f7ab4077760 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aba3b7640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4077760 0x7f7ab4123800 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000e980 tx=0x7f7ab000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 client.?.objecter ms_handle_connect 0x7f7ab4077760 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 1 -- 172.21.15.98:0/1505877971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ab0017070 con 0x7f7ab4077760 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: dump: 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.144 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: _finish_auth 0 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: _check_auth_tickets 2024-09-11T03:00:43.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 1 -- 172.21.15.98:0/1505877971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ab000ce70 con 0x7f7ab4077760 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 5 monclient: authenticate success, global_id 44649 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _renew_subs 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab409f810 con 0x7f7ab4077760 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 1 -- 172.21.15.98:0/1505877971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ab00138e0 con 0x7f7ab4077760 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 monclient: dump: 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:43.146 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:43.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7ab8bb4640 4 set_mon_vals no callback set 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 client.44649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _renew_subs 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab4125e50 con 0x7f7ab4077760 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 librados: init done 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ab40a2ca0 con 0x7f7ab4077760 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 1 -- 172.21.15.98:0/1505877971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f7ab0013470 con 0x7f7ab4077760 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7a9803df70 0x7f7a98040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 1 -- 172.21.15.98:0/1505877971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(431..431 src has 1..431) ==== 6202+0+0 (secure 0 0 0) 0x7f7ab000f040 con 0x7f7ab4077760 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 10 client.44649.objecter ms_dispatch 0x7f7ab4000ba0 osd_map(431..431 src has 1..431) 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 3 client.44649.objecter handle_osd_map got epochs [431,431] > 0 2024-09-11T03:00:43.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 3 client.44649.objecter handle_osd_map decoding full epoch 431 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.129+0000 7f7aaaffd640 20 client.44649.objecter dump_active .. 0 homeless 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.133+0000 7f7ab9bb6640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7a9803df70 0x7f7a98040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.133+0000 7f7ab9bb6640 10 monclient: get_auth_request con 0x7f7a9803df70 auth_method 0 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.133+0000 7f7ab9bb6640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7a9803df70 0x7f7a98040430 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4007a70 tx=0x7f7aa4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.133+0000 7f7aaaffd640 10 client.44649.objecter ms_handle_connect 0x7f7a9803df70 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.137+0000 7f7aaaffd640 1 -- 172.21.15.98:0/1505877971 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f7ab0021070 con 0x7f7ab4077760 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.137+0000 7f7aaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:43.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.137+0000 7f7aaaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:43.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.373+0000 7f7abc61b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:00:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.373+0000 7f7abc61b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.373+0000 7f7abc61b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:00:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.373+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7ab4124930 con 0x7f7ab4077760 2024-09-11T03:00:43.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7aaaffd640 1 -- 172.21.15.98:0/1505877971 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3181 (secure 0 0 0) 0x7f7ab001c450 con 0x7f7ab4077760 2024-09-11T03:00:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7aaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:43.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7aaaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:43.376 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":72,"quorum":[0,1,2,4,6,7,8],"quorum_names":["a","f","b","c","d","i","e"],"quorum_leader_name":"e","quorum_age":93,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:00:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 10 librados: watch_flush enter 2024-09-11T03:00:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 10 librados: watch_flush exit 2024-09-11T03:00:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 20 client.44649.objecter shutdown clearing up homeless session... 2024-09-11T03:00:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 10 client.44649.objecter successfully canceled tick 2024-09-11T03:00:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7a9803df70 msgr2=0x7f7a98040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7a9803df70 0x7f7a98040430 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4007a70 tx=0x7f7aa4009040 comp rx=0 tx=0).stop 2024-09-11T03:00:43.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 10 monclient: shutdown 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4077760 msgr2=0x7f7ab4123800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4077760 0x7f7ab4123800 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000e980 tx=0x7f7ab000ee40 comp rx=0 tx=0).stop 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 shutdown_connections 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7a9803df70 0x7f7a98040430 unknown :-1 s=CLOSED pgs=3568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7ab40a28c0 0x7f7ab4124280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7ab40a2180 0x7f7ab4123d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 --2- 172.21.15.98:0/1505877971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ab4077760 0x7f7ab4123800 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 >> 172.21.15.98:0/1505877971 conn(0x7f7ab406eed0 msgr2=0x7f7ab409d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 shutdown_connections 2024-09-11T03:00:43.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 -- 172.21.15.98:0/1505877971 wait complete. 2024-09-11T03:00:43.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:43.377+0000 7f7abc61b640 1 librados: shutdown 2024-09-11T03:00:46.394 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:00:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.521+0000 7fb937c76640 10 monclient: get_monmap_and_config 2024-09-11T03:00:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.521+0000 7fb937c76640 10 monclient: build_initial_monmap 2024-09-11T03:00:46.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.521+0000 7fb937c76640 10 monclient: monmap: 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:46.524594+0000 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:46.524594+0000 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:46.523 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:46.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:46.524 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:46.524 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:46.524 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:46.524 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 Processor -- start 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 -- start start 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient: init 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient: _add_conns ranks=[2,3,6,4,1,0,8,7,10,9,5] 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): picked mon.noname-b con 0x7fb93006f650 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): picked mon.noname-g con 0x7fb9300a8380 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:46.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): picked mon.noname-d con 0x7fb93006ff70 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb9300a8800 con 0x7fb93006f650 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb930071a80 con 0x7fb9300a8380 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb930071d60 con 0x7fb93006ff70 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb937c76640 10 monclient(hunting): authenticate will time out at 1600.405440s 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:60554/0 (socket says 172.21.15.98:60554) 2024-09-11T03:00:46.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 learned_addr learned my addr 172.21.15.98:0/321450743 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 10 monclient(hunting): get_auth_request con 0x7fb93006ff70 auth_method 0 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:41176/0 (socket says 172.21.15.98:41176) 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 10 monclient(hunting): get_auth_request con 0x7fb93006f650 auth_method 0 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:46.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 10 monclient(hunting): get_auth_request con 0x7fb9300a8380 auth_method 0 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:46.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:46.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.525+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:46.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 10 monclient(hunting): get_auth_request con 0x7fb93006ff70 auth_method 0 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 10 monclient(hunting): get_auth_request con 0x7fb93006f650 auth_method 0 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:46.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 10 monclient(hunting): get_auth_request con 0x7fb9300a8380 auth_method 0 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:46.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:00:46.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:00:46.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:46.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:46.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:46.729+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:00:47.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:47.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 10 monclient(hunting): get_auth_request con 0x7fb93006ff70 auth_method 0 2024-09-11T03:00:47.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:47.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 10 monclient(hunting): get_auth_request con 0x7fb93006f650 auth_method 0 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 10 monclient(hunting): get_auth_request con 0x7fb9300a8380 auth_method 0 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:47.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:00:47.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.133+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:00:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 10 monclient(hunting): get_auth_request con 0x7fb93006f650 auth_method 0 2024-09-11T03:00:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:47.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 10 monclient(hunting): get_auth_request con 0x7fb93006ff70 auth_method 0 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:47.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 10 monclient(hunting): get_auth_request con 0x7fb9300a8380 auth_method 0 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:47.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:47.937+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:00:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): tick 2024-09-11T03:00:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): _check_auth_tickets 2024-09-11T03:00:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 monclient(hunting): continuing hunt 2024-09-11T03:00:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T03:00:49.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 msgr2=0x7fb9301397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb93006ff70 0x7fb9301397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 msgr2=0x7fb93006fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb9300a8380 0x7fb93006fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 msgr2=0x7fb930072640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb93006f650 0x7fb930072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient: _add_conns ranks=[10,4,8,7,5,0,9,1] 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 --2- 172.21.15.98:0/321450743 >> v2:172.21.15.190:3300/0 conn(0x7fb91c003860 0x7fb91c005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): picked mon.noname-f con 0x7fb91c003860 addr v2:172.21.15.190:3300/0 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 0x7fb91c0086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): picked mon.noname-c con 0x7fb91c006250 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 0x7fb91c00b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:49.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): picked mon.noname-e con 0x7fb91c008d40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 -- 172.21.15.98:0/321450743 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fb9300a8800 con 0x7fb91c003860 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 -- 172.21.15.98:0/321450743 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb930071a80 con 0x7fb91c006250 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 1 -- 172.21.15.98:0/321450743 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb930071d60 con 0x7fb91c008d40 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb9277fe640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 0x7fb91c0086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 10 monclient(hunting): get_auth_request con 0x7fb91c006250 auth_method 0 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 0x7fb91c00b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 10 monclient(hunting): get_auth_request con 0x7fb91c008d40 auth_method 0 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> v2:172.21.15.190:3300/0 conn(0x7fb91c003860 0x7fb91c005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 10 monclient(hunting): get_auth_request con 0x7fb91c003860 auth_method 0 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 msgr2=0x7fb91c0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 msgr2=0x7fb91c0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 0x7fb91c0086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 msgr2=0x7fb91c00b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:00:49.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935211640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 0x7fb91c0086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 msgr2=0x7fb91c00b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 0x7fb91c00b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> v2:172.21.15.190:3300/0 conn(0x7fb91c003860 msgr2=0x7fb91c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:00:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> v2:172.21.15.190:3300/0 conn(0x7fb91c003860 msgr2=0x7fb91c005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> v2:172.21.15.190:3300/0 conn(0x7fb91c003860 0x7fb91c005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> v2:172.21.15.190:3300/0 conn(0x7fb91c003860 0x7fb91c005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:49.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.525+0000 7fb936213640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 0x7fb91c00b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:49.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> v2:172.21.15.190:3300/0 conn(0x7fb91c003860 0x7fb91c005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): get_auth_request con 0x7fb91c003860 auth_method 0 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient(hunting): handle_auth_done global_id 25516 payload 274 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 msgr2=0x7fb91c00b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb91c008d40 0x7fb91c00b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 msgr2=0x7fb91c0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:49.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb91c006250 0x7fb91c0086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient: _finish_hunting 0 2024-09-11T03:00:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 1 monclient: found mon.noname-f 2024-09-11T03:00:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:00:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb935a12640 1 -- 172.21.15.98:0/321450743 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9300714c0 con 0x7fb91c003860 2024-09-11T03:00:49.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.729+0000 7fb936213640 1 -- 172.21.15.98:0/321450743 reap_dead start 2024-09-11T03:00:49.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.733+0000 7fb935a12640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb91c003860 0x7fb91c005c70 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0036b0 tx=0x7fb92001d6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b501b804b3ac874e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:49.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 1 -- 172.21.15.98:0/321450743 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb92000f040 con 0x7fb91c003860 2024-09-11T03:00:49.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:49.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: dump: 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:49.736 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: _finish_auth 0 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: _check_auth_tickets 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:49.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 1 -- 172.21.15.98:0/321450743 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9200044a0 con 0x7fb91c003860 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 1 -- 172.21.15.98:0/321450743 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb920003a90 con 0x7fb91c003860 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 5 monclient: authenticate success, global_id 25516 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 4 set_mon_vals no callback set 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb934a10640 10 monclient: dump: 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:49.739 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient: shutdown 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 -- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb91c003860 msgr2=0x7fb91c005c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb91c003860 0x7fb91c005c70 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0036b0 tx=0x7fb92001d6f0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 -- 172.21.15.98:0/321450743 shutdown_connections 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 --2- 172.21.15.98:0/321450743 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb91c003860 0x7fb91c005c70 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 -- 172.21.15.98:0/321450743 >> 172.21.15.98:0/321450743 conn(0x7fb9300ac8e0 msgr2=0x7fb9300a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 -- 172.21.15.98:0/321450743 shutdown_connections 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 -- 172.21.15.98:0/321450743 wait complete. 2024-09-11T03:00:49.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient: build_initial_monmap 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient: monmap: 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:49.746 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 librados: starting msgr at 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 librados: starting objecter 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 Processor -- start 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 -- start start 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 librados: setting wanted keys 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 librados: calling monclient init 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient: init 2024-09-11T03:00:49.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient: _add_conns ranks=[7,8,2,1,3,5,4,0,6] 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb91c003860 0x7fb9301d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient(hunting): picked mon.noname-i con 0x7fb91c003860 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb9301d2160 0x7fb9301d25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient(hunting): picked mon.noname-e con 0x7fb9301d2160 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb930123e20 0x7fb930124240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient(hunting): picked mon.noname-b con 0x7fb930123e20 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.737+0000 7fb937c76640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb93013a360 con 0x7fb930123e20 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb93013a1e0 con 0x7fb91c003860 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:49.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb93013a4e0 con 0x7fb9301d2160 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient(hunting): authenticate will time out at 1603.620046s 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb930123e20 0x7fb930124240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb9301d2160 0x7fb9301d25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb930123e20 0x7fb930124240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:44426/0 (socket says 172.21.15.98:44426) 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 1 -- 172.21.15.98:0/1456626571 learned_addr learned my addr 172.21.15.98:0/1456626571 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb9301d2160 0x7fb9301d25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60238/0 (socket says 172.21.15.98:60238) 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): get_auth_request con 0x7fb9301d2160 auth_method 0 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 10 monclient(hunting): get_auth_request con 0x7fb930123e20 auth_method 0 2024-09-11T03:00:49.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935a12640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb91c003860 0x7fb9301d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935a12640 10 monclient(hunting): get_auth_request con 0x7fb91c003860 auth_method 0 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935a12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:49.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935a12640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935a12640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient(hunting): handle_auth_done global_id 24704 payload 824 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 -- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb91c003860 msgr2=0x7fb9301d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb91c003860 0x7fb9301d1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 -- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb930123e20 msgr2=0x7fb930124240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb930123e20 0x7fb930124240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient: _finish_hunting 0 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 monclient: found mon.noname-e 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 -- 172.21.15.98:0/1456626571 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb930126db0 con 0x7fb9301d2160 2024-09-11T03:00:49.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb936213640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb930123e20 0x7fb930124240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935211640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb9301d2160 0x7fb9301d25a0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fb92c015750 tx=0x7fb92c015c10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 client.?.objecter ms_handle_connect 0x7fb9301d2160 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 1 -- 172.21.15.98:0/1456626571 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb92c015e60 con 0x7fb9301d2160 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: dump: 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: _finish_auth 0 2024-09-11T03:00:49.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: _check_auth_tickets 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 5 monclient: authenticate success, global_id 24704 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 1 -- 172.21.15.98:0/1456626571 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb92c017440 con 0x7fb9301d2160 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient: _renew_subs 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb930127c20 con 0x7fb9301d2160 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 1 -- 172.21.15.98:0/1456626571 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb92c017a60 con 0x7fb9301d2160 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb927fff640 4 set_mon_vals no callback set 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 client.24704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:49.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 monclient: dump: 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:49.755 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient: _renew_subs 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fb930127940 con 0x7fb9301d2160 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 1 librados: init done 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 1 -- 172.21.15.98:0/1456626571 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb92c029030 con 0x7fb9301d2160 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb93013cc00 con 0x7fb9301d2160 2024-09-11T03:00:49.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb91c03f230 0x7fb91c0416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 1 -- 172.21.15.98:0/1456626571 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(431..431 src has 1..431) ==== 6202+0+0 (secure 0 0 0) 0x7fb92c00f040 con 0x7fb9301d2160 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 10 client.24704.objecter ms_dispatch 0x7fb930000ba0 osd_map(431..431 src has 1..431) 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 3 client.24704.objecter handle_osd_map got epochs [431,431] > 0 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 3 client.24704.objecter handle_osd_map decoding full epoch 431 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb9267fc640 20 client.24704.objecter dump_active .. 0 homeless 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935a12640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb91c03f230 0x7fb91c0416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.741+0000 7fb935a12640 10 monclient: get_auth_request con 0x7fb91c03f230 auth_method 0 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.745+0000 7fb9267fc640 10 client.24704.objecter ms_handle_connect 0x7fb91c03f230 2024-09-11T03:00:49.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.745+0000 7fb935a12640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb91c03f230 0x7fb91c0416f0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7fb9200ad000 tx=0x7fb9200046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:49.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.749+0000 7fb9267fc640 1 -- 172.21.15.98:0/1456626571 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fb92c00fb60 con 0x7fb9301d2160 2024-09-11T03:00:49.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.749+0000 7fb9267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:49.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.749+0000 7fb9267fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:49.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:00:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:00:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb9301c42d0 con 0x7fb9301d2160 2024-09-11T03:00:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb9267fc640 1 -- 172.21.15.98:0/1456626571 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7fb92c017c00 con 0x7fb9301d2160 2024-09-11T03:00:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb9267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:49.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb9267fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:49.985 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":74,"quorum":[0,1,2,3,4,6,7,8],"quorum_names":["a","f","b","g","c","d","i","e"],"quorum_leader_name":"e","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 10 librados: watch_flush enter 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 10 librados: watch_flush exit 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 20 client.24704.objecter shutdown clearing up homeless session... 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 10 client.24704.objecter successfully canceled tick 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb91c03f230 msgr2=0x7fb91c0416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb91c03f230 0x7fb91c0416f0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7fb9200ad000 tx=0x7fb9200046d0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 10 monclient: shutdown 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb9301d2160 msgr2=0x7fb9301d25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:49.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb9301d2160 0x7fb9301d25a0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fb92c015750 tx=0x7fb92c015c10 comp rx=0 tx=0).stop 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 shutdown_connections 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb91c03f230 0x7fb91c0416f0 unknown :-1 s=CLOSED pgs=3570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb930123e20 0x7fb930124240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb9301d2160 0x7fb9301d25a0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 --2- 172.21.15.98:0/1456626571 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb91c003860 0x7fb9301d1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 >> 172.21.15.98:0/1456626571 conn(0x7fb9300ac8e0 msgr2=0x7fb9300aa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 shutdown_connections 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.985+0000 7fb937c76640 1 -- 172.21.15.98:0/1456626571 wait complete. 2024-09-11T03:00:49.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:49.989+0000 7fb937c76640 1 librados: shutdown 2024-09-11T03:00:53.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:00:53.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient: get_monmap_and_config 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient: build_initial_monmap 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient: monmap: 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:53.138552+0000 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:53.138552+0000 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:53.137 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:53.138 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:53.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 1 Processor -- start 2024-09-11T03:00:53.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 1 -- start start 2024-09-11T03:00:53.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient: init 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient: _add_conns ranks=[10,0,5,3,9,8,7,2,4,6,1] 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f19400a4710 0x7f19400a4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient(hunting): picked mon.noname-f con 0x7f19400a4710 addr v2:172.21.15.190:3300/0 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19400a5030 addr v1:172.21.15.98:6789/0 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 0x7f1940141d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient(hunting): picked mon.noname-h con 0x7f1940139fd0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1940077320 con 0x7f19400a5030 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f19400a14d0 con 0x7f19400a4710 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1940076ee0 con 0x7f1940139fd0 2024-09-11T03:00:53.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 10 monclient(hunting): authenticate will time out at 1607.019014s 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f19400a5030 0x7f19401398c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46992/0 (socket says 172.21.15.98:46992) 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193effd640 1 -- 172.21.15.98:0/1516708709 learned_addr learned my addr 172.21.15.98:0/1516708709 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 1 --2- 172.21.15.98:0/1516708709 >> v2:172.21.15.190:3300/0 conn(0x7f19400a4710 0x7f19400a4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 0x7f1940141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 10 monclient(hunting): get_auth_request con 0x7f19400a4710 auth_method 0 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 10 monclient(hunting): get_auth_request con 0x7f1940139fd0 auth_method 0 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:53.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436744549 0 0) 0x7f1940077320 con 0x7f19400a5030 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): _negotiate 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): my global_id is 44658 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f192c003530 con 0x7f19400a5030 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 1 -- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 msgr2=0x7f1940141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 1 -- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 msgr2=0x7f1940141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 0x7f1940141d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:00:53.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193ffff640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 0x7f1940141d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2186311710 0 0) 0x7f192c003530 con 0x7f19400a5030 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 msgr2=0x7f1940141d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940139fd0 0x7f1940141d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 >> v2:172.21.15.190:3300/0 conn(0x7f19400a4710 msgr2=0x7f19400a4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 --2- 172.21.15.98:0/1516708709 >> v2:172.21.15.190:3300/0 conn(0x7f19400a4710 0x7f19400a4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: _finish_hunting 0 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19400775a0 con 0x7f19400a5030 2024-09-11T03:00:53.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: _finish_auth 0 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f1928001970 con 0x7f19400a5030 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 5 monclient: authenticate success, global_id 44658 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: dump: 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.153 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:00:53.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 >> v1:172.21.15.98:6789/0 conn(0x7f19400a5030 legacy=0x7f19401398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient: _add_conns ranks=[4,7,3,0,2,5,1,8,6] 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f192c004240 0x7f192c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): picked mon.c con 0x7f192c004240 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f192c004da0 0x7f192c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): picked mon.i con 0x7f192c004da0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c009b80 0x7f192c00c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): picked mon.g con 0x7f192c009b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f19400a14d0 con 0x7f192c009b80 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1940076ee0 con 0x7f192c004240 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f192c00c920 con 0x7f192c004da0 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f193e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.137+0000 7f1945fee640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f192c004240 0x7f192c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): get_auth_request con 0x7f192c004240 auth_method 0 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:53.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193effd640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f192c004da0 0x7f192c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193ffff640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c009b80 0x7f192c00c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient(hunting): handle_auth_done global_id 44658 payload 293 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 -- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f192c004da0 msgr2=0x7f192c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f192c004da0 0x7f192c005220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 -- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c009b80 msgr2=0x7f192c00c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c009b80 0x7f192c00c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:00:53.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 monclient: found mon.c 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 -- 172.21.15.98:0/1516708709 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192c003530 con 0x7f192c004240 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193ffff640 1 -- 172.21.15.98:0/1516708709 reap_dead start 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient: _finish_auth 0 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193f7fe640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f192c004240 0x7f192c004810 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f1930009ae0 tx=0x7f193001ca50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=4e0414f023e633ff server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19300b2070 con 0x7f192c004240 2024-09-11T03:00:53.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 10 monclient: dump: 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:53.159 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19300a59a0 con 0x7f192c004240 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:53.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 1 -- 172.21.15.98:0/1516708709 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19300096b0 con 0x7f192c004240 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 4 set_mon_vals no callback set 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f193e7fc640 10 monclient: dump: 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:53.161 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:53.162 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:53.162 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:53.162 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:53.162 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:53.162 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:53.164 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:53.164 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:53.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 10 monclient: shutdown 2024-09-11T03:00:53.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:53.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 1 -- 172.21.15.98:0/1516708709 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f192c004240 msgr2=0x7f192c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:53.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f192c004240 0x7f192c004810 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f1930009ae0 tx=0x7f193001ca50 comp rx=0 tx=0).stop 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 1 -- 172.21.15.98:0/1516708709 shutdown_connections 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 1 --2- 172.21.15.98:0/1516708709 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f192c004240 0x7f192c004810 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 1 -- 172.21.15.98:0/1516708709 >> 172.21.15.98:0/1516708709 conn(0x7f194006ed00 msgr2=0x7f194006f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 1 -- 172.21.15.98:0/1516708709 shutdown_connections 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 1 -- 172.21.15.98:0/1516708709 wait complete. 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 10 monclient: build_initial_monmap 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.141+0000 7f1945fee640 10 monclient: monmap: 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:53.165 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 librados: starting msgr at 2024-09-11T03:00:53.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 librados: starting objecter 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 Processor -- start 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 -- start start 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 librados: setting wanted keys 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 librados: calling monclient init 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: init 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _add_conns ranks=[3,4,5,7,8,6,1,0,2] 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c004240 0x7f194009eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): picked mon.noname-g con 0x7f192c004240 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1940123760 0x7f19401389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): picked mon.noname-c con 0x7f1940123760 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940123bc0 0x7f1940138ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): picked mon.noname-h con 0x7f1940123bc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f19400777e0 con 0x7f192c004240 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f19400517e0 con 0x7f1940123760 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1940077960 con 0x7f1940123bc0 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient(hunting): authenticate will time out at 1607.024898s 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1940123760 0x7f19401389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1940123760 0x7f19401389b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:46510/0 (socket says 172.21.15.98:46510) 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 -- 172.21.15.98:0/2494266570 learned_addr learned my addr 172.21.15.98:0/2494266570 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): get_auth_request con 0x7f1940123760 auth_method 0 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:53.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient(hunting): handle_auth_done global_id 24205 payload 824 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 -- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940123bc0 msgr2=0x7f1940138ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940123bc0 0x7f1940138ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 -- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c004240 msgr2=0x7f194009eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c004240 0x7f194009eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient: _finish_hunting 0 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 monclient: found mon.noname-c 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:53.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 -- 172.21.15.98:0/2494266570 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194009f4d0 con 0x7f1940123760 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193effd640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1940123760 0x7f19401389b0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f192800d750 tx=0x7f192800dc10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 client.?.objecter ms_handle_connect 0x7f1940123760 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 1 -- 172.21.15.98:0/2494266570 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f192800de60 con 0x7f1940123760 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: dump: 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:53.170 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:53.171 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: _finish_auth 0 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: _check_auth_tickets 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 1 -- 172.21.15.98:0/2494266570 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19280047f0 con 0x7f1940123760 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 5 monclient: authenticate success, global_id 24205 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _renew_subs 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 1 -- 172.21.15.98:0/2494266570 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1928004e10 con 0x7f1940123760 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1940125ad0 con 0x7f1940123760 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:53.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193dffb640 4 set_mon_vals no callback set 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 client.24205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 monclient: dump: 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:53.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _renew_subs 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f19401d1750 con 0x7f1940123760 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 librados: init done 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:53.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19401398c0 con 0x7f1940123760 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 1 -- 172.21.15.98:0/2494266570 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f19280049a0 con 0x7f1940123760 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f192003df40 0x7f1920040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 1 -- 172.21.15.98:0/2494266570 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(432..432 src has 1..432) ==== 6258+0+0 (secure 0 0 0) 0x7f192804f8f0 con 0x7f1940123760 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 10 client.24205.objecter ms_dispatch 0x7f1940000ba0 osd_map(432..432 src has 1..432) 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 3 client.24205.objecter handle_osd_map got epochs [432,432] > 0 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 3 client.24205.objecter handle_osd_map decoding full epoch 432 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f1937fff640 20 client.24205.objecter dump_active .. 0 homeless 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193f7fe640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f192003df40 0x7f1920040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.145+0000 7f193f7fe640 10 monclient: get_auth_request con 0x7f192003df40 auth_method 0 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.149+0000 7f1937fff640 10 client.24205.objecter ms_handle_connect 0x7f192003df40 2024-09-11T03:00:53.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.149+0000 7f193f7fe640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f192003df40 0x7f1920040400 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7f19300af000 tx=0x7f1930004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.149+0000 7f1937fff640 1 -- 172.21.15.98:0/2494266570 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f1928007540 con 0x7f1940123760 2024-09-11T03:00:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.149+0000 7f1937fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:53.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.149+0000 7f1937fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.389+0000 7f1945fee640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:00:53.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.389+0000 7f1945fee640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.389+0000 7f1945fee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.389+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1940069090 con 0x7f1940123760 2024-09-11T03:00:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1937fff640 1 -- 172.21.15.98:0/2494266570 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7f192800bb80 con 0x7f1940123760 2024-09-11T03:00:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1937fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:53.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1937fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:53.392 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":74,"quorum":[0,1,2,3,4,6,7,8],"quorum_names":["a","f","b","g","c","d","i","e"],"quorum_leader_name":"e","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:00:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 10 librados: watch_flush enter 2024-09-11T03:00:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 10 librados: watch_flush exit 2024-09-11T03:00:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 20 client.24205.objecter shutdown clearing up homeless session... 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 10 client.24205.objecter successfully canceled tick 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f192003df40 msgr2=0x7f1920040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f192003df40 0x7f1920040400 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7f19300af000 tx=0x7f1930004740 comp rx=0 tx=0).stop 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 10 monclient: shutdown 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1940123760 msgr2=0x7f19401389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1940123760 0x7f19401389b0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f192800d750 tx=0x7f192800dc10 comp rx=0 tx=0).stop 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 shutdown_connections 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f192003df40 0x7f1920040400 unknown :-1 s=CLOSED pgs=3571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1940123bc0 0x7f1940138ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1940123760 0x7f19401389b0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 --2- 172.21.15.98:0/2494266570 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f192c004240 0x7f194009eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 >> 172.21.15.98:0/2494266570 conn(0x7f194006ed00 msgr2=0x7f19401c4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 shutdown_connections 2024-09-11T03:00:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 -- 172.21.15.98:0/2494266570 wait complete. 2024-09-11T03:00:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:53.393+0000 7f1945fee640 1 librados: shutdown 2024-09-11T03:00:56.411 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:00:56.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient: get_monmap_and_config 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient: build_initial_monmap 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient: monmap: 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:56.542636+0000 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:56.542636+0000 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:56.541 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:56.542 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 Processor -- start 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 -- start start 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient: init 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient: _add_conns ranks=[8,10,2,1,5,6,4,0,3,9,7] 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe094076b00 0x7fe09409d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): picked mon.noname-e con 0x7fe094076b00 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe094076ee0 0x7fe0940772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): picked mon.noname-f con 0x7fe094076ee0 addr v2:172.21.15.190:3300/0 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe094137b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): picked mon.noname-b con 0x7fe0941376e0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe09409fef0 con 0x7fe094076ee0 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe0940a40d0 con 0x7fe0941376e0 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe0940a3180 con 0x7fe094076b00 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe09ace7640 10 monclient(hunting): authenticate will time out at 1610.422918s 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe094076b00 0x7fe09409d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe094137b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe094076b00 0x7fe09409d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:60242/0 (socket says 172.21.15.98:60242) 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 -- 172.21.15.98:0/2413183061 learned_addr learned my addr 172.21.15.98:0/2413183061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:56.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): get_auth_request con 0x7fe094076b00 auth_method 0 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 10 monclient(hunting): get_auth_request con 0x7fe0941376e0 auth_method 0 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe099284640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe093fff640 1 --2- 172.21.15.98:0/2413183061 >> v2:172.21.15.190:3300/0 conn(0x7fe094076ee0 0x7fe0940772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient(hunting): handle_auth_done global_id 24713 payload 274 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 -- 172.21.15.98:0/2413183061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 msgr2=0x7fe094137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 --2- 172.21.15.98:0/2413183061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe094137b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 -- 172.21.15.98:0/2413183061 >> v2:172.21.15.190:3300/0 conn(0x7fe094076ee0 msgr2=0x7fe0940772c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 --2- 172.21.15.98:0/2413183061 >> v2:172.21.15.190:3300/0 conn(0x7fe094076ee0 0x7fe0940772c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient: _finish_hunting 0 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 monclient: found mon.noname-e 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:00:56.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe098a83640 1 -- 172.21.15.98:0/2413183061 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0940a3e20 con 0x7fe094076b00 2024-09-11T03:00:56.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe093fff640 10 monclient: get_auth_request con 0x7fe094076ee0 auth_method 0 2024-09-11T03:00:56.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.541+0000 7fe093fff640 1 --2- 172.21.15.98:0/2413183061 >> v2:172.21.15.190:3300/0 conn(0x7fe094076ee0 0x7fe0940772c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:56.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe098a83640 1 --2- 172.21.15.98:0/2413183061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe094076b00 0x7fe09409d540 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098a0 tx=0x7fe08401c400 comp rx=0 tx=0).ready entity=mon.8 client_cookie=1f59cb4d82fe105c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 1 -- 172.21.15.98:0/2413183061 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0840b0070 con 0x7fe094076b00 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: dump: 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.553 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: _finish_auth 0 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: _check_auth_tickets 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 5 monclient: authenticate success, global_id 24713 2024-09-11T03:00:56.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 1 -- 172.21.15.98:0/2413183061 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0840aa810 con 0x7fe094076b00 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 4 set_mon_vals no callback set 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 1 -- 172.21.15.98:0/2413183061 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0840a9300 con 0x7fe094076b00 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe0937fe640 10 monclient: dump: 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.555 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient: shutdown 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- 172.21.15.98:0/2413183061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe094076b00 msgr2=0x7fe09409d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 --2- 172.21.15.98:0/2413183061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe094076b00 0x7fe09409d540 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fe0840098a0 tx=0x7fe08401c400 comp rx=0 tx=0).stop 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- 172.21.15.98:0/2413183061 shutdown_connections 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 --2- 172.21.15.98:0/2413183061 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe094137b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 --2- 172.21.15.98:0/2413183061 >> v2:172.21.15.190:3300/0 conn(0x7fe094076ee0 0x7fe0940772c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 --2- 172.21.15.98:0/2413183061 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe094076b00 0x7fe09409d540 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- 172.21.15.98:0/2413183061 >> 172.21.15.98:0/2413183061 conn(0x7fe0940a6430 msgr2=0x7fe09406ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- 172.21.15.98:0/2413183061 shutdown_connections 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- 172.21.15.98:0/2413183061 wait complete. 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient: build_initial_monmap 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient: monmap: 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:56.559 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 librados: starting msgr at 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 librados: starting objecter 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 Processor -- start 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- start start 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 librados: setting wanted keys 2024-09-11T03:00:56.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 librados: calling monclient init 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient: init 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient: _add_conns ranks=[3,7,2,6,8,1,4,0,5] 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe094076b00 0x7fe094135e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): picked mon.noname-g con 0x7fe094076b00 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe094076ee0 0x7fe094136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): picked mon.noname-i con 0x7fe094076ee0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe0941368d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): picked mon.noname-b con 0x7fe0941376e0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe094141340 con 0x7fe0941376e0 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe0941411c0 con 0x7fe094076b00 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe0941414c0 con 0x7fe094076ee0 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.545+0000 7fe09ace7640 10 monclient(hunting): authenticate will time out at 1610.427942s 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe0941368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe0941368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:44454/0 (socket says 172.21.15.98:44454) 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 -- 172.21.15.98:0/71434470 learned_addr learned my addr 172.21.15.98:0/71434470 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): get_auth_request con 0x7fe0941376e0 auth_method 0 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient(hunting): handle_auth_done global_id 64334 payload 824 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe098a83640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe094076b00 0x7fe094135e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 -- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe094076ee0 msgr2=0x7fe094136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe094076ee0 0x7fe094136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 -- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe094076b00 msgr2=0x7fe094135e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe094076b00 0x7fe094135e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient: _finish_hunting 0 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 monclient: found mon.noname-b 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 -- 172.21.15.98:0/71434470 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09413d9c0 con 0x7fe0941376e0 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe099284640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe0941368d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00efc0 tx=0x7fe08c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 client.?.objecter ms_handle_connect 0x7fe0941376e0 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 1 -- 172.21.15.98:0/71434470 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe08c018020 con 0x7fe0941376e0 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: dump: 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.564 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.565 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: _finish_auth 0 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: _check_auth_tickets 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 1 -- 172.21.15.98:0/71434470 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe08c00cd30 con 0x7fe0941376e0 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 1 -- 172.21.15.98:0/71434470 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe08c014d60 con 0x7fe0941376e0 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 monclient: dump: 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.566 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.567 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe092ffd640 4 set_mon_vals no callback set 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 5 monclient: authenticate success, global_id 64334 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 monclient: _renew_subs 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe094123ca0 con 0x7fe0941376e0 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 client.64334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 monclient: _renew_subs 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe094123f80 con 0x7fe0941376e0 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 1 librados: init done 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:56.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0940772c0 con 0x7fe0941376e0 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 1 -- 172.21.15.98:0/71434470 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe08c014480 con 0x7fe0941376e0 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe08003df40 0x7fe080040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 1 -- 172.21.15.98:0/71434470 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7fe08c050540 con 0x7fe0941376e0 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 client.64334.objecter ms_dispatch 0x7fe094000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 3 client.64334.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 3 client.64334.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 20 client.64334.objecter dump_active .. 0 homeless 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe098a83640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe08003df40 0x7fe080040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe098a83640 10 monclient: get_auth_request con 0x7fe08003df40 auth_method 0 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe0917fa640 10 client.64334.objecter ms_handle_connect 0x7fe08003df40 2024-09-11T03:00:56.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.549+0000 7fe098a83640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe08003df40 0x7fe080040400 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7fe0840ad000 tx=0x7fe0840a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.553+0000 7fe0917fa640 1 -- 172.21.15.98:0/71434470 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe08c022070 con 0x7fe0941376e0 2024-09-11T03:00:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.553+0000 7fe0917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:56.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.553+0000 7fe0917fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:56.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.792+0000 7fe09ace7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:00:56.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.792+0000 7fe09ace7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.792+0000 7fe09ace7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.792+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe094069190 con 0x7fe0941376e0 2024-09-11T03:00:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.792+0000 7fe0917fa640 1 -- 172.21.15.98:0/71434470 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fe09413d9c0 con 0x7fe0941376e0 2024-09-11T03:00:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.792+0000 7fe0917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:00:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.792+0000 7fe0917fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:56.795 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:00:56.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 10 librados: watch_flush enter 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 10 librados: watch_flush exit 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 20 client.64334.objecter shutdown clearing up homeless session... 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 10 client.64334.objecter successfully canceled tick 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe08003df40 msgr2=0x7fe080040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe08003df40 0x7fe080040400 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7fe0840ad000 tx=0x7fe0840a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 10 monclient: shutdown 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 msgr2=0x7fe0941368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe0941368d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00efc0 tx=0x7fe08c00c490 comp rx=0 tx=0).stop 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 shutdown_connections 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe08003df40 0x7fe080040400 unknown :-1 s=CLOSED pgs=3572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe0941376e0 0x7fe0941368d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe094076ee0 0x7fe094136390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 --2- 172.21.15.98:0/71434470 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe094076b00 0x7fe094135e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 >> 172.21.15.98:0/71434470 conn(0x7fe0940a6430 msgr2=0x7fe09409df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 shutdown_connections 2024-09-11T03:00:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 -- 172.21.15.98:0/71434470 wait complete. 2024-09-11T03:00:56.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.796+0000 7fe09ace7640 1 librados: shutdown 2024-09-11T03:00:56.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:00:56.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T03:00:56.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:00:56.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient: get_monmap_and_config 2024-09-11T03:00:56.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient: build_initial_monmap 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient: monmap: 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:56.930025+0000 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:56.930025+0000 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 Processor -- start 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 -- start start 2024-09-11T03:00:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient: init 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient: _add_conns ranks=[3,9,6,7,2,8,0,4,5,10,1] 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8139b80 0x7f46a8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): picked mon.noname-g con 0x7f46a8139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46a806fe50 0x7f46a80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): picked mon.noname-a con 0x7f46a806fe50 addr v2:172.21.15.98:3300/0 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46a8072640 0x7f46a80a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): picked mon.noname-d con 0x7f46a8072640 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f46a80719c0 con 0x7f46a806fe50 2024-09-11T03:00:56.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f46a8071c10 con 0x7f46a8139b80 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f46a80702d0 con 0x7f46a8072640 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46accf5640 10 monclient(hunting): authenticate will time out at 1610.810246s 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46a806fe50 0x7f46a80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46a8072640 0x7f46a80a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f46a806fe50 0x7f46a80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42716/0 (socket says 172.21.15.98:42716) 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 1 -- 172.21.15.98:0/3603750340 learned_addr learned my addr 172.21.15.98:0/3603750340 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:56.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 10 monclient(hunting): get_auth_request con 0x7f46a806fe50 auth_method 0 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): get_auth_request con 0x7f46a8072640 auth_method 0 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a659c640 1 --2- 172.21.15.98:0/3603750340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8139b80 0x7f46a8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a659c640 10 monclient(hunting): get_auth_request con 0x7f46a8139b80 auth_method 0 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient(hunting): handle_auth_done global_id 44511 payload 274 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 1 -- 172.21.15.98:0/3603750340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8139b80 msgr2=0x7f46a8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 1 --2- 172.21.15.98:0/3603750340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8139b80 0x7f46a8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 1 -- 172.21.15.98:0/3603750340 >> v2:172.21.15.98:3300/0 conn(0x7f46a806fe50 msgr2=0x7f46a80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 1 --2- 172.21.15.98:0/3603750340 >> v2:172.21.15.98:3300/0 conn(0x7f46a806fe50 0x7f46a80a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient: _finish_hunting 0 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 1 monclient: found mon.noname-d 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a6d9d640 1 -- 172.21.15.98:0/3603750340 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a8071370 con 0x7f46a8072640 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.928+0000 7f46a5d9b640 1 --2- 172.21.15.98:0/3603750340 >> v2:172.21.15.98:3300/0 conn(0x7f46a806fe50 0x7f46a80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a6d9d640 1 --2- 172.21.15.98:0/3603750340 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46a8072640 0x7f46a80a7f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f46980097c0 tx=0x7f469801c360 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a1c16175b6c7ff93 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 1 -- 172.21.15.98:0/3603750340 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46980b0070 con 0x7f46a8072640 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: dump: 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.947 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: _finish_auth 0 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: _check_auth_tickets 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 1 -- 172.21.15.98:0/3603750340 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46980aa810 con 0x7f46a8072640 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 5 monclient: authenticate success, global_id 44511 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:56.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 1 -- 172.21.15.98:0/3603750340 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46980a9300 con 0x7f46a8072640 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 4 set_mon_vals no callback set 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a559a640 10 monclient: dump: 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.949 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient: shutdown 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:56.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- 172.21.15.98:0/3603750340 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46a8072640 msgr2=0x7f46a80a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 --2- 172.21.15.98:0/3603750340 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46a8072640 0x7f46a80a7f50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f46980097c0 tx=0x7f469801c360 comp rx=0 tx=0).stop 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- 172.21.15.98:0/3603750340 shutdown_connections 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 --2- 172.21.15.98:0/3603750340 >> v2:172.21.15.98:3300/0 conn(0x7f46a806fe50 0x7f46a80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 --2- 172.21.15.98:0/3603750340 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f46a8072640 0x7f46a80a7f50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 --2- 172.21.15.98:0/3603750340 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8139b80 0x7f46a8139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- 172.21.15.98:0/3603750340 >> 172.21.15.98:0/3603750340 conn(0x7f46a80ac730 msgr2=0x7f46a80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- 172.21.15.98:0/3603750340 shutdown_connections 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- 172.21.15.98:0/3603750340 wait complete. 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient: build_initial_monmap 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient: monmap: 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:56.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:56.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 librados: starting msgr at 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 librados: starting objecter 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 Processor -- start 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- start start 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 librados: setting wanted keys 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 librados: calling monclient init 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient: init 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient: _add_conns ranks=[2,3,7,8,4,5,6,1,0] 2024-09-11T03:00:56.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a806fe50 0x7f46a81349d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): picked mon.noname-b con 0x7f46a806fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8072640 0x7f46a8134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): picked mon.noname-g con 0x7f46a8072640 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f46a8139b80 0x7f46a813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): picked mon.noname-i con 0x7f46a8139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f46a81435c0 con 0x7f46a806fe50 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f46a8143440 con 0x7f46a8072640 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f46a8143740 con 0x7f46a8139b80 2024-09-11T03:00:56.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46accf5640 10 monclient(hunting): authenticate will time out at 1610.815669s 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a806fe50 0x7f46a81349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a806fe50 0x7f46a81349d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:44466/0 (socket says 172.21.15.98:44466) 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 1 -- 172.21.15.98:0/3171668613 learned_addr learned my addr 172.21.15.98:0/3171668613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 10 monclient(hunting): get_auth_request con 0x7f46a806fe50 auth_method 0 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.932+0000 7f46a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a6d9d640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f46a8139b80 0x7f46a813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8072640 0x7f46a8134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 10 monclient(hunting): handle_auth_done global_id 64343 payload 824 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f46a8139b80 msgr2=0x7f46a813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f46a8139b80 0x7f46a813a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8072640 msgr2=0x7f46a8134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8072640 0x7f46a8134f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 10 monclient: _finish_hunting 0 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 1 monclient: found mon.noname-b 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 1 -- 172.21.15.98:0/3171668613 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a81357a0 con 0x7f46a806fe50 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a6d9d640 10 monclient: get_auth_request con 0x7f46a8139b80 auth_method 0 2024-09-11T03:00:56.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient: get_auth_request con 0x7f46a8072640 auth_method 0 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a6d9d640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f46a8139b80 0x7f46a813a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f46a8072640 0x7f46a8134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a659c640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a806fe50 0x7f46a81349d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f46a0007e90 tx=0x7f46a000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 client.?.objecter ms_handle_connect 0x7f46a806fe50 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46a000ee10 con 0x7f46a806fe50 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: dump: 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.957 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.961 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.962 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: _finish_auth 0 2024-09-11T03:00:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: _check_auth_tickets 2024-09-11T03:00:56.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 5 monclient: authenticate success, global_id 64343 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a0019070 con 0x7f46a806fe50 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46a0014d00 con 0x7f46a806fe50 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 10 monclient: _renew_subs 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a80aa2c0 con 0x7f46a806fe50 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a4d99640 4 set_mon_vals no callback set 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: dump: 2024-09-11T03:00:56.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.965 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 10 client.64343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:56.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 10 monclient: _renew_subs 2024-09-11T03:00:56.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:56.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a813ebc0 con 0x7f46a806fe50 2024-09-11T03:00:56.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 1 librados: init done 2024-09-11T03:00:56.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813abf0 0x7f46a813afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806ca10 con 0x7f46a813abf0 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813abf0 0x7f46a813afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient: get_auth_request con 0x7f46a813abf0 auth_method 0 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f46a0014420 con 0x7f46a806fe50 2024-09-11T03:00:56.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_done global_id 44676 payload 824 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 1 -- 172.21.15.98:0/3171668613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46a8139f60 con 0x7f46a813abf0 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f468803df40 0x7f4688040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7f46a004ffb0 con 0x7f46a806fe50 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 client.64343.objecter ms_dispatch 0x7f46a8000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 3 client.64343.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 3 client.64343.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a5d9b640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813abf0 0x7f46a813afd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f46a80a5cf0 tx=0x7f4690010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 20 client.64343.objecter dump_active .. 0 homeless 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 client.64343.objecter ms_handle_connect 0x7f46a813abf0 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 client.64343.objecter resend_mon_ops 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f469000f040 con 0x7f46a813abf0 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: dump: 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.968 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a6d9d640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f468803df40 0x7f4688040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a6d9d640 10 monclient: get_auth_request con 0x7f468803df40 auth_method 0 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4690010e70 con 0x7f46a813abf0 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813abf0 msgr2=0x7f46a813afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813abf0 0x7f46a813afd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f46a80a5cf0 tx=0x7f4690010c20 comp rx=0 tx=0).stop 2024-09-11T03:00:56.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813abf0 msgr2=0x7f46a813afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813abf0 0x7f46a813afd0 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f46a80a5cf0 tx=0x7f4690010c20 comp rx=0 tx=0).stop 2024-09-11T03:00:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f46a6d9d640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f468803df40 0x7f4688040400 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f46980ad000 tx=0x7f46980a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.936+0000 7f469effd640 10 client.64343.objecter ms_handle_connect 0x7f468803df40 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813b510 0x7f46a80a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f46a806e870 con 0x7f46a813b510 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813b510 0x7f46a80a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient: get_auth_request con 0x7f46a813b510 auth_method 0 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:56.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:56.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 10 monclient(hunting): handle_auth_done global_id 44685 payload 824 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 1 -- 172.21.15.98:0/3171668613 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46a813dcf0 con 0x7f46a813b510 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 10 client.64343.objecter ms_handle_connect 0x7f46a813b510 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a5d9b640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813b510 0x7f46a80a0e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46a80a5cf0 tx=0x7f46900046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 10 client.64343.objecter resend_mon_ops 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4690025070 con 0x7f46a813b510 2024-09-11T03:00:56.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 10 monclient: dump: 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:56.975 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f4690006040 con 0x7f46a813b510 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:56.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813b510 msgr2=0x7f46a80a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813b510 0x7f46a80a0e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46a80a5cf0 tx=0x7f46900046d0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813b510 msgr2=0x7f46a80a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f469effd640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f46a813b510 0x7f46a80a0e60 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46a80a5cf0 tx=0x7f46900046d0 comp rx=0 tx=0).stop 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:56.977 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 283384, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:56.978 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:56.979 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:56.980 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:56.981 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:56.982 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:56.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:56.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:56.987 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:56.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.994 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.996 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:56.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:56.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.000 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.001 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.003 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:57.003 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 10 librados: watch_flush enter 2024-09-11T03:00:57.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 10 librados: watch_flush exit 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 20 client.64343.objecter shutdown clearing up homeless session... 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 10 client.64343.objecter successfully canceled tick 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f468803df40 msgr2=0x7f4688040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f468803df40 0x7f4688040400 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f46980ad000 tx=0x7f46980a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 10 monclient: shutdown 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a806fe50 msgr2=0x7f46a81349d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 --2- 172.21.15.98:0/3171668613 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f46a806fe50 0x7f46a81349d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f46a0007e90 tx=0x7f46a000e4b0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a6d9d640 1 -- 172.21.15.98:0/3171668613 reap_dead start 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46a6d9d640 1 -- 172.21.15.98:0/3171668613 reap_dead start 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 shutdown_connections 2024-09-11T03:00:57.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 >> 172.21.15.98:0/3171668613 conn(0x7f46a80ac730 msgr2=0x7f46a81359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 shutdown_connections 2024-09-11T03:00:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 -- 172.21.15.98:0/3171668613 wait complete. 2024-09-11T03:00:57.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:56.944+0000 7f46accf5640 1 librados: shutdown 2024-09-11T03:00:57.006 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:57.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T03:00:57.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:00:57.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient: get_monmap_and_config 2024-09-11T03:00:57.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient: monmap: 2024-09-11T03:00:57.127 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.127 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:57.129403+0000 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:57.129403+0000 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.128 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.129 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.129 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:57.129 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:57.129 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 Processor -- start 2024-09-11T03:00:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 -- start start 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient: init 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient: _add_conns ranks=[9,6,4,3,5,2,8,7,10,0,1] 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb8d81397d0 0x7fb8d8139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): picked mon.noname-a con 0x7fb8d81397d0 addr v2:172.21.15.98:3300/0 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb8d813a0f0 0x7fb8d80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): picked mon.noname-d con 0x7fb8d813a0f0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d809d4c0 0x7fb8d80a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): picked mon.noname-c con 0x7fb8d809d4c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb8d809cbe0 con 0x7fb8d81397d0 2024-09-11T03:00:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb8d809e160 con 0x7fb8d809d4c0 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb8d809de80 con 0x7fb8d813a0f0 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8de201640 10 monclient(hunting): authenticate will time out at 1611.009393s 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb8d813a0f0 0x7fb8d80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb8d813a0f0 0x7fb8d80a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:60884/0 (socket says 172.21.15.98:60884) 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 1 -- 172.21.15.98:0/2604148338 learned_addr learned my addr 172.21.15.98:0/2604148338 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 10 monclient(hunting): get_auth_request con 0x7fb8d813a0f0 auth_method 0 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2604148338 >> v2:172.21.15.98:3300/0 conn(0x7fb8d81397d0 0x7fb8d8139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): get_auth_request con 0x7fb8d81397d0 auth_method 0 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d809d4c0 0x7fb8d80a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 10 monclient(hunting): get_auth_request con 0x7fb8d809d4c0 auth_method 0 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_done global_id 44694 payload 274 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 1 -- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb8d813a0f0 msgr2=0x7fb8d80a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb8d813a0f0 0x7fb8d80a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 1 -- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d809d4c0 msgr2=0x7fb8d80a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d809d4c0 0x7fb8d80a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 1 monclient: found mon.noname-a 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.128+0000 7fb8d77fe640 1 -- 172.21.15.98:0/2604148338 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d809e440 con 0x7fb8d81397d0 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d81397d0 0x7fb8d8139bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80097f0 tx=0x7fb8c801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743fbd4ab84bd82b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 1 -- 172.21.15.98:0/2604148338 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8c80b0070 con 0x7fb8d81397d0 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: dump: 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.143 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 5 monclient: authenticate success, global_id 44694 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 1 -- 172.21.15.98:0/2604148338 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c80aa810 con 0x7fb8d81397d0 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 4 set_mon_vals no callback set 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 1 -- 172.21.15.98:0/2604148338 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8c80a9300 con 0x7fb8d81397d0 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:00:57.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d67fc640 10 monclient: dump: 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.146 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: shutdown 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d81397d0 msgr2=0x7fb8d8139bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d81397d0 0x7fb8d8139bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80097f0 tx=0x7fb8c801c360 comp rx=0 tx=0).stop 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- 172.21.15.98:0/2604148338 shutdown_connections 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d809d4c0 0x7fb8d80a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb8d813a0f0 0x7fb8d80a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 --2- 172.21.15.98:0/2604148338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d81397d0 0x7fb8d8139bb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- 172.21.15.98:0/2604148338 >> 172.21.15.98:0/2604148338 conn(0x7fb8d80a4780 msgr2=0x7fb8d80a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- 172.21.15.98:0/2604148338 shutdown_connections 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- 172.21.15.98:0/2604148338 wait complete. 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: monmap: 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:57.148 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 librados: starting msgr at 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 librados: starting objecter 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 Processor -- start 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- start start 2024-09-11T03:00:57.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 librados: setting wanted keys 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 librados: calling monclient init 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: init 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: _add_conns ranks=[5,7,4,0,8,6,3,1,2] 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb8d809d4c0 0x7fb8d8133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): picked mon.noname-h con 0x7fb8d809d4c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb8d81397d0 0x7fb8d8133540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): picked mon.noname-i con 0x7fb8d81397d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d813a0f0 0x7fb8d813d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): picked mon.noname-c con 0x7fb8d813a0f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb8d8143310 con 0x7fb8d813a0f0 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb8d8143190 con 0x7fb8d809d4c0 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb8d8143490 con 0x7fb8d81397d0 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient(hunting): authenticate will time out at 1611.014481s 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d813a0f0 0x7fb8d813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d813a0f0 0x7fb8d813d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34542/0 (socket says 172.21.15.98:34542) 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 -- 172.21.15.98:0/2052412404 learned_addr learned my addr 172.21.15.98:0/2052412404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): get_auth_request con 0x7fb8d813a0f0 auth_method 0 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d6ffd640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb8d81397d0 0x7fb8d8133540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient(hunting): handle_auth_done global_id 24223 payload 824 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb8d809d4c0 0x7fb8d8133000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb8d81397d0 msgr2=0x7fb8d8133540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb8d81397d0 0x7fb8d8133540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb8d809d4c0 msgr2=0x7fb8d8133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb8d809d4c0 0x7fb8d8133000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 monclient: found mon.noname-c 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 -- 172.21.15.98:0/2052412404 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d811e870 con 0x7fb8d813a0f0 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d77fe640 10 monclient: get_auth_request con 0x7fb8d809d4c0 auth_method 0 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb8d809d4c0 0x7fb8d8133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d7fff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d813a0f0 0x7fb8d813d410 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009e30 tx=0x7fb8d000c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 client.?.objecter ms_handle_connect 0x7fb8d813a0f0 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8d0018020 con 0x7fb8d813a0f0 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: dump: 2024-09-11T03:00:57.153 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.154 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d000cdf0 con 0x7fb8d813a0f0 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8d0014d60 con 0x7fb8d813a0f0 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 5 monclient: authenticate success, global_id 24223 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: _renew_subs 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8d811eb50 con 0x7fb8d813a0f0 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8d57fa640 4 set_mon_vals no callback set 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8bffff640 10 monclient: dump: 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 client.24223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: _renew_subs 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8d811e590 con 0x7fb8d813a0f0 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.132+0000 7fb8de201640 1 librados: init done 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8de201640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8de201640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e110 0x7fb8d802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb8d806ca10 con 0x7fb8d813e110 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e110 0x7fb8d802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient: get_auth_request con 0x7fb8d813e110 auth_method 0 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb8d0014480 con 0x7fb8d813a0f0 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_done global_id 64352 payload 824 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb8c403df40 0x7fb8c4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 1 -- 172.21.15.98:0/2052412404 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8d80a7300 con 0x7fb8d813e110 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7fb8d0050930 con 0x7fb8d813a0f0 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 client.24223.objecter ms_dispatch 0x7fb8d8000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 3 client.24223.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 3 client.24223.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 20 client.24223.objecter dump_active .. 0 homeless 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 client.24223.objecter ms_handle_connect 0x7fb8d813e110 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 client.24223.objecter resend_mon_ops 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e110 0x7fb8d802d450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80ad000 tx=0x7fb8d80a5800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d6ffd640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb8c403df40 0x7fb8c4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d6ffd640 10 monclient: get_auth_request con 0x7fb8c403df40 auth_method 0 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8c80b9070 con 0x7fb8d813e110 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:57.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 monclient: dump: 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.160 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb8c8003620 con 0x7fb8d813e110 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e110 msgr2=0x7fb8d802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e110 0x7fb8d802d450 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80ad000 tx=0x7fb8d80a5800 comp rx=0 tx=0).stop 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e110 msgr2=0x7fb8d802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e110 0x7fb8d802d450 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80ad000 tx=0x7fb8d80a5800 comp rx=0 tx=0).stop 2024-09-11T03:00:57.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8bffff640 10 client.24223.objecter ms_handle_connect 0x7fb8c403df40 2024-09-11T03:00:57.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.136+0000 7fb8d6ffd640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb8c403df40 0x7fb8c4040400 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8134470 tx=0x7fb8cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8de201640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8de201640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e4f0 0x7fb8d802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8de201640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb8d806e870 con 0x7fb8d813e4f0 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e4f0 0x7fb8d802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient: get_auth_request con 0x7fb8d813e4f0 auth_method 0 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 10 monclient(hunting): handle_auth_done global_id 64361 payload 824 2024-09-11T03:00:57.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 1 -- 172.21.15.98:0/2052412404 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb8d8068b40 con 0x7fb8d813e4f0 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8d77fe640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e4f0 0x7fb8d802d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80097c0 tx=0x7fb8c80a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8bffff640 10 client.24223.objecter ms_handle_connect 0x7fb8d813e4f0 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.140+0000 7fb8bffff640 10 client.24223.objecter resend_mon_ops 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8c80b0040 con 0x7fb8d813e4f0 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 10 monclient: dump: 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.165 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7fb8c80c2070 con 0x7fb8d813e4f0 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e4f0 msgr2=0x7fb8d802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e4f0 0x7fb8d802d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80097c0 tx=0x7fb8c80a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e4f0 msgr2=0x7fb8d802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8bffff640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb8d813e4f0 0x7fb8d802d990 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80097c0 tx=0x7fb8c80a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:57.167 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 149338, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:57.168 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.169 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.170 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:57.171 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:57.172 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:57.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:57.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:57.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.178 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:57.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:57.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.187 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:57.189 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.189 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.190 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.191 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:57.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.193 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:57.194 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 10 librados: watch_flush enter 2024-09-11T03:00:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 10 librados: watch_flush exit 2024-09-11T03:00:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 20 client.24223.objecter shutdown clearing up homeless session... 2024-09-11T03:00:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 10 client.24223.objecter successfully canceled tick 2024-09-11T03:00:57.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb8c403df40 msgr2=0x7fb8c4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb8c403df40 0x7fb8c4040400 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8134470 tx=0x7fb8cc008040 comp rx=0 tx=0).stop 2024-09-11T03:00:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 10 monclient: shutdown 2024-09-11T03:00:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d813a0f0 msgr2=0x7fb8d813d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 --2- 172.21.15.98:0/2052412404 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb8d813a0f0 0x7fb8d813d410 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0009e30 tx=0x7fb8d000c490 comp rx=0 tx=0).stop 2024-09-11T03:00:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8d7fff640 1 -- 172.21.15.98:0/2052412404 reap_dead start 2024-09-11T03:00:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8d7fff640 1 -- 172.21.15.98:0/2052412404 reap_dead start 2024-09-11T03:00:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 shutdown_connections 2024-09-11T03:00:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 >> 172.21.15.98:0/2052412404 conn(0x7fb8d80a4780 msgr2=0x7fb8d809fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 shutdown_connections 2024-09-11T03:00:57.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 -- 172.21.15.98:0/2052412404 wait complete. 2024-09-11T03:00:57.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.144+0000 7fb8de201640 1 librados: shutdown 2024-09-11T03:00:57.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:57.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T03:00:57.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:00:57.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient: get_monmap_and_config 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient: monmap: 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:57.321719+0000 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:57.321719+0000 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:57.321 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 1 Processor -- start 2024-09-11T03:00:57.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 1 -- start start 2024-09-11T03:00:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient: init 2024-09-11T03:00:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient: _add_conns ranks=[2,3,1,7,9,6,8,5,4,10,0] 2024-09-11T03:00:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa598139b10 0x7fa598072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient(hunting): picked mon.noname-b con 0x7fa598139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139ef0 0x7fa5980a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient(hunting): picked mon.noname-g con 0x7fa598139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa59813a2d0 addr v1:172.21.15.190:6789/0 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5980702b0 con 0x7fa59813a2d0 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa598071bf0 con 0x7fa598139b10 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa598071350 con 0x7fa598139ef0 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59f9f8640 10 monclient(hunting): authenticate will time out at 1611.202098s 2024-09-11T03:00:57.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa598139b10 0x7fa598072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa598139b10 0x7fa598072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58082/0 (socket says 172.21.15.98:58082) 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 -- 172.21.15.98:0/2772877797 learned_addr learned my addr 172.21.15.98:0/2772877797 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): get_auth_request con 0x7fa598139b10 auth_method 0 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59cf93640 1 --2- 172.21.15.98:0/2772877797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139ef0 0x7fa5980a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient(hunting): handle_auth_done global_id 64370 payload 274 2024-09-11T03:00:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 -- 172.21.15.98:0/2772877797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139ef0 msgr2=0x7fa5980a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 --2- 172.21.15.98:0/2772877797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139ef0 0x7fa5980a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 -- 172.21.15.98:0/2772877797 >> v1:172.21.15.190:6789/0 conn(0x7fa59813a2d0 legacy=0x7fa5980a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 monclient: found mon.noname-b 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59d794640 1 -- 172.21.15.98:0/2772877797 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5980a9620 con 0x7fa598139b10 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59cf93640 10 monclient: get_auth_request con 0x7fa598139ef0 auth_method 0 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.320+0000 7fa59cf93640 1 --2- 172.21.15.98:0/2772877797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139ef0 0x7fa5980a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59d794640 1 --2- 172.21.15.98:0/2772877797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa598139b10 0x7fa598072600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa5880098a0 tx=0x7fa58801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ded6b4f948af71b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 1 -- 172.21.15.98:0/2772877797 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5880b1070 con 0x7fa598139b10 2024-09-11T03:00:57.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: dump: 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.338 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 5 monclient: authenticate success, global_id 64370 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 1 -- 172.21.15.98:0/2772877797 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5880aa7b0 con 0x7fa598139b10 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:57.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 4 set_mon_vals no callback set 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 1 -- 172.21.15.98:0/2772877797 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5880aae10 con 0x7fa598139b10 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58ffff640 10 monclient: dump: 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient: shutdown 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- 172.21.15.98:0/2772877797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa598139b10 msgr2=0x7fa598072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 --2- 172.21.15.98:0/2772877797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa598139b10 0x7fa598072600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa5880098a0 tx=0x7fa58801c440 comp rx=0 tx=0).stop 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- 172.21.15.98:0/2772877797 shutdown_connections 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 --2- 172.21.15.98:0/2772877797 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139ef0 0x7fa5980a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 --2- 172.21.15.98:0/2772877797 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa598139b10 0x7fa598072600 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- 172.21.15.98:0/2772877797 >> 172.21.15.98:0/2772877797 conn(0x7fa5980ac790 msgr2=0x7fa5980a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- 172.21.15.98:0/2772877797 shutdown_connections 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- 172.21.15.98:0/2772877797 wait complete. 2024-09-11T03:00:57.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient: monmap: 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.343 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 librados: starting msgr at 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 librados: starting objecter 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 Processor -- start 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- start start 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 librados: setting wanted keys 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 librados: calling monclient init 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient: init 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient: _add_conns ranks=[3,6,5,8,0,7,1,2,4] 2024-09-11T03:00:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139b10 0x7fa598133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): picked mon.noname-g con 0x7fa598139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa598139ef0 0x7fa598133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): picked mon.noname-d con 0x7fa598139ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa59813a2d0 0x7fa5981341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): picked mon.noname-h con 0x7fa59813a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa598144470 con 0x7fa598139b10 2024-09-11T03:00:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa5980a8870 con 0x7fa59813a2d0 2024-09-11T03:00:57.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa5981445f0 con 0x7fa598139ef0 2024-09-11T03:00:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59f9f8640 10 monclient(hunting): authenticate will time out at 1611.207264s 2024-09-11T03:00:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa598139ef0 0x7fa598133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa598139ef0 0x7fa598133ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:60892/0 (socket says 172.21.15.98:60892) 2024-09-11T03:00:57.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 -- 172.21.15.98:0/402046845 learned_addr learned my addr 172.21.15.98:0/402046845 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): get_auth_request con 0x7fa598139ef0 auth_method 0 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient(hunting): handle_auth_done global_id 44529 payload 824 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa59813a2d0 msgr2=0x7fa5981341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa59813a2d0 0x7fa5981341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139b10 msgr2=0x7fa598133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59d794640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139b10 0x7fa598133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa598139b10 0x7fa598133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 monclient: found mon.noname-d 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 -- 172.21.15.98:0/402046845 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59813c7b0 con 0x7fa598139ef0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa59cf93640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa598139ef0 0x7fa598133ca0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa59400c7c0 tx=0x7fa59400cc80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58dffb640 10 client.?.objecter ms_handle_connect 0x7fa598139ef0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.324+0000 7fa58dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa594007ca0 con 0x7fa598139ef0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:00:57.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: dump: 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.350 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5940113f0 con 0x7fa598139ef0 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 5 monclient: authenticate success, global_id 44529 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 10 monclient: _renew_subs 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5980aa0e0 con 0x7fa598139ef0 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa594011a10 con 0x7fa598139ef0 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:00:57.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: dump: 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.352 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58f7fe640 4 set_mon_vals no callback set 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 10 client.44529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 10 monclient: _renew_subs 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5980a55d0 con 0x7fa598139ef0 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 1 librados: init done 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5981372e0 0x7fa5981376c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa59806ca10 con 0x7fa5981372e0 2024-09-11T03:00:57.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5981372e0 0x7fa5981376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient: get_auth_request con 0x7fa5981372e0 auth_method 0 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa594011bb0 con 0x7fa598139ef0 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 10 monclient(hunting): handle_auth_done global_id 24232 payload 824 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 1 -- 172.21.15.98:0/402046845 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa59806e680 con 0x7fa5981372e0 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa57c03df40 0x7fa57c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7fa59404f990 con 0x7fa598139ef0 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 client.44529.objecter ms_dispatch 0x7fa598000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 3 client.44529.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 3 client.44529.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59d794640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5981372e0 0x7fa5981376c0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fa5880ae000 tx=0x7fa5880191e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 20 client.44529.objecter dump_active .. 0 homeless 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 client.44529.objecter ms_handle_connect 0x7fa5981372e0 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 client.44529.objecter resend_mon_ops 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5880bc070 con 0x7fa5981372e0 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: dump: 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.355 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.356 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59df95640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa57c03df40 0x7fa57c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59df95640 10 monclient: get_auth_request con 0x7fa57c03df40 auth_method 0 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa5880036e0 con 0x7fa5981372e0 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5981372e0 msgr2=0x7fa5981376c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5981372e0 0x7fa5981376c0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fa5880ae000 tx=0x7fa5880191e0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5981372e0 msgr2=0x7fa5981376c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5981372e0 0x7fa5981376c0 secure :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fa5880ae000 tx=0x7fa5880191e0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa59df95640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa57c03df40 0x7fa57c040400 secure :-1 s=READY pgs=3575 cs=0 l=1 rev1=1 crypto rx=0x7fa5980a5e50 tx=0x7fa590007d00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.328+0000 7fa58dffb640 10 client.44529.objecter ms_handle_connect 0x7fa57c03df40 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59f9f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59f9f8640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5980ab220 0x7fa59813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59f9f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa59806e870 con 0x7fa5980ab220 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5980ab220 0x7fa59813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient: get_auth_request con 0x7fa5980ab220 auth_method 0 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 10 monclient(hunting): handle_auth_done global_id 24241 payload 824 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 1 -- 172.21.15.98:0/402046845 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa598072600 con 0x7fa5980ab220 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa58dffb640 10 client.44529.objecter ms_handle_connect 0x7fa5980ab220 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.332+0000 7fa59d794640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5980ab220 0x7fa59813a7d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fa58801aaf0 tx=0x7fa5880a6000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 10 client.44529.objecter resend_mon_ops 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa5880bc040 con 0x7fa5980ab220 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 10 monclient: dump: 2024-09-11T03:00:57.360 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.361 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fa588003e40 con 0x7fa5980ab220 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5980ab220 msgr2=0x7fa59813a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5980ab220 0x7fa59813a7d0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fa58801aaf0 tx=0x7fa5880a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5980ab220 msgr2=0x7fa59813a7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa58dffb640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa5980ab220 0x7fa59813a7d0 secure :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fa58801aaf0 tx=0x7fa5880a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:57.362 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 673121, 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:57.363 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.365 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:57.367 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.368 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.372 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.373 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.374 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:57.375 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.377 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.378 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.379 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.380 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.381 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.382 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.382 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.382 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.382 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.382 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:57.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:57.383 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.383 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.383 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:57.383 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.384 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.384 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.384 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:57.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.384 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.385 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.385 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.385 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.385 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.386 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.386 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.386 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:57.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:57.386 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.386 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.387 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:57.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.387 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:57.388 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.388 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.388 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.389 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.389 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.389 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.389 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.390 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:57.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:57.390 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.391 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:57.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:57.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.393 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.393 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.394 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.394 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:57.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:57.394 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:57.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:57.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.397 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.397 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.398 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.398 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:57.399 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:57.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.399 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.400 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.400 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:00:57.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.402 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.402 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.403 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T03:00:57.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.403 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.403 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:57.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.403 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.404 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.404 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:00:57.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.404 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.405 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:57.405 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 10 librados: watch_flush enter 2024-09-11T03:00:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 10 librados: watch_flush exit 2024-09-11T03:00:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 20 client.44529.objecter shutdown clearing up homeless session... 2024-09-11T03:00:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 10 client.44529.objecter successfully canceled tick 2024-09-11T03:00:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa57c03df40 msgr2=0x7fa57c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa57c03df40 0x7fa57c040400 secure :-1 s=READY pgs=3575 cs=0 l=1 rev1=1 crypto rx=0x7fa5980a5e50 tx=0x7fa590007d00 comp rx=0 tx=0).stop 2024-09-11T03:00:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 10 monclient: shutdown 2024-09-11T03:00:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa598139ef0 msgr2=0x7fa598133ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 --2- 172.21.15.98:0/402046845 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa598139ef0 0x7fa598133ca0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa59400c7c0 tx=0x7fa59400cc80 comp rx=0 tx=0).stop 2024-09-11T03:00:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59df95640 1 -- 172.21.15.98:0/402046845 reap_dead start 2024-09-11T03:00:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59df95640 1 -- 172.21.15.98:0/402046845 reap_dead start 2024-09-11T03:00:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 shutdown_connections 2024-09-11T03:00:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 >> 172.21.15.98:0/402046845 conn(0x7fa5980ac790 msgr2=0x7fa598134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 shutdown_connections 2024-09-11T03:00:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 -- 172.21.15.98:0/402046845 wait complete. 2024-09-11T03:00:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.336+0000 7fa59f9f8640 1 librados: shutdown 2024-09-11T03:00:57.411 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:57.412 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T03:00:57.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:00:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.528+0000 7f9893dd1640 10 monclient: get_monmap_and_config 2024-09-11T03:00:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.528+0000 7f9893dd1640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.528+0000 7f9893dd1640 10 monclient: monmap: 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:57.533164+0000 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:57.533164+0000 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 1 Processor -- start 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 1 -- start start 2024-09-11T03:00:57.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient: init 2024-09-11T03:00:57.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient: _add_conns ranks=[2,7,1,6,3,9,0,4,8,5,10] 2024-09-11T03:00:57.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f988c0a3690 0x7f988c0a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient(hunting): picked mon.noname-b con 0x7f988c0a3690 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:57.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a1eb0 0x7f988c0a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient(hunting): picked mon.noname-i con 0x7f988c0a1eb0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f988c137460 addr v1:172.21.15.190:6789/0 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f988c0a1ca0 con 0x7f988c137460 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f988c09ec40 con 0x7f988c0a3690 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f988c0a44f0 con 0x7f988c0a1eb0 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9893dd1640 10 monclient(hunting): authenticate will time out at 1611.413337s 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f988c0a3690 0x7f988c0a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f988c0a3690 0x7f988c0a3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58098/0 (socket says 172.21.15.98:58098) 2024-09-11T03:00:57.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 -- 172.21.15.98:0/550690495 learned_addr learned my addr 172.21.15.98:0/550690495 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): get_auth_request con 0x7f988c0a3690 auth_method 0 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f989136c640 1 --2- 172.21.15.98:0/550690495 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a1eb0 0x7f988c0a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f989136c640 10 monclient(hunting): get_auth_request con 0x7f988c0a1eb0 auth_method 0 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f989136c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f989136c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f989136c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient(hunting): handle_auth_done global_id 64379 payload 274 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 -- 172.21.15.98:0/550690495 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a1eb0 msgr2=0x7f988c0a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 --2- 172.21.15.98:0/550690495 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a1eb0 0x7f988c0a3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 -- 172.21.15.98:0/550690495 >> v1:172.21.15.190:6789/0 conn(0x7f988c137460 legacy=0x7f988c1378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 monclient: found mon.noname-b 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.532+0000 7f9891b6d640 1 -- 172.21.15.98:0/550690495 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c09dd20 con 0x7f988c0a3690 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9891b6d640 1 --2- 172.21.15.98:0/550690495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f988c0a3690 0x7f988c0a3a70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98880098a0 tx=0x7f988801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf3c9ff34d62a83d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 1 -- 172.21.15.98:0/550690495 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98880b1070 con 0x7f988c0a3690 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: dump: 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.547 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 5 monclient: authenticate success, global_id 64379 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 1 -- 172.21.15.98:0/550690495 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98880aa7b0 con 0x7f988c0a3690 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 4 set_mon_vals no callback set 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 1 -- 172.21.15.98:0/550690495 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98880aae50 con 0x7f988c0a3690 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9890b6b640 10 monclient: dump: 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.549 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.550 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient: shutdown 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- 172.21.15.98:0/550690495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f988c0a3690 msgr2=0x7f988c0a3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 --2- 172.21.15.98:0/550690495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f988c0a3690 0x7f988c0a3a70 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98880098a0 tx=0x7f988801c440 comp rx=0 tx=0).stop 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- 172.21.15.98:0/550690495 shutdown_connections 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 --2- 172.21.15.98:0/550690495 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a1eb0 0x7f988c0a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 --2- 172.21.15.98:0/550690495 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f988c0a3690 0x7f988c0a3a70 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- 172.21.15.98:0/550690495 >> 172.21.15.98:0/550690495 conn(0x7f988c0a4780 msgr2=0x7f988c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- 172.21.15.98:0/550690495 shutdown_connections 2024-09-11T03:00:57.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- 172.21.15.98:0/550690495 wait complete. 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient: monmap: 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.552 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 librados: starting msgr at 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 librados: starting objecter 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 Processor -- start 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- start start 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 librados: setting wanted keys 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 librados: calling monclient init 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient: init 2024-09-11T03:00:57.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient: _add_conns ranks=[3,7,8,1,6,2,0,4,5] 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f988c0a1eb0 0x7f988c1d15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): picked mon.noname-g con 0x7f988c0a1eb0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a3690 0x7f988c136670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): picked mon.noname-i con 0x7f988c0a3690 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f988c137460 0x7f988c136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): picked mon.noname-e con 0x7f988c137460 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f988c1420f0 con 0x7f988c0a1eb0 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f988c0a1ae0 con 0x7f988c0a3690 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f988c142270 con 0x7f988c137460 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9893dd1640 10 monclient(hunting): authenticate will time out at 1611.419310s 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f988c137460 0x7f988c136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f988c137460 0x7f988c136bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35060/0 (socket says 172.21.15.98:35060) 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 -- 172.21.15.98:0/795826102 learned_addr learned my addr 172.21.15.98:0/795826102 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): get_auth_request con 0x7f988c137460 auth_method 0 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9891b6d640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f988c0a1eb0 0x7f988c1d15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9891b6d640 10 monclient(hunting): get_auth_request con 0x7f988c0a1eb0 auth_method 0 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9891b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9891b6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f9891b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989136c640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a3690 0x7f988c136670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient(hunting): handle_auth_done global_id 24722 payload 824 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a3690 msgr2=0x7f988c136670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a3690 0x7f988c136670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f988c0a1eb0 msgr2=0x7f988c1d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f988c0a1eb0 0x7f988c1d15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 monclient: found mon.noname-e 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 -- 172.21.15.98:0/795826102 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c120cc0 con 0x7f988c137460 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989136c640 10 monclient: get_auth_request con 0x7f988c0a3690 auth_method 0 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989136c640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f988c0a3690 0x7f988c136670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.536+0000 7f989236e640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f988c137460 0x7f988c136bb0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f988400c730 tx=0x7f988400cbf0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 client.?.objecter ms_handle_connect 0x7f988c137460 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98840017b0 con 0x7f988c137460 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:00:57.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: dump: 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.558 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f988400ce40 con 0x7f988c137460 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 5 monclient: authenticate success, global_id 24722 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 10 monclient: _renew_subs 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c0733e0 con 0x7f988c137460 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9884003e50 con 0x7f988c137460 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: dump: 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.560 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9883fff640 4 set_mon_vals no callback set 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 10 client.24722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 10 monclient: _renew_subs 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c121820 con 0x7f988c137460 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 1 librados: init done 2024-09-11T03:00:57.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11f200 0x7f988c11f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f988c06ca10 con 0x7f988c11f200 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11f200 0x7f988c11f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient: get_auth_request con 0x7f988c11f200 auth_method 0 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f988400a850 con 0x7f988c137460 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 10 monclient(hunting): handle_auth_done global_id 44538 payload 824 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 1 -- 172.21.15.98:0/795826102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f988c0a6d40 con 0x7f988c11f200 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f9891b6d640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11f200 0x7f988c11f5e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f988c074a00 tx=0x7f98880191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f987003e0e0 0x7f98700405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 client.24722.objecter ms_handle_connect 0x7f988c11f200 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 client.24722.objecter resend_mon_ops 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7f988404f2c0 con 0x7f988c137460 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 client.24722.objecter ms_dispatch 0x7f988c000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 3 client.24722.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 3 client.24722.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 20 client.24722.objecter dump_active .. 0 homeless 2024-09-11T03:00:57.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98880ba030 con 0x7f988c11f200 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: dump: 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.564 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f989136c640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f987003e0e0 0x7f98700405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f98880b2040 con 0x7f988c11f200 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:57.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11f200 msgr2=0x7f988c11f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11f200 0x7f988c11f5e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f988c074a00 tx=0x7f98880191e0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11f200 msgr2=0x7f988c11f5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11f200 0x7f988c11f5e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f989136c640 10 monclient: get_auth_request con 0x7f987003e0e0 auth_method 0 2024-09-11T03:00:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f98827fc640 10 client.24722.objecter ms_handle_connect 0x7f987003e0e0 2024-09-11T03:00:57.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.540+0000 7f989136c640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f987003e0e0 0x7f98700405a0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f988c121330 tx=0x7f987c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.544+0000 7f9893dd1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.544+0000 7f9893dd1640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11fb20 0x7f988c06f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.544+0000 7f9893dd1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.544+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f988c06e870 con 0x7f988c11fb20 2024-09-11T03:00:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11fb20 0x7f988c06f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient: get_auth_request con 0x7f988c11fb20 auth_method 0 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 10 monclient(hunting): handle_auth_done global_id 44547 payload 824 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 1 -- 172.21.15.98:0/795826102 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f988c068b40 con 0x7f988c11fb20 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 10 client.24722.objecter ms_handle_connect 0x7f988c11fb20 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 10 client.24722.objecter resend_mon_ops 2024-09-11T03:00:57.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9891b6d640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11fb20 0x7f988c06f920 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f98880098a0 tx=0x7f98880a6000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98880ba070 con 0x7f988c11fb20 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 10 monclient: dump: 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.570 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f98880b6400 con 0x7f988c11fb20 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11fb20 msgr2=0x7f988c06f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11fb20 0x7f988c06f920 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f98880098a0 tx=0x7f98880a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11fb20 msgr2=0x7f988c06f920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f98827fc640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f988c11fb20 0x7f988c06f920 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f98880098a0 tx=0x7f98880a6000 comp rx=0 tx=0).stop 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 283976, 2024-09-11T03:00:57.572 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:00:57.573 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.574 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.575 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:57.576 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:57.577 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:57.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:57.582 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.583 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.584 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.586 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.587 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.588 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:57.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.590 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.592 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.595 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.596 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:57.597 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 10 librados: watch_flush enter 2024-09-11T03:00:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 10 librados: watch_flush exit 2024-09-11T03:00:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 20 client.24722.objecter shutdown clearing up homeless session... 2024-09-11T03:00:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 10 client.24722.objecter successfully canceled tick 2024-09-11T03:00:57.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f987003e0e0 msgr2=0x7f98700405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f987003e0e0 0x7f98700405a0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f988c121330 tx=0x7f987c008040 comp rx=0 tx=0).stop 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 10 monclient: shutdown 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f988c137460 msgr2=0x7f988c136bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 --2- 172.21.15.98:0/795826102 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f988c137460 0x7f988c136bb0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f988400c730 tx=0x7f988400cbf0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f989236e640 1 -- 172.21.15.98:0/795826102 reap_dead start 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f989236e640 1 -- 172.21.15.98:0/795826102 reap_dead start 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 shutdown_connections 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 >> 172.21.15.98:0/795826102 conn(0x7f988c0a4780 msgr2=0x7f988c0a5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 shutdown_connections 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 -- 172.21.15.98:0/795826102 wait complete. 2024-09-11T03:00:57.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.548+0000 7f9893dd1640 1 librados: shutdown 2024-09-11T03:00:57.600 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:57.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T03:00:57.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient: get_monmap_and_config 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient: monmap: 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:57.718024+0000 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:57.718024+0000 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.716 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:57.717 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 1 Processor -- start 2024-09-11T03:00:57.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 1 -- start start 2024-09-11T03:00:57.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient: init 2024-09-11T03:00:57.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient: _add_conns ranks=[3,8,1,9,0,4,2,10,7,5,6] 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba0139bd0 0x7fdba00a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient(hunting): picked mon.noname-g con 0x7fdba0139bd0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba0139fb0 0x7fdba00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient(hunting): picked mon.noname-e con 0x7fdba0139fb0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdba013a3b0 addr v1:172.21.15.190:6789/0 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba00a0f60 con 0x7fdba013a3b0 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdba00777d0 con 0x7fdba0139bd0 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdba0070050 con 0x7fdba0139fb0 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdba6a0e640 10 monclient(hunting): authenticate will time out at 1611.598273s 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba0139fb0 0x7fdba00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba0139fb0 0x7fdba00a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35074/0 (socket says 172.21.15.98:35074) 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/3772177270 learned_addr learned my addr 172.21.15.98:0/3772177270 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): get_auth_request con 0x7fdba0139fb0 auth_method 0 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/3772177270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba0139bd0 0x7fdba00a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_done global_id 24731 payload 274 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/3772177270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba0139bd0 msgr2=0x7fdba00a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 --2- 172.21.15.98:0/3772177270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba0139bd0 0x7fdba00a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/3772177270 >> v1:172.21.15.190:6789/0 conn(0x7fdba013a3b0 legacy=0x7fdba00a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 monclient: found mon.noname-e 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/3772177270 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba00a1ac0 con 0x7fdba0139fb0 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9ffff640 10 monclient: get_auth_request con 0x7fdba0139bd0 auth_method 0 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.716+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/3772177270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba0139bd0 0x7fdba00a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 --2- 172.21.15.98:0/3772177270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba0139fb0 0x7fdba00a4610 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fdb940098a0 tx=0x7fdb9401c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=6fe9c882fb30b9e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 1 -- 172.21.15.98:0/3772177270 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb940b0070 con 0x7fdba0139fb0 2024-09-11T03:00:57.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: dump: 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.731 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 1 -- 172.21.15.98:0/3772177270 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb940aa810 con 0x7fdba0139fb0 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 1 -- 172.21.15.98:0/3772177270 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb940a9300 con 0x7fdba0139fb0 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:57.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9effd640 10 monclient: dump: 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.733 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 5 monclient: authenticate success, global_id 24731 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 4 set_mon_vals no callback set 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient: shutdown 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- 172.21.15.98:0/3772177270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba0139fb0 msgr2=0x7fdba00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 --2- 172.21.15.98:0/3772177270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba0139fb0 0x7fdba00a4610 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fdb940098a0 tx=0x7fdb9401c440 comp rx=0 tx=0).stop 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- 172.21.15.98:0/3772177270 shutdown_connections 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 --2- 172.21.15.98:0/3772177270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba0139fb0 0x7fdba00a4610 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 --2- 172.21.15.98:0/3772177270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba0139bd0 0x7fdba00a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- 172.21.15.98:0/3772177270 >> 172.21.15.98:0/3772177270 conn(0x7fdba006ed00 msgr2=0x7fdba006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- 172.21.15.98:0/3772177270 shutdown_connections 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- 172.21.15.98:0/3772177270 wait complete. 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient: monmap: 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:57.735 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 librados: starting msgr at 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 librados: starting objecter 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 Processor -- start 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- start start 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 librados: setting wanted keys 2024-09-11T03:00:57.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 librados: calling monclient init 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient: init 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient: _add_conns ranks=[5,6,3,7,0,4,8,2,1] 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdba0139bd0 0x7fdba01d1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): picked mon.noname-h con 0x7fdba0139bd0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdba0139fb0 0x7fdba01d1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): picked mon.noname-d con 0x7fdba0139fb0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba013a3b0 0x7fdba013a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): picked mon.noname-g con 0x7fdba013a3b0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdba0144450 con 0x7fdba013a3b0 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fdba00a5470 con 0x7fdba0139bd0 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdba01445d0 con 0x7fdba0139fb0 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba6a0e640 10 monclient(hunting): authenticate will time out at 1611.603264s 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdba0139fb0 0x7fdba01d1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdba0139fb0 0x7fdba01d1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:60920/0 (socket says 172.21.15.98:60920) 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/2498537758 learned_addr learned my addr 172.21.15.98:0/2498537758 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): get_auth_request con 0x7fdba0139fb0 auth_method 0 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba4fab640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba013a3b0 0x7fdba013a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdba0139bd0 0x7fdba01d1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba4fab640 10 monclient(hunting): get_auth_request con 0x7fdba013a3b0 auth_method 0 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba4fab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba4fab640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdba4fab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient(hunting): handle_auth_done global_id 44556 payload 824 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdba0139bd0 msgr2=0x7fdba01d1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdba0139bd0 0x7fdba01d1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba013a3b0 msgr2=0x7fdba013a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdba013a3b0 0x7fdba013a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 monclient: found mon.noname-d 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 -- 172.21.15.98:0/2498537758 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba013c550 con 0x7fdba0139fb0 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9ffff640 10 monclient: get_auth_request con 0x7fdba0139bd0 auth_method 0 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdba0139bd0 0x7fdba01d1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9f7fe640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdba0139fb0 0x7fdba01d1c80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097c0 tx=0x7fdb940191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9cff9640 10 client.?.objecter ms_handle_connect 0x7fdba0139fb0 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.720+0000 7fdb9cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb940b9070 con 0x7fdba0139fb0 2024-09-11T03:00:57.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: dump: 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.741 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb940044b0 con 0x7fdba0139fb0 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb940b0040 con 0x7fdba0139fb0 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 5 monclient: authenticate success, global_id 44556 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9e7fc640 4 set_mon_vals no callback set 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: dump: 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.743 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 10 monclient: _renew_subs 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba013dfd0 con 0x7fdba0139fb0 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 10 client.44556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 10 monclient: _renew_subs 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba0076f50 con 0x7fdba0139fb0 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 1 librados: init done 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013d970 0x7fdba013b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdba006ca10 con 0x7fdba013d970 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fdb94004040 con 0x7fdba0139fb0 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013d970 0x7fdba013b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient: get_auth_request con 0x7fdba013d970 auth_method 0 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb8c03df60 0x7fdb8c040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7fdb940e8e40 con 0x7fdba0139fb0 2024-09-11T03:00:57.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 client.44556.objecter ms_dispatch 0x7fdba0000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 3 client.44556.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 3 client.44556.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_done global_id 24740 payload 824 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 1 -- 172.21.15.98:0/2498537758 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdba013c350 con 0x7fdba013d970 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 20 client.44556.objecter dump_active .. 0 homeless 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 client.44556.objecter ms_handle_connect 0x7fdba013d970 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 client.44556.objecter resend_mon_ops 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013d970 0x7fdba013b490 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000d790 tx=0x7fdb9000dc50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba4fab640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb8c03df60 0x7fdb8c040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba4fab640 10 monclient: get_auth_request con 0x7fdb8c03df60 auth_method 0 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb90016070 con 0x7fdba013d970 2024-09-11T03:00:57.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: dump: 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.747 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb9000de40 con 0x7fdba013d970 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013d970 msgr2=0x7fdba013b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013d970 0x7fdba013b490 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000d790 tx=0x7fdb9000dc50 comp rx=0 tx=0).stop 2024-09-11T03:00:57.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013d970 msgr2=0x7fdba013b490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013d970 0x7fdba013b490 secure :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fdb9000d790 tx=0x7fdb9000dc50 comp rx=0 tx=0).stop 2024-09-11T03:00:57.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdb9cff9640 10 client.44556.objecter ms_handle_connect 0x7fdb8c03df60 2024-09-11T03:00:57.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.724+0000 7fdba4fab640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb8c03df60 0x7fdb8c040420 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 crypto rx=0x7fdba00710e0 tx=0x7fdb98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:57.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013b9d0 0x7fdba002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdba006e870 con 0x7fdba013b9d0 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013b9d0 0x7fdba002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient: get_auth_request con 0x7fdba013b9d0 auth_method 0 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 10 monclient(hunting): handle_auth_done global_id 24749 payload 824 2024-09-11T03:00:57.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 1 -- 172.21.15.98:0/2498537758 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdba0072890 con 0x7fdba013b9d0 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 10 client.44556.objecter ms_handle_connect 0x7fdba013b9d0 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 10 client.44556.objecter resend_mon_ops 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9ffff640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013b9d0 0x7fdba002d450 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fdb90008010 tx=0x7fdb90004740 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb90012de0 con 0x7fdba013b9d0 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 10 monclient: dump: 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9269 (secure 0 0 0) 0x7fdb90012de0 con 0x7fdba013b9d0 2024-09-11T03:00:57.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013b9d0 msgr2=0x7fdba002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013b9d0 0x7fdba002d450 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fdb90008010 tx=0x7fdb90004740 comp rx=0 tx=0).stop 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013b9d0 msgr2=0x7fdba002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdb9cff9640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdba013b9d0 0x7fdba002d450 secure :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7fdb90008010 tx=0x7fdb90004740 comp rx=0 tx=0).stop 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:00:57.754 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 673525, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:57.755 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.756 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.757 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:57.758 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.759 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:57.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.761 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:57.762 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.763 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.764 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.765 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.766 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.767 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:57.769 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.771 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:57.772 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.774 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:57.776 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 10 librados: watch_flush enter 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 10 librados: watch_flush exit 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 20 client.44556.objecter shutdown clearing up homeless session... 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 10 client.44556.objecter successfully canceled tick 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb8c03df60 msgr2=0x7fdb8c040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb8c03df60 0x7fdb8c040420 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 crypto rx=0x7fdba00710e0 tx=0x7fdb98009040 comp rx=0 tx=0).stop 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 10 monclient: shutdown 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdba0139fb0 msgr2=0x7fdba01d1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 --2- 172.21.15.98:0/2498537758 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdba0139fb0 0x7fdba01d1c80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097c0 tx=0x7fdb940191e0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba4fab640 1 -- 172.21.15.98:0/2498537758 reap_dead start 2024-09-11T03:00:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba4fab640 1 -- 172.21.15.98:0/2498537758 reap_dead start 2024-09-11T03:00:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 shutdown_connections 2024-09-11T03:00:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 >> 172.21.15.98:0/2498537758 conn(0x7fdba006ed00 msgr2=0x7fdba0070610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 shutdown_connections 2024-09-11T03:00:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 -- 172.21.15.98:0/2498537758 wait complete. 2024-09-11T03:00:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.732+0000 7fdba6a0e640 1 librados: shutdown 2024-09-11T03:00:57.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:57.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T03:00:57.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:00:57.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient: get_monmap_and_config 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient: monmap: 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:57.897847+0000 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:57.897847+0000 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.896 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:57.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 1 Processor -- start 2024-09-11T03:00:57.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 1 -- start start 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient: init 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient: _add_conns ranks=[1,8,10,5,2,9,0,6,3,4,7] 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3d74139b10 addr v1:172.21.15.190:6789/0 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d7413ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient(hunting): picked mon.noname-e con 0x7f3d74072600 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3d7413ef60 0x7f3d74141350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient(hunting): picked mon.noname-f con 0x7f3d7413ef60 addr v2:172.21.15.190:3300/0 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d7413a600 con 0x7f3d74139b10 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f3d740702b0 con 0x7f3d7413ef60 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3d74071bf0 con 0x7f3d74072600 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7c685640 10 monclient(hunting): authenticate will time out at 1611.778948s 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d7413ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d7413ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35104/0 (socket says 172.21.15.98:35104) 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 -- 172.21.15.98:0/3133274779 learned_addr learned my addr 172.21.15.98:0/3133274779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f3d7413ef60 0x7f3d74141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): get_auth_request con 0x7f3d74072600 auth_method 0 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 10 monclient(hunting): get_auth_request con 0x7f3d7413ef60 auth_method 0 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d7ac22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient(hunting): handle_auth_done global_id 24758 payload 274 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 -- 172.21.15.98:0/3133274779 >> v2:172.21.15.190:3300/0 conn(0x7f3d7413ef60 msgr2=0x7f3d74141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 --2- 172.21.15.98:0/3133274779 >> v2:172.21.15.190:3300/0 conn(0x7f3d7413ef60 0x7f3d74141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 -- 172.21.15.98:0/3133274779 >> v1:172.21.15.190:6789/0 conn(0x7f3d74139b10 legacy=0x7f3d74139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 monclient: found mon.noname-e 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.896+0000 7f3d79c20640 1 -- 172.21.15.98:0/3133274779 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d74071350 con 0x7f3d74072600 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d79c20640 1 --2- 172.21.15.98:0/3133274779 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d7413ea20 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009870 tx=0x7f3d6401c390 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f5c759140e38ca06 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 1 -- 172.21.15.98:0/3133274779 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d640b1070 con 0x7f3d74072600 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: dump: 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.913 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 1 -- 172.21.15.98:0/3133274779 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d640ab820 con 0x7f3d74072600 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 1 -- 172.21.15.98:0/3133274779 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d640aa310 con 0x7f3d74072600 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7941f640 10 monclient: dump: 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 5 monclient: authenticate success, global_id 24758 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 4 set_mon_vals no callback set 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 10 monclient: shutdown 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 -- 172.21.15.98:0/3133274779 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 msgr2=0x7f3d7413ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 --2- 172.21.15.98:0/3133274779 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d7413ea20 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009870 tx=0x7f3d6401c390 comp rx=0 tx=0).stop 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 -- 172.21.15.98:0/3133274779 shutdown_connections 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 --2- 172.21.15.98:0/3133274779 >> v2:172.21.15.190:3300/0 conn(0x7f3d7413ef60 0x7f3d74141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 --2- 172.21.15.98:0/3133274779 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d7413ea20 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 -- 172.21.15.98:0/3133274779 >> 172.21.15.98:0/3133274779 conn(0x7f3d740ac790 msgr2=0x7f3d740a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 -- 172.21.15.98:0/3133274779 shutdown_connections 2024-09-11T03:00:57.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 -- 172.21.15.98:0/3133274779 wait complete. 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 10 monclient: build_initial_monmap 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 10 monclient: monmap: 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:57.918 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 librados: starting msgr at 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 librados: starting objecter 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 Processor -- start 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 -- start start 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 librados: setting wanted keys 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 1 librados: calling monclient init 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.900+0000 7f3d7c685640 10 monclient: init 2024-09-11T03:00:57.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient: _add_conns ranks=[8,4,6,3,1,5,0,7,2] 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d74138700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): picked mon.noname-e con 0x7f3d74072600 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3d74139b10 0x7f3d74138c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): picked mon.noname-c con 0x7f3d74139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3d7413ef60 0x7f3d740a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): picked mon.noname-d con 0x7f3d7413ef60 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3d74144230 con 0x7f3d74139b10 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f3d741440b0 con 0x7f3d7413ef60 2024-09-11T03:00:57.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f3d741443b0 con 0x7f3d74072600 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): authenticate will time out at 1611.784225s 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d74138700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3d7413ef60 0x7f3d740a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3d74139b10 0x7f3d74138c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3d7413ef60 0x7f3d740a2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:60932/0 (socket says 172.21.15.98:60932) 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3d74139b10 0x7f3d74138c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34590/0 (socket says 172.21.15.98:34590) 2024-09-11T03:00:57.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 1 -- 172.21.15.98:0/3141238881 learned_addr learned my addr 172.21.15.98:0/3141238881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d74138700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35120/0 (socket says 172.21.15.98:35120) 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient(hunting): get_auth_request con 0x7f3d7413ef60 auth_method 0 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): get_auth_request con 0x7f3d74072600 auth_method 0 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): get_auth_request con 0x7f3d74139b10 auth_method 0 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient(hunting): handle_auth_done global_id 24250 payload 824 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 msgr2=0x7f3d74138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d74138700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3d7413ef60 msgr2=0x7f3d740a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3d7413ef60 0x7f3d740a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient: _finish_hunting 0 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 monclient: found mon.noname-c 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 -- 172.21.15.98:0/3141238881 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7413e810 con 0x7f3d74139b10 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f3d7413ef60 0x7f3d740a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f3d74072600 0x7f3d74138700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d79c20640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3d74139b10 0x7f3d74138c40 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f3d640ac000 tx=0x7f3d64019190 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 client.?.objecter ms_handle_connect 0x7f3d74139b10 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:57.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d640b1040 con 0x7f3d74139b10 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: dump: 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: _finish_auth 0 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: _check_auth_tickets 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d640044b0 con 0x7f3d74139b10 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d640abce0 con 0x7f3d74139b10 2024-09-11T03:00:57.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 5 monclient: authenticate success, global_id 24250 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient: _renew_subs 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d740a8540 con 0x7f3d74139b10 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d78c1e640 4 set_mon_vals no callback set 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: dump: 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.927 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 client.24250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:57.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient: _renew_subs 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d74122930 con 0x7f3d74139b10 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 librados: init done 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122b70 0x7f3d740a3320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7406ca10 con 0x7f3d74122b70 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3d64004040 con 0x7f3d74139b10 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d5803df40 0x7f3d58040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7f3d640a9070 con 0x7f3d74139b10 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 client.24250.objecter ms_dispatch 0x7f3d74000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 3 client.24250.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:57.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 3 client.24250.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122b70 0x7f3d740a3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient: get_auth_request con 0x7f3d74122b70 auth_method 0 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 20 client.24250.objecter dump_active .. 0 homeless 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d5803df40 0x7f3d58040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 10 monclient: get_auth_request con 0x7f3d5803df40 auth_method 0 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 10 monclient(hunting): handle_auth_done global_id 25543 payload 824 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 1 -- 172.21.15.98:0/3141238881 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d7406e680 con 0x7f3d74122b70 2024-09-11T03:00:57.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 client.24250.objecter ms_handle_connect 0x7f3d74122b70 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 client.24250.objecter resend_mon_ops 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7a421640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122b70 0x7f3d740a3320 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f70 tx=0x7f3d7000ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d700052e0 con 0x7f3d74122b70 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d7ac22640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d5803df40 0x7f3d58040400 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0070b0 tx=0x7f3d6c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 monclient: dump: 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.904+0000 7f3d6affd640 10 client.24250.objecter ms_handle_connect 0x7f3d5803df40 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.908+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d70018070 con 0x7f3d74122b70 2024-09-11T03:00:57.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.908+0000 7f3d6affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:57.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.908+0000 7f3d6affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:57.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.908+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122b70 msgr2=0x7f3d740a3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.908+0000 7f3d6affd640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122b70 0x7f3d740a3320 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f70 tx=0x7f3d7000ff70 comp rx=0 tx=0).stop 2024-09-11T03:00:57.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.908+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122b70 msgr2=0x7f3d740a3320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.908+0000 7f3d6affd640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122b70 0x7f3d740a3320 secure :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f70 tx=0x7f3d7000ff70 comp rx=0 tx=0).stop 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7c685640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7c685640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122f50 0x7f3d740a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7c685640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7406e870 con 0x7f3d74122f50 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122f50 0x7f3d740a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient: get_auth_request con 0x7f3d74122f50 auth_method 0 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:57.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 10 monclient(hunting): handle_auth_done global_id 25552 payload 824 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 1 -- 172.21.15.98:0/3141238881 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d74139ef0 con 0x7f3d74122f50 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d7a421640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122f50 0x7f3d740a3860 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f70 tx=0x7f3d70001130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d6affd640 10 client.24250.objecter ms_handle_connect 0x7f3d74122f50 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d6affd640 10 client.24250.objecter resend_mon_ops 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d7000de70 con 0x7f3d74122f50 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d6affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:00:57.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.912+0000 7f3d6affd640 10 monclient: dump: 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:57.937 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f3d7000da00 con 0x7f3d74122f50 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d6affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d6affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122f50 msgr2=0x7f3d740a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d6affd640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122f50 0x7f3d740a3860 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f70 tx=0x7f3d70001130 comp rx=0 tx=0).stop 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d6affd640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122f50 msgr2=0x7f3d740a3860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d6affd640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d74122f50 0x7f3d740a3860 secure :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f3d70000f70 tx=0x7f3d70001130 comp rx=0 tx=0).stop 2024-09-11T03:00:57.938 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 673685, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:57.939 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.941 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:57.942 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:57.943 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:57.944 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:57.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:57.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.954 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:57.957 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:57.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.960 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:57.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:57.962 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:57.962 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:57.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 10 librados: watch_flush enter 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 10 librados: watch_flush exit 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 20 client.24250.objecter shutdown clearing up homeless session... 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 10 client.24250.objecter successfully canceled tick 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d5803df40 msgr2=0x7f3d58040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3d5803df40 0x7f3d58040400 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c0070b0 tx=0x7f3d6c00b040 comp rx=0 tx=0).stop 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 10 monclient: shutdown 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3d74139b10 msgr2=0x7f3d74138c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 --2- 172.21.15.98:0/3141238881 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3d74139b10 0x7f3d74138c40 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f3d640ac000 tx=0x7f3d64019190 comp rx=0 tx=0).stop 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7ac22640 1 -- 172.21.15.98:0/3141238881 reap_dead start 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7ac22640 1 -- 172.21.15.98:0/3141238881 reap_dead start 2024-09-11T03:00:57.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 shutdown_connections 2024-09-11T03:00:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 >> 172.21.15.98:0/3141238881 conn(0x7f3d740ac790 msgr2=0x7f3d740aae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 shutdown_connections 2024-09-11T03:00:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 -- 172.21.15.98:0/3141238881 wait complete. 2024-09-11T03:00:57.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:57.916+0000 7f3d7c685640 1 librados: shutdown 2024-09-11T03:00:57.965 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:57.965 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T03:00:57.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:00:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient: get_monmap_and_config 2024-09-11T03:00:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient: monmap: 2024-09-11T03:00:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:58.082363+0000 2024-09-11T03:00:58.080 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:58.082363+0000 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:58.082 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:58.082 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 Processor -- start 2024-09-11T03:00:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 -- start start 2024-09-11T03:00:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient: init 2024-09-11T03:00:58.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient: _add_conns ranks=[2,6,8,5,9,3,0,4,10,7,1] 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): picked mon.noname-b con 0x7fd408139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd408139ef0 0x7fd4080a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): picked mon.noname-d con 0x7fd408139ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd40813a2d0 0x7fd4080a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): picked mon.noname-e con 0x7fd40813a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd4080702b0 con 0x7fd408139b10 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd408071bf0 con 0x7fd408139ef0 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd408071350 con 0x7fd40813a2d0 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40cfb0640 10 monclient(hunting): authenticate will time out at 1611.963399s 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd40813a2d0 0x7fd4080a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd408139ef0 0x7fd4080a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58102/0 (socket says 172.21.15.98:58102) 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd40813a2d0 0x7fd4080a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35136/0 (socket says 172.21.15.98:35136) 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd408139ef0 0x7fd4080a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:60940/0 (socket says 172.21.15.98:60940) 2024-09-11T03:00:58.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 1 -- 172.21.15.98:0/2689614038 learned_addr learned my addr 172.21.15.98:0/2689614038 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 10 monclient(hunting): get_auth_request con 0x7fd408139b10 auth_method 0 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): get_auth_request con 0x7fd40813a2d0 auth_method 0 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 10 monclient(hunting): get_auth_request con 0x7fd408139ef0 auth_method 0 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 10 monclient(hunting): handle_auth_done global_id 24776 payload 274 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 1 -- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd408139ef0 msgr2=0x7fd4080a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 1 --2- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd408139ef0 0x7fd4080a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 1 -- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 msgr2=0x7fd408072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.080+0000 7fd406d9d640 1 --2- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd406d9d640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd406d9d640 1 monclient: found mon.noname-e 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd406d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd406d9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:00:58.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd406d9d640 1 -- 172.21.15.98:0/2689614038 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4080a9620 con 0x7fd40813a2d0 2024-09-11T03:00:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd406d9d640 1 --2- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd40813a2d0 0x7fd4080a7f50 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fd400009dc0 tx=0x7fd40001c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=2c5c3c16856992ce server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 1 -- 172.21.15.98:0/2689614038 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd4000ac070 con 0x7fd40813a2d0 2024-09-11T03:00:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:00:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: dump: 2024-09-11T03:00:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 1 -- 172.21.15.98:0/2689614038 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4000a68b0 con 0x7fd40813a2d0 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 5 monclient: authenticate success, global_id 24776 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 1 -- 172.21.15.98:0/2689614038 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd4000a7420 con 0x7fd40813a2d0 2024-09-11T03:00:58.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 4 set_mon_vals no callback set 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd405d9b640 10 monclient: dump: 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 10 monclient: shutdown 2024-09-11T03:00:58.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 -- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd40813a2d0 msgr2=0x7fd4080a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd40813a2d0 0x7fd4080a7f50 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fd400009dc0 tx=0x7fd40001c850 comp rx=0 tx=0).stop 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 -- 172.21.15.98:0/2689614038 shutdown_connections 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd40813a2d0 0x7fd4080a7f50 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd408139ef0 0x7fd4080a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/2689614038 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 -- 172.21.15.98:0/2689614038 >> 172.21.15.98:0/2689614038 conn(0x7fd4080ac790 msgr2=0x7fd4080a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 -- 172.21.15.98:0/2689614038 shutdown_connections 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 -- 172.21.15.98:0/2689614038 wait complete. 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 10 monclient: monmap: 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.101 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.102 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 librados: starting msgr at 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 librados: starting objecter 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 Processor -- start 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 -- start start 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 librados: setting wanted keys 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 1 librados: calling monclient init 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.084+0000 7fd40cfb0640 10 monclient: init 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient: _add_conns ranks=[2,0,5,4,6,3,7,1,8] 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): picked mon.noname-b con 0x7fd408139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:58.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd408139ef0 0x7fd408134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): picked mon.noname-a con 0x7fd408139ef0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd40813a2d0 0x7fd40813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): picked mon.noname-h con 0x7fd40813a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd408143460 con 0x7fd408139ef0 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd4080a88a0 con 0x7fd408139b10 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd4081435e0 con 0x7fd40813a2d0 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): authenticate will time out at 1611.968421s 2024-09-11T03:00:58.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd408139ef0 0x7fd408134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408134a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58118/0 (socket says 172.21.15.98:58118) 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 -- 172.21.15.98:0/969837554 learned_addr learned my addr 172.21.15.98:0/969837554 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): get_auth_request con 0x7fd408139b10 auth_method 0 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd408139ef0 0x7fd408134f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33184/0 (socket says 172.21.15.98:33184) 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): get_auth_request con 0x7fd408139ef0 auth_method 0 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient(hunting): handle_auth_done global_id 64397 payload 824 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd406d9d640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd40813a2d0 0x7fd40813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd40813a2d0 msgr2=0x7fd40813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd40813a2d0 0x7fd40813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd408139ef0 msgr2=0x7fd408134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd408139ef0 0x7fd408134f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 monclient: found mon.noname-b 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 -- 172.21.15.98:0/969837554 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd408135830 con 0x7fd408139b10 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40659c640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408134a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800e890 tx=0x7fd3f800ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 client.?.objecter ms_handle_connect 0x7fd408139b10 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3f8017070 con 0x7fd408139b10 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: dump: 2024-09-11T03:00:58.107 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.108 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.109 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.109 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 5 monclient: authenticate success, global_id 64397 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3f800cdb0 con 0x7fd408139b10 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient: _renew_subs 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4080aa2c0 con 0x7fd408139b10 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3f80138e0 con 0x7fd408139b10 2024-09-11T03:00:58.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 monclient: dump: 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.111 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40559a640 4 set_mon_vals no callback set 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 client.64397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient: _renew_subs 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd40813ebc0 con 0x7fd408139b10 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 librados: init done 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b3b0 0x7fd40802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd40806ca10 con 0x7fd40813b3b0 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd3f8024060 con 0x7fd408139b10 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd3e803dea0 0x7fd3e8040360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7fd3f800f040 con 0x7fd408139b10 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 client.64397.objecter ms_dispatch 0x7fd408000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 3 client.64397.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 3 client.64397.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b3b0 0x7fd40802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient: get_auth_request con 0x7fd40813b3b0 auth_method 0 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 20 client.64397.objecter dump_active .. 0 homeless 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd406d9d640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd3e803dea0 0x7fd3e8040360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd406d9d640 10 monclient: get_auth_request con 0x7fd3e803dea0 auth_method 0 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_done global_id 74127 payload 824 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 1 -- 172.21.15.98:0/969837554 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd40806e680 con 0x7fd40813b3b0 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 client.64397.objecter ms_handle_connect 0x7fd3e803dea0 2024-09-11T03:00:58.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 client.64397.objecter ms_handle_connect 0x7fd40813b3b0 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3ff7fe640 10 client.64397.objecter resend_mon_ops 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd406d9d640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd3e803dea0 0x7fd3e8040360 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7fd4000ae000 tx=0x7fd4000a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.088+0000 7fd3fdd9b640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b3b0 0x7fd40802d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd40813c4a0 tx=0x7fd3f0011cf0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3f0019070 con 0x7fd40813b3b0 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 10 monclient: dump: 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.115 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd3f0004240 con 0x7fd40813b3b0 2024-09-11T03:00:58.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:58.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b3b0 msgr2=0x7fd40802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b3b0 0x7fd40802d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd40813c4a0 tx=0x7fd3f0011cf0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b3b0 msgr2=0x7fd40802d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:58.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.092+0000 7fd3ff7fe640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b3b0 0x7fd40802d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd40cfb0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b790 0x7fd40802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd40cfb0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd40806e870 con 0x7fd40813b790 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b790 0x7fd40802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient: get_auth_request con 0x7fd40813b790 auth_method 0 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 10 monclient(hunting): handle_auth_done global_id 74136 payload 824 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.096+0000 7fd3fdd9b640 1 -- 172.21.15.98:0/969837554 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd408072600 con 0x7fd40813b790 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3fdd9b640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b790 0x7fd40802d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd40813c4a0 tx=0x7fd3f00046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 10 client.64397.objecter ms_handle_connect 0x7fd40813b790 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 10 client.64397.objecter resend_mon_ops 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3f0019040 con 0x7fd40813b790 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 10 monclient: dump: 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.120 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7fd3f001e450 con 0x7fd40813b790 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b790 msgr2=0x7fd40802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b790 0x7fd40802d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd40813c4a0 tx=0x7fd3f00046d0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b790 msgr2=0x7fd40802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd3ff7fe640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd40813b790 0x7fd40802d990 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd40813c4a0 tx=0x7fd3f00046d0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:58.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 15031, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:58.123 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.125 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:58.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.127 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.128 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:58.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:58.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:58.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.135 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:58.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:58.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:58.142 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:58.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.144 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:58.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:58.146 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 10 librados: watch_flush enter 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 10 librados: watch_flush exit 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 20 client.64397.objecter shutdown clearing up homeless session... 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 10 client.64397.objecter successfully canceled tick 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd3e803dea0 msgr2=0x7fd3e8040360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd3e803dea0 0x7fd3e8040360 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7fd4000ae000 tx=0x7fd4000a7000 comp rx=0 tx=0).stop 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 10 monclient: shutdown 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 msgr2=0x7fd408134a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408134a00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd3f800e890 tx=0x7fd3f800ed50 comp rx=0 tx=0).stop 2024-09-11T03:00:58.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd406d9d640 1 -- 172.21.15.98:0/969837554 reap_dead start 2024-09-11T03:00:58.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 shutdown_connections 2024-09-11T03:00:58.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 --2- 172.21.15.98:0/969837554 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd408139b10 0x7fd408134a00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 >> 172.21.15.98:0/969837554 conn(0x7fd4080ac790 msgr2=0x7fd408138310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 shutdown_connections 2024-09-11T03:00:58.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 -- 172.21.15.98:0/969837554 wait complete. 2024-09-11T03:00:58.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.100+0000 7fd40cfb0640 1 librados: shutdown 2024-09-11T03:00:58.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:58.150 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T03:00:58.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:00:58.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient: get_monmap_and_config 2024-09-11T03:00:58.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient: monmap: 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:58.261752+0000 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:58.261752+0000 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.260 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:58.261 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 1 Processor -- start 2024-09-11T03:00:58.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 1 -- start start 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient: init 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient: _add_conns ranks=[9,7,1,2,6,10,8,4,3,0,5] 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc78139b10 0x7fdc78072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient(hunting): picked mon.noname-a con 0x7fdc78139b10 addr v2:172.21.15.98:3300/0 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fdc78139ef0 0x7fdc780a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient(hunting): picked mon.noname-i con 0x7fdc78139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdc7813a2d0 addr v1:172.21.15.190:6789/0 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc780702b0 con 0x7fdc7813a2d0 2024-09-11T03:00:58.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fdc78071bf0 con 0x7fdc78139b10 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fdc78071350 con 0x7fdc78139ef0 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7cd37640 10 monclient(hunting): authenticate will time out at 1612.142814s 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc78139b10 0x7fdc78072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fdc78139b10 0x7fdc78072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33200/0 (socket says 172.21.15.98:33200) 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 -- 172.21.15.98:0/376527182 learned_addr learned my addr 172.21.15.98:0/376527182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): get_auth_request con 0x7fdc78139b10 auth_method 0 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc75d9b640 1 --2- 172.21.15.98:0/376527182 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fdc78139ef0 0x7fdc780a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc75d9b640 10 monclient(hunting): get_auth_request con 0x7fdc78139ef0 auth_method 0 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc75d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient(hunting): handle_auth_done global_id 44712 payload 274 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 -- 172.21.15.98:0/376527182 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fdc78139ef0 msgr2=0x7fdc780a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 --2- 172.21.15.98:0/376527182 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fdc78139ef0 0x7fdc780a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 -- 172.21.15.98:0/376527182 >> v1:172.21.15.190:6789/0 conn(0x7fdc7813a2d0 legacy=0x7fdc780a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 monclient: found mon.noname-a 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc7659c640 1 -- 172.21.15.98:0/376527182 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc780a9620 con 0x7fdc78139b10 2024-09-11T03:00:58.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.260+0000 7fdc75d9b640 1 --2- 172.21.15.98:0/376527182 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fdc78139ef0 0x7fdc780a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7659c640 1 --2- 172.21.15.98:0/376527182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc78139b10 0x7fdc78072600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009870 tx=0x7fdc6801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8362ad47796efaa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 1 -- 172.21.15.98:0/376527182 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc680b0070 con 0x7fdc78139b10 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: dump: 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.276 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.277 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 1 -- 172.21.15.98:0/376527182 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc680aa810 con 0x7fdc78139b10 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 5 monclient: authenticate success, global_id 44712 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 1 -- 172.21.15.98:0/376527182 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc680a9300 con 0x7fdc78139b10 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 4 set_mon_vals no callback set 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7559a640 10 monclient: dump: 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.279 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 10 monclient: shutdown 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 -- 172.21.15.98:0/376527182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc78139b10 msgr2=0x7fdc78072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 --2- 172.21.15.98:0/376527182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc78139b10 0x7fdc78072600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009870 tx=0x7fdc6801c4b0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 -- 172.21.15.98:0/376527182 shutdown_connections 2024-09-11T03:00:58.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 --2- 172.21.15.98:0/376527182 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fdc78139ef0 0x7fdc780a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 --2- 172.21.15.98:0/376527182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdc78139b10 0x7fdc78072600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 -- 172.21.15.98:0/376527182 >> 172.21.15.98:0/376527182 conn(0x7fdc780ac790 msgr2=0x7fdc780a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 -- 172.21.15.98:0/376527182 shutdown_connections 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 -- 172.21.15.98:0/376527182 wait complete. 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 10 monclient: monmap: 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:58.281 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 librados: starting msgr at 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 librados: starting objecter 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 Processor -- start 2024-09-11T03:00:58.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 -- start start 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 librados: setting wanted keys 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 1 librados: calling monclient init 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.264+0000 7fdc7cd37640 10 monclient: init 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient: _add_conns ranks=[8,5,6,7,0,2,3,4,1] 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdc78139b10 0x7fdc780a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): picked mon.noname-e con 0x7fdc78139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc78139ef0 0x7fdc781409f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): picked mon.noname-h con 0x7fdc78139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdc7813a2d0 0x7fdc78140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): picked mon.noname-d con 0x7fdc7813a2d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fdc78144470 con 0x7fdc78139ef0 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdc780a8870 con 0x7fdc7813a2d0 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fdc781445f0 con 0x7fdc78139b10 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): authenticate will time out at 1612.148251s 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdc78139b10 0x7fdc780a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdc7813a2d0 0x7fdc78140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdc78139b10 0x7fdc780a4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35138/0 (socket says 172.21.15.98:35138) 2024-09-11T03:00:58.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 1 -- 172.21.15.98:0/4111771049 learned_addr learned my addr 172.21.15.98:0/4111771049 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): get_auth_request con 0x7fdc78139b10 auth_method 0 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): get_auth_request con 0x7fdc7813a2d0 auth_method 0 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc78139ef0 0x7fdc781409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 10 monclient(hunting): get_auth_request con 0x7fdc78139ef0 auth_method 0 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient(hunting): handle_auth_done global_id 44583 payload 824 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdc78139b10 msgr2=0x7fdc780a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdc78139b10 0x7fdc780a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc78139ef0 msgr2=0x7fdc781409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc78139ef0 0x7fdc781409f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 monclient: found mon.noname-d 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 -- 172.21.15.98:0/4111771049 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc780a6230 con 0x7fdc7813a2d0 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fdc78139b10 0x7fdc780a4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc78139ef0 0x7fdc781409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc76d9d640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdc7813a2d0 0x7fdc78140f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000c650 tx=0x7fdc7000cb10 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 client.?.objecter ms_handle_connect 0x7fdc7813a2d0 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc70003df0 con 0x7fdc7813a2d0 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:00:58.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: dump: 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.288 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc700050d0 con 0x7fdc7813a2d0 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 5 monclient: authenticate success, global_id 44583 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient: _renew_subs 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc78141470 con 0x7fdc7813a2d0 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc70005770 con 0x7fdc7813a2d0 2024-09-11T03:00:58.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: dump: 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.290 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 client.44583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient: _renew_subs 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc74d99640 4 set_mon_vals no callback set 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc780a5a80 con 0x7fdc7813a2d0 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 librados: init done 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813a760 0x7fdc7813ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fdc7806ca10 con 0x7fdc7813a760 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fdc70005300 con 0x7fdc7813a2d0 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813a760 0x7fdc7813ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient: get_auth_request con 0x7fdc7813a760 auth_method 0 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdc5803df40 0x7fdc58040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7fdc7004f150 con 0x7fdc7813a2d0 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 client.44583.objecter ms_dispatch 0x7fdc78000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 3 client.44583.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 3 client.44583.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 20 client.44583.objecter dump_active .. 0 homeless 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdc5803df40 0x7fdc58040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 10 monclient: get_auth_request con 0x7fdc5803df40 auth_method 0 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 10 monclient(hunting): handle_auth_done global_id 74111 payload 824 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 1 -- 172.21.15.98:0/4111771049 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc7813d5c0 con 0x7fdc7813a760 2024-09-11T03:00:58.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 client.44583.objecter ms_handle_connect 0x7fdc7813a760 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 client.44583.objecter resend_mon_ops 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc7659c640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813a760 0x7fdc7813ab40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc680097c0 tx=0x7fdc68019250 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 client.44583.objecter ms_handle_connect 0x7fdc5803df40 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc75d9b640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdc5803df40 0x7fdc58040400 secure :-1 s=READY pgs=3580 cs=0 l=1 rev1=1 crypto rx=0x7fdc78073770 tx=0x7fdc6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc680b0040 con 0x7fdc7813a760 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.268+0000 7fdc66ffd640 10 monclient: dump: 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.294 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.272+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdc680aa810 con 0x7fdc7813a760 2024-09-11T03:00:58.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.272+0000 7fdc66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.272+0000 7fdc66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:58.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.272+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813a760 msgr2=0x7fdc7813ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.272+0000 7fdc66ffd640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813a760 0x7fdc7813ab40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc680097c0 tx=0x7fdc68019250 comp rx=0 tx=0).stop 2024-09-11T03:00:58.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.272+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813a760 msgr2=0x7fdc7813ab40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:58.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.272+0000 7fdc66ffd640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813a760 0x7fdc7813ab40 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc680097c0 tx=0x7fdc68019250 comp rx=0 tx=0).stop 2024-09-11T03:00:58.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7cd37640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:58.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7cd37640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813b080 0x7fdc780a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7cd37640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fdc7806e870 con 0x7fdc7813b080 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813b080 0x7fdc780a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient: get_auth_request con 0x7fdc7813b080 auth_method 0 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 10 monclient(hunting): handle_auth_done global_id 74120 payload 824 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 1 -- 172.21.15.98:0/4111771049 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdc7806e680 con 0x7fdc7813b080 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc66ffd640 10 client.44583.objecter ms_handle_connect 0x7fdc7813b080 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc7659c640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813b080 0x7fdc780a0e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdc6801c4f0 tx=0x7fdc680a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc66ffd640 10 client.44583.objecter resend_mon_ops 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc680b0040 con 0x7fdc7813b080 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc66ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.276+0000 7fdc66ffd640 10 monclient: dump: 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.299 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7fdc680038f0 con 0x7fdc7813b080 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813b080 msgr2=0x7fdc780a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc66ffd640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813b080 0x7fdc780a0e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdc6801c4f0 tx=0x7fdc680a8000 comp rx=0 tx=0).stop 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc66ffd640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813b080 msgr2=0x7fdc780a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc66ffd640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdc7813b080 0x7fdc780a0e60 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdc6801c4f0 tx=0x7fdc680a8000 comp rx=0 tx=0).stop 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:58.301 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 15207, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:58.302 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.303 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.304 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:58.305 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.306 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.307 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:58.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:58.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.314 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:58.317 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:58.321 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:58.323 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.325 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:58.325 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:58.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 10 librados: watch_flush enter 2024-09-11T03:00:58.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 10 librados: watch_flush exit 2024-09-11T03:00:58.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 20 client.44583.objecter shutdown clearing up homeless session... 2024-09-11T03:00:58.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 10 client.44583.objecter successfully canceled tick 2024-09-11T03:00:58.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdc5803df40 msgr2=0x7fdc58040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdc5803df40 0x7fdc58040400 secure :-1 s=READY pgs=3580 cs=0 l=1 rev1=1 crypto rx=0x7fdc78073770 tx=0x7fdc6c008040 comp rx=0 tx=0).stop 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 10 monclient: shutdown 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdc7813a2d0 msgr2=0x7fdc78140f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 --2- 172.21.15.98:0/4111771049 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdc7813a2d0 0x7fdc78140f30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000c650 tx=0x7fdc7000cb10 comp rx=0 tx=0).stop 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc76d9d640 1 -- 172.21.15.98:0/4111771049 reap_dead start 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc76d9d640 1 -- 172.21.15.98:0/4111771049 reap_dead start 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 shutdown_connections 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 >> 172.21.15.98:0/4111771049 conn(0x7fdc780ac790 msgr2=0x7fdc780a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 shutdown_connections 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 -- 172.21.15.98:0/4111771049 wait complete. 2024-09-11T03:00:58.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.280+0000 7fdc7cd37640 1 librados: shutdown 2024-09-11T03:00:58.328 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:58.328 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T03:00:58.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:00:58.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient: get_monmap_and_config 2024-09-11T03:00:58.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient: monmap: 2024-09-11T03:00:58.440 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:58.442375+0000 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:58.442375+0000 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.441 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:58.442 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 Processor -- start 2024-09-11T03:00:58.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 -- start start 2024-09-11T03:00:58.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient: init 2024-09-11T03:00:58.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient: _add_conns ranks=[2,5,3,10,6,4,1,0,7,9,8] 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550139b10 0x7f5550072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): picked mon.noname-b con 0x7f5550139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5550139ef0 0x7f55500a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): picked mon.noname-h con 0x7f5550139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f555013a2d0 0x7f55500a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): picked mon.noname-g con 0x7f555013a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f55500702b0 con 0x7f5550139b10 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5550071bf0 con 0x7f555013a2d0 2024-09-11T03:00:58.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5550071350 con 0x7f5550139ef0 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5557e6e640 10 monclient(hunting): authenticate will time out at 1612.322702s 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550139b10 0x7f5550072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550139b10 0x7f5550072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58128/0 (socket says 172.21.15.98:58128) 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 -- 172.21.15.98:0/1091717078 learned_addr learned my addr 172.21.15.98:0/1091717078 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): get_auth_request con 0x7f5550139b10 auth_method 0 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555409640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5550139ef0 0x7f55500a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f555640b640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f555013a2d0 0x7f55500a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555409640 10 monclient(hunting): get_auth_request con 0x7f5550139ef0 auth_method 0 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555409640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555409640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555409640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f555640b640 10 monclient(hunting): get_auth_request con 0x7f555013a2d0 auth_method 0 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f555640b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f555640b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f555640b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient(hunting): handle_auth_done global_id 64406 payload 274 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 -- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5550139ef0 msgr2=0x7f55500a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5550139ef0 0x7f55500a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 -- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f555013a2d0 msgr2=0x7f55500a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f555013a2d0 0x7f55500a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 monclient: found mon.noname-b 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:00:58.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.440+0000 7f5555c0a640 1 -- 172.21.15.98:0/1091717078 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55500a9620 con 0x7f5550139b10 2024-09-11T03:00:58.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5555c0a640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550139b10 0x7f5550072600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5548009e80 tx=0x7f554801c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97d99a523ee13aa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 1 -- 172.21.15.98:0/1091717078 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55480090d0 con 0x7f5550139b10 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: dump: 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.456 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 5 monclient: authenticate success, global_id 64406 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 1 -- 172.21.15.98:0/1091717078 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55480a5a70 con 0x7f5550139b10 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 1 -- 172.21.15.98:0/1091717078 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55480a5c60 con 0x7f5550139b10 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 4 set_mon_vals no callback set 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5554c08640 10 monclient: dump: 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.458 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.459 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 10 monclient: shutdown 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 -- 172.21.15.98:0/1091717078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550139b10 msgr2=0x7f5550072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550139b10 0x7f5550072600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5548009e80 tx=0x7f554801c910 comp rx=0 tx=0).stop 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 -- 172.21.15.98:0/1091717078 shutdown_connections 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f555013a2d0 0x7f55500a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5550139ef0 0x7f55500a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 --2- 172.21.15.98:0/1091717078 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5550139b10 0x7f5550072600 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 -- 172.21.15.98:0/1091717078 >> 172.21.15.98:0/1091717078 conn(0x7f55500ac790 msgr2=0x7f55500a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 -- 172.21.15.98:0/1091717078 shutdown_connections 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 -- 172.21.15.98:0/1091717078 wait complete. 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 10 monclient: monmap: 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:58.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 librados: starting msgr at 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 librados: starting objecter 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 Processor -- start 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 -- start start 2024-09-11T03:00:58.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 librados: setting wanted keys 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 1 librados: calling monclient init 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.444+0000 7f5557e6e640 10 monclient: init 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient: _add_conns ranks=[7,0,2,1,3,8,5,4,6] 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5550139b10 0x7f55501d18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): picked mon.noname-i con 0x7f5550139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550139ef0 0x7f555013faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): picked mon.noname-a con 0x7f5550139ef0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f555013a2d0 0x7f555013ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): picked mon.noname-b con 0x7f555013a2d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5550143460 con 0x7f5550139ef0 2024-09-11T03:00:58.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f55500a88a0 con 0x7f555013a2d0 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f55501435e0 con 0x7f5550139b10 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): authenticate will time out at 1612.328127s 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f555013a2d0 0x7f555013ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f555013a2d0 0x7f555013ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58140/0 (socket says 172.21.15.98:58140) 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550139ef0 0x7f555013faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 1 -- 172.21.15.98:0/2054711450 learned_addr learned my addr 172.21.15.98:0/2054711450 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient(hunting): get_auth_request con 0x7f555013a2d0 auth_method 0 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): get_auth_request con 0x7f5550139ef0 auth_method 0 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5550139b10 0x7f55501d18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): get_auth_request con 0x7f5550139b10 auth_method 0 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient(hunting): handle_auth_done global_id 44721 payload 824 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5550139b10 msgr2=0x7f55501d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5550139b10 0x7f55501d18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f555013a2d0 msgr2=0x7f555013ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f555013a2d0 0x7f555013ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 monclient: found mon.noname-a 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 -- 172.21.15.98:0/2054711450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555013c230 con 0x7f5550139ef0 2024-09-11T03:00:58.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f555013a2d0 0x7f555013ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555409640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550139ef0 0x7f555013faa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f555013c400 tx=0x7f554000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 client.?.objecter ms_handle_connect 0x7f5550139ef0 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5540012a60 con 0x7f5550139ef0 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: dump: 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.467 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5540019070 con 0x7f5550139ef0 2024-09-11T03:00:58.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5540012c20 con 0x7f5550139ef0 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: dump: 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.469 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5547fff640 4 set_mon_vals no callback set 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 5 monclient: authenticate success, global_id 44721 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient: _renew_subs 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:00:58.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55500aa200 con 0x7f5550139ef0 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 client.44721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient: _renew_subs 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f555013e570 con 0x7f5550139ef0 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 librados: init done 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013a9f0 0x7f555013add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f555013e1f0 con 0x7f555013a9f0 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5540013070 con 0x7f5550139ef0 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f553403df40 0x7f5534040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013a9f0 0x7f555013add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7f554000f040 con 0x7f5550139ef0 2024-09-11T03:00:58.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 client.44721.objecter ms_dispatch 0x7f5550000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 3 client.44721.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 3 client.44721.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient: get_auth_request con 0x7f555013a9f0 auth_method 0 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 20 client.44721.objecter dump_active .. 0 homeless 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f553403df40 0x7f5534040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 10 monclient: get_auth_request con 0x7f553403df40 auth_method 0 2024-09-11T03:00:58.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 10 monclient(hunting): handle_auth_done global_id 19843 payload 824 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 1 -- 172.21.15.98:0/2054711450 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f555006e680 con 0x7f555013a9f0 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f5555c0a640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013a9f0 0x7f555013add0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f5548009e50 tx=0x7f55480196b0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 client.44721.objecter ms_handle_connect 0x7f555013a9f0 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 client.44721.objecter resend_mon_ops 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55480bd070 con 0x7f555013a9f0 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 monclient: dump: 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.473 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.474 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f55467fc640 10 client.44721.objecter ms_handle_connect 0x7f553403df40 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.448+0000 7f555640b640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f553403df40 0x7f5534040400 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7f555013ca90 tx=0x7f554c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.452+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5548004250 con 0x7f555013a9f0 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.452+0000 7f55467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.452+0000 7f55467fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.452+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013a9f0 msgr2=0x7f555013add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.452+0000 7f55467fc640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013a9f0 0x7f555013add0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f5548009e50 tx=0x7f55480196b0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.452+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013a9f0 msgr2=0x7f555013add0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:58.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.452+0000 7f55467fc640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013a9f0 0x7f555013add0 secure :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f5548009e50 tx=0x7f55480196b0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5557e6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:00:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5557e6e640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013b310 0x7f55500a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5557e6e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f555006ca10 con 0x7f555013b310 2024-09-11T03:00:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013b310 0x7f55500a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient: get_auth_request con 0x7f555013b310 auth_method 0 2024-09-11T03:00:58.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 10 monclient(hunting): handle_auth_done global_id 19852 payload 824 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 1 -- 172.21.15.98:0/2054711450 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f55500aa430 con 0x7f555013b310 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f55467fc640 10 client.44721.objecter ms_handle_connect 0x7f555013b310 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f55467fc640 10 client.44721.objecter resend_mon_ops 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f5555c0a640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013b310 0x7f55500a0e60 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5548000f90 tx=0x7f55480a7000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55480a5600 con 0x7f555013b310 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f55467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f55467fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:00:58.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.456+0000 7f55467fc640 10 monclient: dump: 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.479 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f55480c6070 con 0x7f555013b310 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f55467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f55467fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013b310 msgr2=0x7f55500a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f55467fc640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013b310 0x7f55500a0e60 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5548000f90 tx=0x7f55480a7000 comp rx=0 tx=0).stop 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f55467fc640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013b310 msgr2=0x7f55500a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f55467fc640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f555013b310 0x7f55500a0e60 secure :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f5548000f90 tx=0x7f55480a7000 comp rx=0 tx=0).stop 2024-09-11T03:00:58.480 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 674229, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:00:58.481 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.482 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:00:58.484 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:00:58.485 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:00:58.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.487 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.489 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:00:58.492 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:00:58.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.498 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:00:58.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:00:58.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:58.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:00:58.503 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:00:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:00:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:00:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:00:58.504 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:00:58.504 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:00:58.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 10 librados: watch_flush enter 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 10 librados: watch_flush exit 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 20 client.44721.objecter shutdown clearing up homeless session... 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 10 client.44721.objecter successfully canceled tick 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f553403df40 msgr2=0x7f5534040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f553403df40 0x7f5534040400 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7f555013ca90 tx=0x7f554c008040 comp rx=0 tx=0).stop 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 10 monclient: shutdown 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550139ef0 msgr2=0x7f555013faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 --2- 172.21.15.98:0/2054711450 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5550139ef0 0x7f555013faa0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f555013c400 tx=0x7f554000dab0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f555640b640 1 -- 172.21.15.98:0/2054711450 reap_dead start 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f555640b640 1 -- 172.21.15.98:0/2054711450 reap_dead start 2024-09-11T03:00:58.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 shutdown_connections 2024-09-11T03:00:58.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 >> 172.21.15.98:0/2054711450 conn(0x7f55500ac790 msgr2=0x7f5550135a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 shutdown_connections 2024-09-11T03:00:58.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 -- 172.21.15.98:0/2054711450 wait complete. 2024-09-11T03:00:58.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.460+0000 7f5557e6e640 1 librados: shutdown 2024-09-11T03:00:58.507 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:00:58.507 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T03:00:58.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T03:00:58.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.620+0000 7f29ba11f640 10 monclient: get_monmap_and_config 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.620+0000 7f29ba11f640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.620+0000 7f29ba11f640 10 monclient: monmap: 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:00:58.623897+0000 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:00:58.623897+0000 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:00:58.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:00:58.623 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.620+0000 7f29ba11f640 1 Processor -- start 2024-09-11T03:00:58.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.620+0000 7f29ba11f640 1 -- start start 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.620+0000 7f29ba11f640 10 monclient: init 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient: _add_conns ranks=[9,0,8,2,1,5,3,10,4,6,7] 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29b40a7aa0 0x7f29b40a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient(hunting): picked mon.noname-a con 0x7f29b40a7aa0 addr v2:172.21.15.98:3300/0 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29b40a15b0 addr v1:172.21.15.98:6789/0 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f29b40a8590 0x7f29b4140b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient(hunting): picked mon.noname-e con 0x7f29b40a8590 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b40a2ad0 con 0x7f29b40a15b0 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f29b40a3120 con 0x7f29b40a7aa0 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f29b40a1a30 con 0x7f29b40a8590 2024-09-11T03:00:58.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient(hunting): authenticate will time out at 1612.504386s 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f29b40a7aa0 0x7f29b40a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f29b40a15b0 0x7f29b40a7e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54678/0 (socket says 172.21.15.98:54678) 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b2ffd640 1 -- 172.21.15.98:0/2072016794 learned_addr learned my addr 172.21.15.98:0/2072016794 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): get_auth_request con 0x7f29b40a7aa0 auth_method 0 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 1 --2- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f29b40a8590 0x7f29b4140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 10 monclient(hunting): get_auth_request con 0x7f29b40a8590 auth_method 0 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient(hunting): handle_auth_done global_id 44730 payload 274 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 1 -- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f29b40a8590 msgr2=0x7f29b4140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 1 --2- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f29b40a8590 0x7f29b4140b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 1 -- 172.21.15.98:0/2072016794 >> v1:172.21.15.98:6789/0 conn(0x7f29b40a15b0 legacy=0x7f29b40a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 1 monclient: found mon.noname-a 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 1 -- 172.21.15.98:0/2072016794 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b40a96f0 con 0x7f29b40a7aa0 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b3fff640 1 --2- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f29b40a8590 0x7f29b4140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b37fe640 1 --2- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29b40a7aa0 0x7f29b40a38e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009d90 tx=0x7f29ac01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4fa6983746497ef server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 1 -- 172.21.15.98:0/2072016794 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29ac0b2070 con 0x7f29b40a7aa0 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: dump: 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.631 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.632 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 5 monclient: authenticate success, global_id 44730 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 1 -- 172.21.15.98:0/2072016794 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ac0a5a70 con 0x7f29b40a7aa0 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient: get_monmap_and_config success 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 1 -- 172.21.15.98:0/2072016794 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29ac0a5ca0 con 0x7f29b40a7aa0 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 4 set_mon_vals no callback set 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:00:58.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29b27fc640 10 monclient: dump: 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.634 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 10 monclient: shutdown 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29b40a7aa0 msgr2=0x7f29b40a38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 --2- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29b40a7aa0 0x7f29b40a38e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009d90 tx=0x7f29ac01c830 comp rx=0 tx=0).stop 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- 172.21.15.98:0/2072016794 shutdown_connections 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 --2- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f29b40a8590 0x7f29b4140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 --2- 172.21.15.98:0/2072016794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29b40a7aa0 0x7f29b40a38e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- 172.21.15.98:0/2072016794 >> 172.21.15.98:0/2072016794 conn(0x7f29b40ac7a0 msgr2=0x7f29b40a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- 172.21.15.98:0/2072016794 shutdown_connections 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.624+0000 7f29ba11f640 1 -- 172.21.15.98:0/2072016794 wait complete. 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: build_initial_monmap 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: monmap: 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:00:58.636 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 librados: starting msgr at 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 librados: starting objecter 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 Processor -- start 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 -- start start 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 librados: setting wanted keys 2024-09-11T03:00:58.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 librados: calling monclient init 2024-09-11T03:00:58.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: init 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _reopen_session rank -1 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _add_conns ranks=[3,5,4,0,8,1,6,7,2] 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b40a7aa0 0x7f29b4136c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): picked mon.noname-g con 0x7f29b40a7aa0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f29b40a8590 0x7f29b4137140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): picked mon.noname-h con 0x7f29b40a8590 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f29b41d1080 0x7f29b41d14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): picked mon.noname-c con 0x7f29b41d1080 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f29b4143c70 con 0x7f29b40a7aa0 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f29b4143af0 con 0x7f29b41d1080 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f29b4143df0 con 0x7f29b40a8590 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): _renew_subs 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient(hunting): authenticate will time out at 1612.509326s 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f29b41d1080 0x7f29b41d14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f29b41d1080 0x7f29b41d14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34592/0 (socket says 172.21.15.98:34592) 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 -- 172.21.15.98:0/4058335982 learned_addr learned my addr 172.21.15.98:0/4058335982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): get_auth_request con 0x7f29b41d1080 auth_method 0 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b2ffd640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f29b40a8590 0x7f29b4137140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b37fe640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b40a7aa0 0x7f29b4136c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b2ffd640 10 monclient(hunting): get_auth_request con 0x7f29b40a8590 auth_method 0 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b37fe640 10 monclient(hunting): get_auth_request con 0x7f29b40a7aa0 auth_method 0 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:00:58.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient(hunting): handle_auth_done global_id 24259 payload 824 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 -- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f29b40a8590 msgr2=0x7f29b4137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f29b40a8590 0x7f29b4137140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 -- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b40a7aa0 msgr2=0x7f29b4136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b40a7aa0 0x7f29b4136c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient: _finish_hunting 0 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 monclient: found mon.noname-c 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 -- 172.21.15.98:0/4058335982 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b413f940 con 0x7f29b41d1080 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b3fff640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f29b41d1080 0x7f29b41d14e0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e960 tx=0x7f29a400ee20 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 client.?.objecter ms_handle_connect 0x7f29b41d1080 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 1 -- 172.21.15.98:0/4058335982 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29a4017070 con 0x7f29b41d1080 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: dump: 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.646 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: _finish_auth 0 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: _check_auth_tickets 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 5 monclient: authenticate success, global_id 24259 2024-09-11T03:00:58.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 1 -- 172.21.15.98:0/4058335982 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29a400ce70 con 0x7f29b41d1080 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _renew_subs 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b40ab3b0 con 0x7f29b41d1080 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 1 -- 172.21.15.98:0/4058335982 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29a40138e0 con 0x7f29b41d1080 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 client.24259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 monclient: dump: 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.648 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:00:58.649 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b1ffb640 4 set_mon_vals no callback set 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _renew_subs 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b40a54e0 con 0x7f29b41d1080 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 librados: init done 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 1 -- 172.21.15.98:0/4058335982 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f29a4013470 con 0x7f29b41d1080 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b40a38e0 con 0x7f29b41d1080 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299803df40 0x7f2998040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:00:58.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 1 -- 172.21.15.98:0/4058335982 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(433..433 src has 1..433) ==== 6230+0+0 (secure 0 0 0) 0x7f29a400f040 con 0x7f29b41d1080 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 10 client.24259.objecter ms_dispatch 0x7f29b4000ba0 osd_map(433..433 src has 1..433) 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 3 client.24259.objecter handle_osd_map got epochs [433,433] > 0 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 3 client.24259.objecter handle_osd_map decoding full epoch 433 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29abfff640 20 client.24259.objecter dump_active .. 0 homeless 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b37fe640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299803df40 0x7f2998040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.628+0000 7f29b37fe640 10 monclient: get_auth_request con 0x7f299803df40 auth_method 0 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.632+0000 7f29abfff640 10 client.24259.objecter ms_handle_connect 0x7f299803df40 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.632+0000 7f29b37fe640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299803df40 0x7f2998040400 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0af000 tx=0x7f29ac0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.632+0000 7f29abfff640 1 -- 172.21.15.98:0/4058335982 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f29a4020070 con 0x7f29b41d1080 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.632+0000 7f29abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:00:58.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.632+0000 7f29abfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:00:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.860+0000 7f29ba11f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T03:00:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.860+0000 7f29ba11f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T03:00:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.860+0000 7f29ba11f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:00:58.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.860+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f29b413d5e0 con 0x7f29b41d1080 2024-09-11T03:00:58.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29abfff640 1 -- 172.21.15.98:0/4058335982 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f29a401ad70 con 0x7f29b41d1080 2024-09-11T03:00:58.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T03:00:58.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29abfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:00:58.866 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.866 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=1930129636,config=692619927,health=3244156202,kv=3991532845,logm=45848768}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=9,logm=64} crc {auth=4216296307,config=692619927,health=1200695584,kv=893004536,logm=2258900025}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=1,health=12,kv=10,logm=62} crc {auth=2649418866,config=1228700967,health=3244156203,kv=3991532845,logm=518995639}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=3740293404,config=692619927,health=3497721739,kv=3991532845,logm=2936164436}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.867 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=14,config=2,health=12,kv=10,logm=62} crc {auth=2332169903,config=692619927,health=1392961310,kv=3991532845,logm=3127746512}) 2024-09-11T03:00:58.868 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 10 librados: watch_flush enter 2024-09-11T03:00:58.868 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.868 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=15,config=2,health=12,kv=10,logm=61} crc {auth=2649418866,config=692619927,health=2229384721,kv=3991532845,logm=2832570301}) 2024-09-11T03:00:58.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 10 librados: watch_flush exit 2024-09-11T03:00:58.869 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 20 client.24259.objecter shutdown clearing up homeless session... 2024-09-11T03:00:58.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 10 client.24259.objecter successfully canceled tick 2024-09-11T03:00:58.870 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.870 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=15,config=2,health=12,kv=10,logm=61} crc {auth=2649418866,config=692619927,health=3244156202,kv=3991532845,logm=1765068249}) 2024-09-11T03:00:58.870 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.870 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1534535625,config=692619927,health=3244156202,kv=1551318347,logm=1441792139}) 2024-09-11T03:00:58.870 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.864+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=15,config=1,health=11,kv=9,logm=64} crc {auth=3220550032,config=1228700967,health=3920437608,kv=894428524,logm=3067970589}) 2024-09-11T03:00:58.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299803df40 msgr2=0x7f2998040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299803df40 0x7f2998040400 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7f29ac0af000 tx=0x7f29ac0a7070 comp rx=0 tx=0).stop 2024-09-11T03:00:58.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 10 monclient: shutdown 2024-09-11T03:00:58.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:00:58.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f29b41d1080 msgr2=0x7f29b41d14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f29b41d1080 0x7f29b41d14e0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e960 tx=0x7f29a400ee20 comp rx=0 tx=0).stop 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 shutdown_connections 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299803df40 0x7f2998040400 unknown :-1 s=CLOSED pgs=3582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f29b41d1080 0x7f29b41d14e0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f29b40a8590 0x7f29b4137140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 --2- 172.21.15.98:0/4058335982 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b40a7aa0 0x7f29b4136c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.864+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 >> 172.21.15.98:0/4058335982 conn(0x7f29b40ac7a0 msgr2=0x7f29b4073b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.868+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 shutdown_connections 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.868+0000 7f29ba11f640 1 -- 172.21.15.98:0/4058335982 wait complete. 2024-09-11T03:00:58.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:00:58.868+0000 7f29ba11f640 1 librados: shutdown 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611982719}) 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018225860}) 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.877 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646251703}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1304837888}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2183211121}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=385208140}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.878 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2379776811}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482157655}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.868+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3243404031}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384235227}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3988864481}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.879 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005396795}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=351468032}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=107094054}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3913090509}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3720002698}) 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.880 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2476548424}) 2024-09-11T03:00:58.881 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.876+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2196518716}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401667805}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2576994615}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=104127092}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.883 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=253281017}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=522506991}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=602963604}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2780036942}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=987981113}) 2024-09-11T03:00:58.884 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.880+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2872450055}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433524374}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461565450}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254476580}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.891 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4218258187}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=712291670}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.884+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1091062355}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=914024868}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3713334458}) 2024-09-11T03:00:58.892 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3462286563}) 2024-09-11T03:00:58.893 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T03:00:58.897 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.897 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.897 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142363062}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2781688221}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3356552977}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3557678284}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.888+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1405845635}) 2024-09-11T03:00:58.898 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3750175959}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2833407094}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705605919}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3239116703}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291344796}) 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.899 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919034115}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945594319}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=586133839}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3235881337}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.892+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=962155433}) 2024-09-11T03:00:58.900 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.901 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.901 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=102857189}) 2024-09-11T03:00:58.901 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.901 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=906458355}) 2024-09-11T03:00:58.901 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1048537128}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46397216}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327392772}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020937910}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.908 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2695510657}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2076831441}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1108073357}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1843864159}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923078837}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.896+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1058049620}) 2024-09-11T03:00:58.909 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535274531}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=92820464}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246499580}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4046736110}) 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.910 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2047988966}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1496460136}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1026194488}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1844022910}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.900+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2766453122}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605531047}) 2024-09-11T03:00:58.911 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1085467951}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1367514191}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=310986788}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1171992259}) 2024-09-11T03:00:58.912 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=634899271}) 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1527739965}) 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=23491122}) 2024-09-11T03:00:58.913 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.904+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=275629514}) 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449421996}) 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=861758469}) 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=277644431}) 2024-09-11T03:00:58.918 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2680228445}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4002048260}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=726155663}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.908+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1073848530}) 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.919 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2877159156}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98726231}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305988196}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146144017}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621255039}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.920 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1399663722}) 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=144953066}) 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.912+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2400275384}) 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=96413620}) 2024-09-11T03:00:58.921 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=510004200}) 2024-09-11T03:00:58.922 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=778600174}) 2024-09-11T03:00:58.924 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736180613}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792822129}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4235358971}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.916+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3943696053}) 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.925 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1590227801}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=592788541}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3229557518}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=721310373}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2471814065}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.926 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1156181161}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319150793}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3304214339}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4093546211}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4290370055}) 2024-09-11T03:00:58.927 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2462646198}) 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2241015483}) 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1277717483}) 2024-09-11T03:00:58.928 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.920+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1509627214}) 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762195094}) 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3102582179}) 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.929 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2308488092}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4075554909}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3378801166}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3662376374}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3561818655}) 2024-09-11T03:00:58.930 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045100632}) 2024-09-11T03:00:58.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.924+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1170197614}) 2024-09-11T03:00:58.931 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430691358}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1717813008}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2103031752}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=217718672}) 2024-09-11T03:00:58.932 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2821841263}) 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=100080970}) 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=652548869}) 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2669773601}) 2024-09-11T03:00:58.933 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.928+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=652900512}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498639315}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1722753686}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2939665191}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.935 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1255882185}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1889691704}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4274778671}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3599532050}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3562471034}) 2024-09-11T03:00:58.936 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.932+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=248094494}) 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1345068231}) 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817501401}) 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337678731}) 2024-09-11T03:00:58.940 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2651673029}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1465473405}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2880290146}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2105537968}) 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.941 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=725571426}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3033712256}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140581226}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4101695805}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1044959949}) 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.942 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=43949183}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2827661595}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1092421595}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=141584633}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=776532398}) 2024-09-11T03:00:58.943 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.936+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1049073514}) 2024-09-11T03:00:58.951 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2876235303}) 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987580960}) 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140819325}) 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.952 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=307942011}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1216725133}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2550876261}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2809235688}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129393856}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.940+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356376806}) 2024-09-11T03:00:58.953 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880334978}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786783545}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2947296934}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=964458164}) 2024-09-11T03:00:58.954 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1070105307}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2660746270}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4039104918}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=709424340}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1660566801}) 2024-09-11T03:00:58.955 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518825505}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1295348595}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1184600323}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.944+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2629484780}) 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.956 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1921528380}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1164520972}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3531932362}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358607489}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3412826003}) 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.957 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182060344}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987224969}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269922847}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=401321728}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4063056128}) 2024-09-11T03:00:58.958 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2323200597}) 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2598697586}) 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2934954981}) 2024-09-11T03:00:58.959 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.948+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4195341213}) 2024-09-11T03:00:58.966 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.966 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.966 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309595782}) 2024-09-11T03:00:58.966 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.966 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.966 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=668577514}) 2024-09-11T03:00:58.966 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=836669252}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=971563160}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1202082403}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2583725896}) 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.967 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1176916353}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2212606379}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.952+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607256967}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175119687}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2688787122}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.968 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=562293312}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3711392819}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1821069700}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3202076797}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.969 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1203822503}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1396985889}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1381606454}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=176660839}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3362219546}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3358683788}) 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.970 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2244409213}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1729583323}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2228996332}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=45769782}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1987003733}) 2024-09-11T03:00:58.971 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.956+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3299576573}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467242340}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224379774}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282794545}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.972 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2265709533}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3566937477}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1752264051}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=119978943}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=125371022}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.960+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=877149253}) 2024-09-11T03:00:58.973 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246562826}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497675182}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1647613526}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3257149500}) 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.974 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3401919110}) 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2939584307}) 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1782216604}) 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917734647}) 2024-09-11T03:00:58.975 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.964+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2906709491}) 2024-09-11T03:00:58.992 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.992 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022581846}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552293009}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155684192}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=565277046}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.993 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1711006498}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=800850622}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1955496670}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1097079222}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=565956327}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565303644}) 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.994 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440024397}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948566756}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3046954555}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2654515811}) 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.995 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1029547398}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3772076671}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1079169663}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.968+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4267656739}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524359281}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.996 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796967819}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571701397}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=279467295}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2474395495}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1883483577}) 2024-09-11T03:00:58.997 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2627502848}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2224968616}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.972+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768043286}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221664508}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744788364}) 2024-09-11T03:00:58.998 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=521468276}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1269575420}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2831749167}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2061689009}) 2024-09-11T03:00:58.999 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2543332976}) 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1581108206}) 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=455491671}) 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3994426372}) 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.000 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29740791}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1463422770}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=498692982}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4194286807}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1475412906}) 2024-09-11T03:00:59.001 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3009773378}) 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1483214736}) 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.976+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3408727841}) 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2064082316}) 2024-09-11T03:00:59.002 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2142681681}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128994332}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=62547497}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=236752356}) 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.003 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3519723114}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1152151952}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=563016575}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.980+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=634045996}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355884307}) 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.004 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1585184006}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2656464123}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2069243679}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3614339467}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.005 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3028558260}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3955673086}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2750002795}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.984+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1100017206}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2514740233}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3902098216}) 2024-09-11T03:00:59.006 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3658004693}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=454209113}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4015725454}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3634821426}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.007 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3515810644}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3584713890}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.988+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1164102461}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2326897253}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2691915890}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=451847703}) 2024-09-11T03:00:59.008 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1536223597}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2636888713}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2322436980}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2174169880}) 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.009 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878239901}) 2024-09-11T03:00:59.010 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1150833270}) 2024-09-11T03:00:59.037 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.037 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.037 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51332570}) 2024-09-11T03:00:59.037 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.037 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.037 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223538879}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648385439}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=254130412}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1212383849}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2053532104}) 2024-09-11T03:00:59.038 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3144355783}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1598961918}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.992+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=416588192}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781273453}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=771678341}) 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.039 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731290385}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4107405021}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1022880916}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3755974686}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.040 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3985760067}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1636126356}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:58.996+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2300165768}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148966529}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2325586304}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.041 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799095907}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1683548709}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=932466008}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3609744675}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2357269200}) 2024-09-11T03:00:59.042 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536535849}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.000+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3785275266}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517376182}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2473576073}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030953690}) 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.043 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2555726200}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4157353571}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2436079430}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3530262600}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792579966}) 2024-09-11T03:00:59.044 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867322578}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033608602}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3005656008}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3489160324}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.045 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3059423077}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=755097290}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=880421362}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=354223715}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=129634337}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.004+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=867057243}) 2024-09-11T03:00:59.046 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3656721697}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688524463}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936133073}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4131450207}) 2024-09-11T03:00:59.047 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2361538515}) 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1222380096}) 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3510269821}) 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468303215}) 2024-09-11T03:00:59.048 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=569713020}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232858414}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=477334426}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=432686800}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.049 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1652050943}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4232828053}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1479053840}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2206509175}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3065405531}) 2024-09-11T03:00:59.050 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.008+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1461022035}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270080270}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1391038756}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1119616351}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.051 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089244319}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2530238595}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3191917270}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=801024897}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2756529881}) 2024-09-11T03:00:59.052 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1109526969}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205798234}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.012+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=433353602}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469689341}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.053 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=426736801}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3868901095}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2901948211}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=614226668}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=934413647}) 2024-09-11T03:00:59.054 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1910733128}) 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258421853}) 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238150123}) 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3388926854}) 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.055 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3974732767}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1634384399}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3505891137}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3465464943}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1890601488}) 2024-09-11T03:00:59.056 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.016+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3107709185}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121379544}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863432189}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700661569}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4097751475}) 2024-09-11T03:00:59.057 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1299925571}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2087289084}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4180503546}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608319484}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3167745839}) 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.058 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2197865741}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4224960964}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515658483}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3615374104}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=151334803}) 2024-09-11T03:00:59.059 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2506996301}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=478394390}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4052360495}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.020+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2335068331}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066101216}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.060 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2615903818}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831081443}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=928692609}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1986220116}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.061 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3750991387}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=649878725}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2107475379}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3059736379}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187288032}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2145604134}) 2024-09-11T03:00:59.062 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=459200557}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458042632}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2088342487}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=643947892}) 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.063 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3229619044}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1028727279}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.024+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3705551697}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737635580}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3742780844}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.064 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180283735}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1770713636}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3362413476}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1146009373}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4103137273}) 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.065 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=989482844}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4036449847}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320126058}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1928359427}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205898936}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1652002239}) 2024-09-11T03:00:59.066 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=981656323}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=754914679}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2981646432}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2802095853}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.028+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2407498097}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.067 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976787383}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=444926686}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714019770}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1039347269}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2103040815}) 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.068 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1310650543}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2555605302}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3565993860}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1349125233}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087322649}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.069 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3759206367}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939909130}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=838953456}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=915444730}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.070 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1019098662}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4093197477}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2530532812}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.032+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2859603605}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150358974}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=379030027}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.071 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=228048581}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4044317062}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2840530320}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2251922062}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2786685140}) 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.072 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3994443917}) 2024-09-11T03:00:59.073 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=47441162}) 2024-09-11T03:00:59.147 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.147 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.147 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136580272}) 2024-09-11T03:00:59.147 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.147 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.147 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992855787}) 2024-09-11T03:00:59.147 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1421710494}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1412774615}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=261036213}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2432553240}) 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.148 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=616896791}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=455457811}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.036+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3558281084}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292683246}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3951185914}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.149 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2023521972}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2323195061}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2469997574}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=975764646}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=792194153}) 2024-09-11T03:00:59.150 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1047565689}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=16969899}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3887014851}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880655865}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041623030}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.151 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3280510153}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3897727641}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=789788618}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2552312067}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3117167447}) 2024-09-11T03:00:59.152 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.040+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2395967000}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928297040}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=378202305}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653081007}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.153 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3610569073}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=849540326}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3454531159}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1942832746}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1165106827}) 2024-09-11T03:00:59.154 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1445266203}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198565207}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1351350620}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=544607024}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2842542024}) 2024-09-11T03:00:59.155 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4049857458}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=539231542}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1743103402}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902055349}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.044+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=354321183}) 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.156 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19178999}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450534157}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126016875}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=789559848}) 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.157 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3659675440}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2361151740}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3407357988}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3027371471}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1579617964}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219218547}) 2024-09-11T03:00:59.158 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993236108}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588328186}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2673256228}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2084426763}) 2024-09-11T03:00:59.159 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=971134058}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2302307418}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1705355205}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.048+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2017074605}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508568352}) 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.160 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466093422}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548811273}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1181395244}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2753654849}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.161 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3920175459}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=45324657}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=153724904}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563476028}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132392990}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.162 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244015490}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1635118486}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3613240313}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1782644312}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1586728471}) 2024-09-11T03:00:59.163 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=828336228}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847763438}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.052+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2845002021}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116423873}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4204197923}) 2024-09-11T03:00:59.164 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3209433006}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=962513661}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1811951089}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=956820502}) 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.165 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2538505712}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1607489994}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1336257208}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2829690567}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2302904768}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=131618763}) 2024-09-11T03:00:59.166 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1547011936}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=50422714}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2684398555}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1624980676}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450413745}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.056+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=841648648}) 2024-09-11T03:00:59.167 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836728362}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4242002084}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2778353994}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371384682}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.168 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=85429982}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3464039119}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3489789840}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2936276713}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3838128627}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084134103}) 2024-09-11T03:00:59.169 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880338579}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594177637}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2561306288}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3886025326}) 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.170 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4074394868}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3294259487}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2745757269}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.060+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2133242616}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3751402988}) 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.171 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1300521422}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253376967}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4196489345}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2340924564}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.172 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=213564516}) 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3262538255}) 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1492964941}) 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3425808460}) 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655014923}) 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.173 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.064+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=140650838}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290203173}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3339309045}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976324584}) 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.174 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2033532260}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3256405800}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1052239797}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=299866603}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994388512}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.175 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899973452}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783708145}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1607142361}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=839502066}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.176 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1727876739}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1116039391}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1988640089}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471194733}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974348091}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=633423290}) 2024-09-11T03:00:59.177 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.068+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987788377}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=423886847}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3785864907}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2329310853}) 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.178 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3763353846}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1694632429}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291741356}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2885056987}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353272490}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.179 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2026435785}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=105495716}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=178695882}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2171173194}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.180 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2283148403}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842430084}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.072+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4156153870}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940496185}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1311804611}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616236658}) 2024-09-11T03:00:59.181 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1448923891}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=601185156}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2157911917}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1749204059}) 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.182 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1940937337}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4035543895}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773072727}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3744667425}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3506163726}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.183 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3653766925}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2489293526}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3268957806}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3451180508}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4046163744}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.076+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2078469806}) 2024-09-11T03:00:59.184 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184307521}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=842259488}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768119294}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4154188420}) 2024-09-11T03:00:59.185 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2764523100}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2247269116}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1651742801}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4069912214}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1813504092}) 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.186 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683890278}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2586257970}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2002497970}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3483278498}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=625113066}) 2024-09-11T03:00:59.187 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1963583832}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3325315365}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3374350580}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.080+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2897597041}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445271835}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.188 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4134939868}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984562110}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2405598122}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2706151046}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1594802092}) 2024-09-11T03:00:59.189 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=141907029}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2165423174}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.084+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2070670463}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811815571}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423804808}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.190 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2134516978}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=127674443}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4072954553}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2178447952}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=499553520}) 2024-09-11T03:00:59.191 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3840470453}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3828411734}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080282305}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3752291523}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4183053568}) 2024-09-11T03:00:59.192 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1518483690}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3824738929}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3304788600}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4258890030}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4265405918}) 2024-09-11T03:00:59.193 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.088+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2389052675}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806315504}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077556307}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1852722821}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.194 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=26405762}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1349518248}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1753753851}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3775289016}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4183771176}) 2024-09-11T03:00:59.195 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=192590904}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=3088828819,mdsmap=3491474967}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1144423289}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.092+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564862120}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882596200}) 2024-09-11T03:00:59.196 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1838184337}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=1724530838,mdsmap=696496163}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3551156433}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2315275365}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2234348955}) 2024-09-11T03:00:59.197 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=22} crc {logm=578779492,mdsmap=3491474967,mgr=4089010074,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3830183181,mon_config_key=2106052021,monmap=3140514009,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3429050040,osd_snap=923470457,osdmap=1261858950}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=31,osdmap=22} crc {logm=578779492,mdsmap=3491474967,mgr=3910322640,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2856002440,mon_config_key=3782314913,monmap=2495409540,nvmeofgw=3140514009,osd_metadata=1489146596,osd_pg_creating=3429050040,osd_snap=2469951386,osdmap=4179611738}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=20} crc {logm=578779492,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3188032072,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=2613832008,osd_metadata=1477139452,osd_pg_creating=3429050040,osd_snap=2130537020,osdmap=1571893538}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=3,mdsmap=1,mgr=7,mgr_command_descs=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=22} crc {logm=578779492,mdsmap=1228700967,mgr=839680553,mgr_command_descs=2308730518,mgrstat=4231004834,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3429050040,osd_snap=1819167050,osdmap=3845054397}) 2024-09-11T03:00:59.198 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=19} crc {logm=578779492,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2856002440,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=1596038039,osd_metadata=1477139452,osd_pg_creating=3429050040,osd_snap=2343134935,osdmap=1731438223}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=3,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=31,osdmap=23} crc {logm=578779492,mdsmap=3140514009,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=733514576,mon_config_key=3163979437,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1489146596,osd_pg_creating=3429050040,osd_snap=2328529210,osdmap=3301234094}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=3,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=32,osdmap=20} crc {logm=578779492,mdsmap=3491474967,mgr=741095862,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2856002440,mon_config_key=2997118629,monmap=416611756,nvmeofgw=598337686,osd_metadata=1863826368,osd_pg_creating=3429050040,osd_snap=1340965417,osdmap=1177293174}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=578570312,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=3033106205,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3429050040,osd_snap=121182886,osdmap=1708652614}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.096+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=2,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=9,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=22} crc {logm=73960208,mdsmap=3491474967,mgr=839680554,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1249783831,mon_config_key=2606551168,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3429050040,osd_snap=3689185621,osdmap=1460312597}) 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.199 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3031839859}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4051876022}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2938405632}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1607347852}) 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.200 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3576510990}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1326985294}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3859912171}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2215844680}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.100+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4177522629}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.201 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3394194665}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4246933050}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2035573499}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3171053540}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=898858952}) 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.202 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2147092774}) 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1328393956}) 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=401722014}) 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4244062626}) 2024-09-11T03:00:59.203 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=29384619}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3338206599}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3039663833}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1114504909}) 2024-09-11T03:00:59.204 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3302671583}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2747243307}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=4261475851}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1670069762}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.104+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4246336815}) 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.205 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1493943295}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1473567465}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=992576203}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2928939469}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.206 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3075726080}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2299864941}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=13262143}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2385672364}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.108+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3867647677}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.207 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2636543698}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2928876661}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4014697428}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2222618352}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2837005920}) 2024-09-11T03:00:59.208 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2506109273}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=341141954}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1636090453}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.112+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1456364057}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2082674654}) 2024-09-11T03:00:59.209 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4087003547}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3015572317}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1505331170}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=212679995}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.210 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=566344655}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2295018558}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3747683839}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.116+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2905492721}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=19} crc {osdmap=2612831944}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=18} crc {osdmap=603287439}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=19} crc {osdmap=2612831944}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=17} crc {osdmap=3553801905}) 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.211 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=19} crc {osdmap=2612831944}) 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=18} crc {osdmap=1458485790}) 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=19} crc {osdmap=2612831944}) 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=18} crc {osdmap=1705865917}) 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=19} crc {osdmap=2612831944}) 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=18} crc {osdmap=170673284}) 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=19} crc {osdmap=2612831944}) 2024-09-11T03:00:59.212 INFO:tasks.ceph.mon.e.smithi098.stderr:2024-09-11T03:00:59.132+0000 7fdf460b1640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=17} crc {osdmap=675761814}) 2024-09-11T03:00:59.894 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:00:59.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient: get_monmap_and_config 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient: monmap: 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:00.042770+0000 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:00.042770+0000 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:00.042 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 1 Processor -- start 2024-09-11T03:01:00.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 1 -- start start 2024-09-11T03:01:00.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient: init 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient: _add_conns ranks=[6,0,9,5,7,1,4,3,2,8,10] 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f900006f650 0x7f9000072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient(hunting): picked mon.noname-d con 0x7f900006f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90000a8380 addr v1:172.21.15.98:6789/0 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9000070140 0x7f9000140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient(hunting): picked mon.noname-a con 0x7f9000070140 addr v2:172.21.15.98:3300/0 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9000071a80 con 0x7f90000a8380 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f90000a8800 con 0x7f9000070140 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9000071d60 con 0x7f900006f650 2024-09-11T03:01:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f9004fea640 10 monclient(hunting): authenticate will time out at 1613.923022s 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9000070140 0x7f9000140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f9000070140 0x7f9000140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33234/0 (socket says 172.21.15.98:33234) 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 -- 172.21.15.98:0/4197055423 learned_addr learned my addr 172.21.15.98:0/4197055423 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): get_auth_request con 0x7f9000070140 auth_method 0 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ff5d9b640 1 --1- 172.21.15.98:0/4197055423 >> v1:172.21.15.98:6789/0 conn(0x7f90000a8380 0x7f900006fa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54692/0 (socket says 172.21.15.98:54692) 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 1 --2- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f900006f650 0x7f9000072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 10 monclient(hunting): get_auth_request con 0x7f900006f650 auth_method 0 2024-09-11T03:01:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient(hunting): handle_auth_done global_id 44748 payload 274 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 -- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f900006f650 msgr2=0x7f9000072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 --2- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f900006f650 0x7f9000072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 -- 172.21.15.98:0/4197055423 >> v1:172.21.15.98:6789/0 conn(0x7f90000a8380 legacy=0x7f900006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 monclient: found mon.noname-a 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffed9d640 1 -- 172.21.15.98:0/4197055423 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90000714c0 con 0x7f9000070140 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffdd9b640 1 -- 172.21.15.98:0/4197055423 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101490977 0 0) 0x7f9000071a80 con 0x7f90000a8380 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffe59c640 1 --2- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f900006f650 0x7f9000072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.040+0000 7f8ffdd9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffed9d640 1 --2- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9000070140 0x7f9000140c20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009e50 tx=0x7f8ff801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebd7a615867b30e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 1 -- 172.21.15.98:0/4197055423 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ff80090d0 con 0x7f9000070140 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: dump: 2024-09-11T03:01:00.054 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 5 monclient: authenticate success, global_id 44748 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 1 -- 172.21.15.98:0/4197055423 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ff80a5c40 con 0x7f9000070140 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 4 set_mon_vals no callback set 2024-09-11T03:01:00.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 1 -- 172.21.15.98:0/4197055423 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ff80a5df0 con 0x7f9000070140 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f8ffdd9b640 10 monclient: dump: 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.057 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 10 monclient: shutdown 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 -- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9000070140 msgr2=0x7f9000140c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 --2- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9000070140 0x7f9000140c20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8009e50 tx=0x7f8ff801c8f0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 -- 172.21.15.98:0/4197055423 shutdown_connections 2024-09-11T03:01:00.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 --2- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9000070140 0x7f9000140c20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 --2- 172.21.15.98:0/4197055423 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f900006f650 0x7f9000072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 -- 172.21.15.98:0/4197055423 >> 172.21.15.98:0/4197055423 conn(0x7f90000ac8e0 msgr2=0x7f90000a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 -- 172.21.15.98:0/4197055423 shutdown_connections 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 -- 172.21.15.98:0/4197055423 wait complete. 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 10 monclient: monmap: 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.059 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 librados: starting msgr at 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 librados: starting objecter 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 Processor -- start 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 -- start start 2024-09-11T03:01:00.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 librados: setting wanted keys 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 1 librados: calling monclient init 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 10 monclient: init 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.044+0000 7f9004fea640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient: _add_conns ranks=[1,2,5,7,3,8,6,0,4] 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f900006f650 0x7f900013a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): picked mon.noname-f con 0x7f900006f650 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9000070140 0x7f900013aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): picked mon.noname-b con 0x7f9000070140 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90000a8380 0x7f900013afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): picked mon.noname-h con 0x7f90000a8380 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9000143d10 con 0x7f900006f650 2024-09-11T03:01:00.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9000143b90 con 0x7f9000070140 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9000143e90 con 0x7f90000a8380 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient(hunting): authenticate will time out at 1613.928088s 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9000070140 0x7f900013aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9000070140 0x7f900013aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58154/0 (socket says 172.21.15.98:58154) 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 -- 172.21.15.98:0/1883534747 learned_addr learned my addr 172.21.15.98:0/1883534747 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): get_auth_request con 0x7f9000070140 auth_method 0 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffe59c640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f900006f650 0x7f900013a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient(hunting): handle_auth_done global_id 64424 payload 824 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffed9d640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90000a8380 0x7f900013afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 -- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90000a8380 msgr2=0x7f900013afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90000a8380 0x7f900013afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 -- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f900006f650 msgr2=0x7f900013a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f900006f650 0x7f900013a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 monclient: found mon.noname-b 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 -- 172.21.15.98:0/1883534747 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900013d510 con 0x7f9000070140 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffe59c640 10 monclient: get_auth_request con 0x7f900006f650 auth_method 0 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffe59c640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f900006f650 0x7f900013a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff5d9b640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9000070140 0x7f900013aa60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c3f0 tx=0x7f8fe800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 client.?.objecter ms_handle_connect 0x7f9000070140 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 1 -- 172.21.15.98:0/1883534747 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8fe80047f0 con 0x7f9000070140 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:01:00.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: dump: 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.065 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 5 monclient: authenticate success, global_id 64424 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 1 -- 172.21.15.98:0/1883534747 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe80049b0 con 0x7f9000070140 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 1 -- 172.21.15.98:0/1883534747 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8fe80133c0 con 0x7f9000070140 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffd59a640 4 set_mon_vals no callback set 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:00.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 monclient: dump: 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.067 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient: _renew_subs 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9000140e00 con 0x7f9000070140 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 client.64424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient: _renew_subs 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f900013d230 con 0x7f9000070140 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 librados: init done 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9000077ca0 con 0x7f9000070140 2024-09-11T03:01:00.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 1 -- 172.21.15.98:0/1883534747 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8fe8004b60 con 0x7f9000070140 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fe003df40 0x7f8fe0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 1 -- 172.21.15.98:0/1883534747 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(434..434 src has 1..434) ==== 6230+0+0 (secure 0 0 0) 0x7f8fe804f510 con 0x7f9000070140 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 client.64424.objecter ms_dispatch 0x7f9000000ba0 osd_map(434..434 src has 1..434) 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 3 client.64424.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 3 client.64424.objecter handle_osd_map decoding full epoch 434 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 20 client.64424.objecter dump_active .. 0 homeless 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffe59c640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fe003df40 0x7f8fe0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffe59c640 10 monclient: get_auth_request con 0x7f8fe003df40 auth_method 0 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ff77fe640 10 client.64424.objecter ms_handle_connect 0x7f8fe003df40 2024-09-11T03:01:00.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.048+0000 7f8ffe59c640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fe003df40 0x7f8fe0040400 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009ae0 tx=0x7f8ff0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.052+0000 7f8ff77fe640 1 -- 172.21.15.98:0/1883534747 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8fe8011360 con 0x7f9000070140 2024-09-11T03:01:00.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.052+0000 7f8ff77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.052+0000 7f8ff77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:00.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.288+0000 7f9004fea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:01:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.288+0000 7f9004fea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:01:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.288+0000 7f9004fea640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.288+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9000072640 con 0x7f9000070140 2024-09-11T03:01:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f8ff77fe640 1 -- 172.21.15.98:0/1883534747 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f8fe8013a80 con 0x7f9000070140 2024-09-11T03:01:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f8ff77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:01:00.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f8ff77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:00.292 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":76,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 10 librados: watch_flush enter 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 10 librados: watch_flush exit 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 20 client.64424.objecter shutdown clearing up homeless session... 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 10 client.64424.objecter successfully canceled tick 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fe003df40 msgr2=0x7f8fe0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fe003df40 0x7f8fe0040400 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f8ff0009ae0 tx=0x7f8ff0008040 comp rx=0 tx=0).stop 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 10 monclient: shutdown 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9000070140 msgr2=0x7f900013aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9000070140 0x7f900013aa60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8fe800c3f0 tx=0x7f8fe800c8b0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 shutdown_connections 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90000a8380 0x7f900013afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8fe003df40 0x7f8fe0040400 unknown :-1 s=CLOSED pgs=3583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9000070140 0x7f900013aa60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 --2- 172.21.15.98:0/1883534747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f900006f650 0x7f900013a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 >> 172.21.15.98:0/1883534747 conn(0x7f90000ac8e0 msgr2=0x7f90000a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 shutdown_connections 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 -- 172.21.15.98:0/1883534747 wait complete. 2024-09-11T03:01:00.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.292+0000 7f9004fea640 1 librados: shutdown 2024-09-11T03:01:00.310 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:01:00.310 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T03:01:00.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T03:01:00.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:01:00.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient: get_monmap_and_config 2024-09-11T03:01:00.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient: monmap: 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:00.430428+0000 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:00.430428+0000 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.429 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.430 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.430 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.430 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:00.430 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:00.430 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 1 Processor -- start 2024-09-11T03:01:00.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 1 -- start start 2024-09-11T03:01:00.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient: init 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient: _add_conns ranks=[4,7,1,8,6,0,10,3,2,5,9] 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed88139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient(hunting): picked mon.noname-c con 0x7fed88139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fed8806fe50 0x7fed880a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient(hunting): picked mon.noname-i con 0x7fed8806fe50 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fed88072640 addr v1:172.21.15.190:6789/0 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed88071370 con 0x7fed88072640 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fed880719c0 con 0x7fed88139b80 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fed88071c10 con 0x7fed8806fe50 2024-09-11T03:01:00.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8ed11640 10 monclient(hunting): authenticate will time out at 1614.310926s 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed88139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed88139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34598/0 (socket says 172.21.15.98:34598) 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 -- 172.21.15.98:0/992798652 learned_addr learned my addr 172.21.15.98:0/992798652 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): get_auth_request con 0x7fed88139b80 auth_method 0 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed87fff640 1 --2- 172.21.15.98:0/992798652 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fed8806fe50 0x7fed880a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed87fff640 10 monclient(hunting): get_auth_request con 0x7fed8806fe50 auth_method 0 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed87fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient(hunting): handle_auth_done global_id 24268 payload 274 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 -- 172.21.15.98:0/992798652 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fed8806fe50 msgr2=0x7fed880a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 --2- 172.21.15.98:0/992798652 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fed8806fe50 0x7fed880a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 -- 172.21.15.98:0/992798652 >> v1:172.21.15.190:6789/0 conn(0x7fed88072640 legacy=0x7fed880a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 monclient: found mon.noname-c 2024-09-11T03:01:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:01:00.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.428+0000 7fed8caad640 1 -- 172.21.15.98:0/992798652 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed880a9600 con 0x7fed88139b80 2024-09-11T03:01:00.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8caad640 1 --2- 172.21.15.98:0/992798652 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed88139f60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fed740098d0 tx=0x7fed7401c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=6a742ad0377e8590 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 1 -- 172.21.15.98:0/992798652 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed740b1070 con 0x7fed88139b80 2024-09-11T03:01:00.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: dump: 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.445 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 1 -- 172.21.15.98:0/992798652 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed740aa7b0 con 0x7fed88139b80 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 5 monclient: authenticate success, global_id 24268 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:00.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 4 set_mon_vals no callback set 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 1 -- 172.21.15.98:0/992798652 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed740aae50 con 0x7fed88139b80 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed877fe640 10 monclient: dump: 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.447 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 10 monclient: shutdown 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 -- 172.21.15.98:0/992798652 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 msgr2=0x7fed88139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 --2- 172.21.15.98:0/992798652 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed88139f60 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fed740098d0 tx=0x7fed7401c440 comp rx=0 tx=0).stop 2024-09-11T03:01:00.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 -- 172.21.15.98:0/992798652 shutdown_connections 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 --2- 172.21.15.98:0/992798652 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fed8806fe50 0x7fed880a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 --2- 172.21.15.98:0/992798652 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed88139f60 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 -- 172.21.15.98:0/992798652 >> 172.21.15.98:0/992798652 conn(0x7fed880ac730 msgr2=0x7fed880a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 -- 172.21.15.98:0/992798652 shutdown_connections 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 1 -- 172.21.15.98:0/992798652 wait complete. 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.432+0000 7fed8ed11640 10 monclient: monmap: 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:00.449 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.450 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 librados: starting msgr at 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 librados: starting objecter 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 Processor -- start 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 -- start start 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 librados: setting wanted keys 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 librados: calling monclient init 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient: init 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient: _add_conns ranks=[6,2,4,8,5,3,0,1,7] 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8806fe50 0x7fed8813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): picked mon.noname-d con 0x7fed8806fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed88072640 0x7fed8813ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): picked mon.noname-b con 0x7fed88072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed8813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): picked mon.noname-c con 0x7fed88139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fed88144450 con 0x7fed88072640 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fed880a8870 con 0x7fed88139b80 2024-09-11T03:01:00.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fed881445d0 con 0x7fed8806fe50 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): authenticate will time out at 1614.316763s 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed8813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed88072640 0x7fed8813ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8806fe50 0x7fed8813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed88072640 0x7fed8813ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58162/0 (socket says 172.21.15.98:58162) 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed8813b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34614/0 (socket says 172.21.15.98:34614) 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 1 -- 172.21.15.98:0/3309762054 learned_addr learned my addr 172.21.15.98:0/3309762054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): get_auth_request con 0x7fed88072640 auth_method 0 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): get_auth_request con 0x7fed8806fe50 auth_method 0 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient(hunting): get_auth_request con 0x7fed88139b80 auth_method 0 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient(hunting): handle_auth_done global_id 44601 payload 824 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 msgr2=0x7fed8813b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed8813b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed88072640 msgr2=0x7fed8813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed88072640 0x7fed8813ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 monclient: found mon.noname-d 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 -- 172.21.15.98:0/3309762054 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8813cd80 con 0x7fed8806fe50 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fed88072640 0x7fed8813ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fed88139b80 0x7fed8813b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8caad640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8806fe50 0x7fed8813a7d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed740ab000 tx=0x7fed740191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 client.?.objecter ms_handle_connect 0x7fed8806fe50 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:00.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed740b1040 con 0x7fed8806fe50 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: dump: 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.457 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 5 monclient: authenticate success, global_id 44601 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed740ba070 con 0x7fed8806fe50 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient: _renew_subs 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8813d060 con 0x7fed8806fe50 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed740aadd0 con 0x7fed8806fe50 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed86ffd640 4 set_mon_vals no callback set 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:00.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 client.44601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: dump: 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.460 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient: _renew_subs 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fed88134800 con 0x7fed8806fe50 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 librados: init done 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fed740b4070 con 0x7fed8806fe50 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88133f20 0x7fed8802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed8813bb40 con 0x7fed88133f20 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88133f20 0x7fed8802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient: get_auth_request con 0x7fed88133f20 auth_method 0 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed6c03df40 0x7fed6c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(434..434 src has 1..434) ==== 6230+0+0 (secure 0 0 0) 0x7fed740e90e0 con 0x7fed8806fe50 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 client.44601.objecter ms_dispatch 0x7fed88000ba0 osd_map(434..434 src has 1..434) 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 3 client.44601.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 3 client.44601.objecter handle_osd_map decoding full epoch 434 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 10 monclient(hunting): handle_auth_done global_id 44766 payload 824 2024-09-11T03:01:00.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 20 client.44601.objecter dump_active .. 0 homeless 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 1 -- 172.21.15.98:0/3309762054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed880aa120 con 0x7fed88133f20 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 client.44601.objecter ms_handle_connect 0x7fed88133f20 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed6c03df40 0x7fed6c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 client.44601.objecter resend_mon_ops 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed87fff640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88133f20 0x7fed8802d450 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed880a4cb0 tx=0x7fed7c00e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed8d2ae640 10 monclient: get_auth_request con 0x7fed6c03df40 auth_method 0 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed7c018070 con 0x7fed88133f20 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:00.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: dump: 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed7c01d450 con 0x7fed88133f20 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88133f20 msgr2=0x7fed8802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88133f20 0x7fed8802d450 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed880a4cb0 tx=0x7fed7c00e5a0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88133f20 msgr2=0x7fed8802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.436+0000 7fed857fa640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88133f20 0x7fed8802d450 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.440+0000 7fed857fa640 10 client.44601.objecter ms_handle_connect 0x7fed6c03df40 2024-09-11T03:01:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.440+0000 7fed8d2ae640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed6c03df40 0x7fed6c040400 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 crypto rx=0x7fed800012f0 tx=0x7fed80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88134300 0x7fed8802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed8806ca10 con 0x7fed88134300 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88134300 0x7fed8802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient: get_auth_request con 0x7fed88134300 auth_method 0 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 10 monclient(hunting): handle_auth_done global_id 44775 payload 824 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 1 -- 172.21.15.98:0/3309762054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed880a82b0 con 0x7fed88134300 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed87fff640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88134300 0x7fed8802d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed7c01a000 tx=0x7fed7c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 10 client.44601.objecter ms_handle_connect 0x7fed88134300 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 10 client.44601.objecter resend_mon_ops 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed7c014e00 con 0x7fed88134300 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 10 monclient: dump: 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fed7c014990 con 0x7fed88134300 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88134300 msgr2=0x7fed8802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88134300 0x7fed8802d990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed7c01a000 tx=0x7fed7c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88134300 msgr2=0x7fed8802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed857fa640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed88134300 0x7fed8802d990 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed7c01a000 tx=0x7fed7c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 286884, 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:00.472 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:00.473 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.474 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.475 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:00.476 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:00.477 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.478 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.480 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.484 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.485 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.487 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:00.487 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:00.487 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.487 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.487 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.488 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:00.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:00.488 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.488 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.488 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.489 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:00.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.489 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.490 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:00.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.490 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.490 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.491 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:00.491 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:00.491 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.491 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.491 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.491 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.491 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.492 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:00.492 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:00.492 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.492 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.492 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.492 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:00.493 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:00.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:00.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:00.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.496 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:00.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:00.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:00.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:00.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:00.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:00.499 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.499 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.499 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.500 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:00.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:00.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:00.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:00.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.502 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:00.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:00.503 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.503 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.503 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.504 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:00.504 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:00.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:00.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:00.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:00.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:00.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:00.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.509 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:00.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.511 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:01:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.512 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:00.512 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 10 librados: watch_flush enter 2024-09-11T03:01:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 10 librados: watch_flush exit 2024-09-11T03:01:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 20 client.44601.objecter shutdown clearing up homeless session... 2024-09-11T03:01:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 10 client.44601.objecter successfully canceled tick 2024-09-11T03:01:00.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed6c03df40 msgr2=0x7fed6c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fed6c03df40 0x7fed6c040400 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 crypto rx=0x7fed800012f0 tx=0x7fed80009040 comp rx=0 tx=0).stop 2024-09-11T03:01:00.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 10 monclient: shutdown 2024-09-11T03:01:00.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8806fe50 msgr2=0x7fed8813a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 --2- 172.21.15.98:0/3309762054 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fed8806fe50 0x7fed8813a7d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed740ab000 tx=0x7fed740191e0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8d2ae640 1 -- 172.21.15.98:0/3309762054 reap_dead start 2024-09-11T03:01:00.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8d2ae640 1 -- 172.21.15.98:0/3309762054 reap_dead start 2024-09-11T03:01:00.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 shutdown_connections 2024-09-11T03:01:00.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 >> 172.21.15.98:0/3309762054 conn(0x7fed880ac730 msgr2=0x7fed88134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 shutdown_connections 2024-09-11T03:01:00.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 -- 172.21.15.98:0/3309762054 wait complete. 2024-09-11T03:01:00.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.444+0000 7fed8ed11640 1 librados: shutdown 2024-09-11T03:01:00.523 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:00.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T03:01:00.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:01:00.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient: get_monmap_and_config 2024-09-11T03:01:00.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient: monmap: 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:00.651405+0000 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:00.651405+0000 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.650 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:00.651 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 Processor -- start 2024-09-11T03:01:00.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 -- start start 2024-09-11T03:01:00.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient: init 2024-09-11T03:01:00.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient: _add_conns ranks=[8,4,10,6,9,1,2,0,5,7,3] 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac00a3de0 0x7f2ac00a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): picked mon.noname-e con 0x7f2ac00a3de0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ac00a41c0 0x7f2ac00a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): picked mon.noname-c con 0x7f2ac00a41c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f2ac009b240 0x7f2ac009b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): picked mon.noname-f con 0x7f2ac009b240 addr v2:172.21.15.190:3300/0 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f2ac00a0170 con 0x7f2ac009b240 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2ac0068fa0 con 0x7f2ac00a41c0 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2ac0068b40 con 0x7f2ac00a3de0 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac73f2640 10 monclient(hunting): authenticate will time out at 1614.531625s 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac498d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ac00a41c0 0x7f2ac00a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac498d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ac00a41c0 0x7f2ac00a45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34626/0 (socket says 172.21.15.98:34626) 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac498d640 1 -- 172.21.15.98:0/1413138225 learned_addr learned my addr 172.21.15.98:0/1413138225 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac498d640 10 monclient(hunting): get_auth_request con 0x7f2ac00a41c0 auth_method 0 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac498d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac498d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac498d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac518e640 1 --2- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac00a3de0 0x7f2ac00a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac518e640 10 monclient(hunting): get_auth_request con 0x7f2ac00a3de0 auth_method 0 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac518e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac518e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.648+0000 7f2ac518e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac598f640 1 --2- 172.21.15.98:0/1413138225 >> v2:172.21.15.190:3300/0 conn(0x7f2ac009b240 0x7f2ac009b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac598f640 10 monclient(hunting): get_auth_request con 0x7f2ac009b240 auth_method 0 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac598f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac598f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac598f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 10 monclient(hunting): handle_auth_done global_id 24803 payload 274 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 1 -- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ac00a41c0 msgr2=0x7f2ac00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 1 --2- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ac00a41c0 0x7f2ac00a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 1 -- 172.21.15.98:0/1413138225 >> v2:172.21.15.190:3300/0 conn(0x7f2ac009b240 msgr2=0x7f2ac009b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 1 --2- 172.21.15.98:0/1413138225 >> v2:172.21.15.190:3300/0 conn(0x7f2ac009b240 0x7f2ac009b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 1 monclient: found mon.noname-e 2024-09-11T03:01:00.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 1 -- 172.21.15.98:0/1413138225 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac009eaf0 con 0x7f2ac00a3de0 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac518e640 1 --2- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac00a3de0 0x7f2ac00a0570 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097c0 tx=0x7f2ab001c360 comp rx=0 tx=0).ready entity=mon.8 client_cookie=1e5fed72de3cdd0d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 1 -- 172.21.15.98:0/1413138225 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2ab00b0070 con 0x7f2ac00a3de0 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: dump: 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.666 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 5 monclient: authenticate success, global_id 24803 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 1 -- 172.21.15.98:0/1413138225 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab00aa810 con 0x7f2ac00a3de0 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:00.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 4 set_mon_vals no callback set 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 1 -- 172.21.15.98:0/1413138225 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2ab00a9300 con 0x7f2ac00a3de0 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ab7fff640 10 monclient: dump: 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 10 monclient: shutdown 2024-09-11T03:01:00.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 -- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac00a3de0 msgr2=0x7f2ac00a0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac00a3de0 0x7f2ac00a0570 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00097c0 tx=0x7f2ab001c360 comp rx=0 tx=0).stop 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 -- 172.21.15.98:0/1413138225 shutdown_connections 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/1413138225 >> v2:172.21.15.190:3300/0 conn(0x7f2ac009b240 0x7f2ac009b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2ac00a41c0 0x7f2ac00a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/1413138225 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac00a3de0 0x7f2ac00a0570 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 -- 172.21.15.98:0/1413138225 >> 172.21.15.98:0/1413138225 conn(0x7f2ac00a4ed0 msgr2=0x7f2ac00a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 -- 172.21.15.98:0/1413138225 shutdown_connections 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 1 -- 172.21.15.98:0/1413138225 wait complete. 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.652+0000 7f2ac73f2640 10 monclient: monmap: 2024-09-11T03:01:00.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 librados: starting msgr at 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 librados: starting objecter 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 Processor -- start 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 -- start start 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 librados: setting wanted keys 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 librados: calling monclient init 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient: init 2024-09-11T03:01:00.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient: _add_conns ranks=[8,1,2,0,6,7,4,3,5] 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac009b240 0x7f2ac0137790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): picked mon.noname-e con 0x7f2ac009b240 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac00a3de0 0x7f2ac0137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): picked mon.noname-f con 0x7f2ac00a3de0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac00a41c0 0x7f2ac013ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): picked mon.noname-b con 0x7f2ac00a41c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac01413a0 con 0x7f2ac00a3de0 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac0141220 con 0x7f2ac00a41c0 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2ac0141520 con 0x7f2ac009b240 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): authenticate will time out at 1614.537059s 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac009b240 0x7f2ac0137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac009b240 0x7f2ac0137790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35178/0 (socket says 172.21.15.98:35178) 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 -- 172.21.15.98:0/320516083 learned_addr learned my addr 172.21.15.98:0/320516083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): get_auth_request con 0x7f2ac009b240 auth_method 0 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac00a41c0 0x7f2ac013ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient(hunting): get_auth_request con 0x7f2ac00a41c0 auth_method 0 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac00a3de0 0x7f2ac0137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): get_auth_request con 0x7f2ac00a3de0 auth_method 0 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient(hunting): handle_auth_done global_id 24812 payload 824 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac00a41c0 msgr2=0x7f2ac013ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac00a41c0 0x7f2ac013ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac00a3de0 msgr2=0x7f2ac0137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ac00a3de0 0x7f2ac0137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 monclient: found mon.noname-e 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 -- 172.21.15.98:0/320516083 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac013c9c0 con 0x7f2ac009b240 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac518e640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac009b240 0x7f2ac0137790 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f2ab001cc00 tx=0x7f2ab0019100 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 client.?.objecter ms_handle_connect 0x7f2ac009b240 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2ab00b0040 con 0x7f2ac009b240 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:01:00.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: dump: 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab00b9070 con 0x7f2ac009b240 2024-09-11T03:01:00.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 5 monclient: authenticate success, global_id 24812 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient: _renew_subs 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac009b6f0 con 0x7f2ac009b240 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2ab00044b0 con 0x7f2ac009b240 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: dump: 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab77fe640 4 set_mon_vals no callback set 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 client.24812.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient: _renew_subs 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac008ebf0 con 0x7f2ac009b240 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 librados: init done 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b0a0 0x7f2ac013b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac006ca10 con 0x7f2ac013b0a0 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b0a0 0x7f2ac013b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient: get_auth_request con 0x7f2ac013b0a0 auth_method 0 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2ab00b3070 con 0x7f2ac009b240 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 10 monclient(hunting): handle_auth_done global_id 64451 payload 824 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 1 -- 172.21.15.98:0/320516083 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ac00a45c0 con 0x7f2ac013b0a0 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2aa403df40 0x7f2aa4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(434..434 src has 1..434) ==== 6230+0+0 (secure 0 0 0) 0x7f2ab00e8fe0 con 0x7f2ac009b240 2024-09-11T03:01:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 client.24812.objecter ms_dispatch 0x7f2ac0000ba0 osd_map(434..434 src has 1..434) 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 3 client.24812.objecter handle_osd_map got epochs [434,434] > 0 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 3 client.24812.objecter handle_osd_map decoding full epoch 434 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 20 client.24812.objecter dump_active .. 0 homeless 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 client.24812.objecter ms_handle_connect 0x7f2ac013b0a0 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 client.24812.objecter resend_mon_ops 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac498d640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b0a0 0x7f2ac013b480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006f4e0 tx=0x7f2abc011920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2abc015ae0 con 0x7f2ac013b0a0 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:00.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: dump: 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2aa403df40 0x7f2aa4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ac598f640 10 monclient: get_auth_request con 0x7f2aa403df40 auth_method 0 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2abc015670 con 0x7f2ac013b0a0 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:00.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b0a0 msgr2=0x7f2ac013b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b0a0 0x7f2ac013b480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006f4e0 tx=0x7f2abc011920 comp rx=0 tx=0).stop 2024-09-11T03:01:00.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b0a0 msgr2=0x7f2ac013b480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:00.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.656+0000 7f2ab5ffb640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b0a0 0x7f2ac013b480 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006f4e0 tx=0x7f2abc011920 comp rx=0 tx=0).stop 2024-09-11T03:01:00.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.660+0000 7f2ac598f640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2aa403df40 0x7f2aa4040400 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7f2ac013c8e0 tx=0x7f2ab8009820 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.660+0000 7f2ab5ffb640 10 client.24812.objecter ms_handle_connect 0x7f2aa403df40 2024-09-11T03:01:00.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:00.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b9c0 0x7f2ac00992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac006e870 con 0x7f2ac013b9c0 2024-09-11T03:01:00.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b9c0 0x7f2ac00992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient: get_auth_request con 0x7f2ac013b9c0 auth_method 0 2024-09-11T03:01:00.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 10 monclient(hunting): handle_auth_done global_id 64460 payload 824 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 1 -- 172.21.15.98:0/320516083 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ac00a0570 con 0x7f2ac013b9c0 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac498d640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b9c0 0x7f2ac00992a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006f4e0 tx=0x7f2abc0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 10 client.24812.objecter ms_handle_connect 0x7f2ac013b9c0 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 10 client.24812.objecter resend_mon_ops 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2abc004490 con 0x7f2ac013b9c0 2024-09-11T03:01:00.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 10 monclient: dump: 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.692 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f2abc006100 con 0x7f2ac013b9c0 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:00.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b9c0 msgr2=0x7f2ac00992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b9c0 0x7f2ac00992a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006f4e0 tx=0x7f2abc0046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b9c0 msgr2=0x7f2ac00992a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ab5ffb640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2ac013b9c0 0x7f2ac00992a0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006f4e0 tx=0x7f2abc0046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:00.694 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 152858, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:00.695 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.696 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.697 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.698 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:00.699 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.701 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:00.703 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:00.707 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.720 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.721 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.725 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:00.725 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 10 librados: watch_flush enter 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 10 librados: watch_flush exit 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 20 client.24812.objecter shutdown clearing up homeless session... 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 10 client.24812.objecter successfully canceled tick 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2aa403df40 msgr2=0x7f2aa4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2aa403df40 0x7f2aa4040400 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7f2ac013c8e0 tx=0x7f2ab8009820 comp rx=0 tx=0).stop 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 10 monclient: shutdown 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac009b240 msgr2=0x7f2ac0137790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 --2- 172.21.15.98:0/320516083 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2ac009b240 0x7f2ac0137790 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f2ab001cc00 tx=0x7f2ab0019100 comp rx=0 tx=0).stop 2024-09-11T03:01:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac598f640 1 -- 172.21.15.98:0/320516083 reap_dead start 2024-09-11T03:01:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac598f640 1 -- 172.21.15.98:0/320516083 reap_dead start 2024-09-11T03:01:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 shutdown_connections 2024-09-11T03:01:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 >> 172.21.15.98:0/320516083 conn(0x7f2ac00a4ed0 msgr2=0x7f2ac00a61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 shutdown_connections 2024-09-11T03:01:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 -- 172.21.15.98:0/320516083 wait complete. 2024-09-11T03:01:00.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.664+0000 7f2ac73f2640 1 librados: shutdown 2024-09-11T03:01:00.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:00.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T03:01:00.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:01:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient: get_monmap_and_config 2024-09-11T03:01:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient: monmap: 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:00.855258+0000 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:00.855258+0000 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.855 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.855 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.855 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.855 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:00.855 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:00.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 1 Processor -- start 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 1 -- start start 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient: init 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient: _add_conns ranks=[1,2,6,9,5,8,4,0,3,7,10] 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc5000a3690 addr v1:172.21.15.190:6789/0 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5000a1eb0 0x7fc50013c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient(hunting): picked mon.noname-b con 0x7fc5000a1eb0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc5000a4180 0x7fc50013ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient(hunting): picked mon.noname-d con 0x7fc5000a4180 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5000a1ca0 con 0x7fc5000a3690 2024-09-11T03:01:00.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc50009ec40 con 0x7fc5000a1eb0 2024-09-11T03:01:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc50009dd20 con 0x7fc5000a4180 2024-09-11T03:01:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.852+0000 7fc507854640 10 monclient(hunting): authenticate will time out at 1614.735924s 2024-09-11T03:01:00.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5000a1eb0 0x7fc50013c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5000a1eb0 0x7fc50013c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58190/0 (socket says 172.21.15.98:58190) 2024-09-11T03:01:00.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 1 -- 172.21.15.98:0/1489497957 learned_addr learned my addr 172.21.15.98:0/1489497957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 10 monclient(hunting): get_auth_request con 0x7fc5000a1eb0 auth_method 0 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 1 --2- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc5000a4180 0x7fc50013ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): get_auth_request con 0x7fc5000a4180 auth_method 0 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc504def640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient(hunting): handle_auth_done global_id 44610 payload 274 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 1 -- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5000a1eb0 msgr2=0x7fc50013c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 1 --2- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5000a1eb0 0x7fc50013c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 1 -- 172.21.15.98:0/1489497957 >> v1:172.21.15.190:6789/0 conn(0x7fc5000a3690 legacy=0x7fc5000a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 1 monclient: found mon.noname-d 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:01:00.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.856+0000 7fc505df1640 1 -- 172.21.15.98:0/1489497957 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50009d480 con 0x7fc5000a4180 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc505df1640 1 --2- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc5000a4180 0x7fc50013ee00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009dc0 tx=0x7fc4f801c8e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=99a876031f1850ac server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 1 -- 172.21.15.98:0/1489497957 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4f80b2070 con 0x7fc5000a4180 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: dump: 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.862 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 1 -- 172.21.15.98:0/1489497957 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f80a5ae0 con 0x7fc5000a4180 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 1 -- 172.21.15.98:0/1489497957 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4f80a5d10 con 0x7fc5000a4180 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 5 monclient: authenticate success, global_id 44610 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 4 set_mon_vals no callback set 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc4f3fff640 10 monclient: dump: 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 10 monclient: shutdown 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 -- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc5000a4180 msgr2=0x7fc50013ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 --2- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc5000a4180 0x7fc50013ee00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009dc0 tx=0x7fc4f801c8e0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 -- 172.21.15.98:0/1489497957 shutdown_connections 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 --2- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc5000a4180 0x7fc50013ee00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 --2- 172.21.15.98:0/1489497957 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc5000a1eb0 0x7fc50013c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 -- 172.21.15.98:0/1489497957 >> 172.21.15.98:0/1489497957 conn(0x7fc5000a4780 msgr2=0x7fc5000a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 -- 172.21.15.98:0/1489497957 shutdown_connections 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.860+0000 7fc507854640 1 -- 172.21.15.98:0/1489497957 wait complete. 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: build_initial_monmap 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: monmap: 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:00.871 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:00.872 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 librados: starting msgr at 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 librados: starting objecter 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 Processor -- start 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 -- start start 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 librados: setting wanted keys 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 librados: calling monclient init 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: init 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: _add_conns ranks=[7,1,8,6,3,2,4,0,5] 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5000a1eb0 0x7fc5001268e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): picked mon.noname-i con 0x7fc5000a1eb0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:00.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 0x7fc50013d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): picked mon.noname-f con 0x7fc5000a3690 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5000a4180 0x7fc5001245e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): picked mon.noname-e con 0x7fc5000a4180 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc50013fee0 con 0x7fc5000a3690 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc50013fd60 con 0x7fc5000a1eb0 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc500140060 con 0x7fc5000a4180 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient(hunting): authenticate will time out at 1614.745793s 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5000a4180 0x7fc5001245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5000a4180 0x7fc5001245e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35182/0 (socket says 172.21.15.98:35182) 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 -- 172.21.15.98:0/1948585538 learned_addr learned my addr 172.21.15.98:0/1948585538 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): get_auth_request con 0x7fc5000a4180 auth_method 0 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc504def640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 0x7fc50013d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc5055f0640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5000a1eb0 0x7fc5001268e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc5055f0640 10 monclient(hunting): get_auth_request con 0x7fc5000a1eb0 auth_method 0 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc5055f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc504def640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 msgr2=0x7fc50013d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc5055f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc5055f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc504def640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 msgr2=0x7fc50013d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc504def640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 0x7fc50013d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc504def640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 0x7fc50013d750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient(hunting): handle_auth_done global_id 24821 payload 824 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5000a1eb0 msgr2=0x7fc5001268e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc5000a1eb0 0x7fc5001268e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 msgr2=0x7fc50013d750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5000a3690 0x7fc50013d750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient: _finish_hunting 0 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 monclient: found mon.noname-e 2024-09-11T03:01:00.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 -- 172.21.15.98:0/1948585538 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5001cdd70 con 0x7fc5000a4180 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc505df1640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5000a4180 0x7fc5001245e0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80b0040 tx=0x7fc4f8019680 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 client.?.objecter ms_handle_connect 0x7fc5000a4180 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4f80b2050 con 0x7fc5000a4180 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: dump: 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.877 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: _finish_auth 0 2024-09-11T03:01:00.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 5 monclient: authenticate success, global_id 24821 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f80be070 con 0x7fc5000a4180 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: _renew_subs 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5000a30b0 con 0x7fc5000a4180 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4f8003e10 con 0x7fc5000a4180 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f37fe640 4 set_mon_vals no callback set 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc4f1ffb640 10 monclient: dump: 2024-09-11T03:01:00.879 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.880 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 client.24821.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: _renew_subs 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fc500127270 con 0x7fc5000a4180 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.864+0000 7fc507854640 1 librados: init done 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc507854640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc507854640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002d450 0x7fc50002d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc507854640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc50006ca10 con 0x7fc50002d450 2024-09-11T03:01:00.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002d450 0x7fc50002d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient: get_auth_request con 0x7fc50002d450 auth_method 0 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc4f80039a0 con 0x7fc5000a4180 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 10 monclient(hunting): handle_auth_done global_id 24295 payload 824 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 1 -- 172.21.15.98:0/1948585538 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc500068b40 con 0x7fc50002d450 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4e403df40 0x7fc4e4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(435..435 src has 1..435) ==== 5817+0+0 (secure 0 0 0) 0x7fc5000a1200 con 0x7fc5000a4180 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 client.24821.objecter ms_dispatch 0x7fc500000ba0 osd_map(435..435 src has 1..435) 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 3 client.24821.objecter handle_osd_map got epochs [435,435] > 0 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 3 client.24821.objecter handle_osd_map decoding full epoch 435 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc5055f0640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002d450 0x7fc50002d830 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400e500 tx=0x7fc4f400e9c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 20 client.24821.objecter dump_active .. 0 homeless 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 client.24821.objecter ms_handle_connect 0x7fc50002d450 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 client.24821.objecter resend_mon_ops 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc504def640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4e403df40 0x7fc4e4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc504def640 10 monclient: get_auth_request con 0x7fc4e403df40 auth_method 0 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4f40052e0 con 0x7fc50002d450 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 monclient: dump: 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.884 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc4f4018070 con 0x7fc50002d450 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002d450 msgr2=0x7fc50002d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002d450 0x7fc50002d830 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400e500 tx=0x7fc4f400e9c0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002d450 msgr2=0x7fc50002d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002d450 0x7fc50002d830 secure :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fc4f400e500 tx=0x7fc4f400e9c0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc4f1ffb640 10 client.24821.objecter ms_handle_connect 0x7fc4e403df40 2024-09-11T03:01:00.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.868+0000 7fc504def640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4e403df40 0x7fc4e4040400 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7fc500125cc0 tx=0x7fc4fc011040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002dd70 0x7fc50002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:00.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc50006e870 con 0x7fc50002dd70 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002dd70 0x7fc50002e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient: get_auth_request con 0x7fc50002dd70 auth_method 0 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 10 monclient(hunting): handle_auth_done global_id 24304 payload 824 2024-09-11T03:01:00.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 1 -- 172.21.15.98:0/1948585538 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5000a3a70 con 0x7fc50002dd70 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc5055f0640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002dd70 0x7fc50002e180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4000fc0 tx=0x7fc4f4000ee0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 10 client.24821.objecter ms_handle_connect 0x7fc50002dd70 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 10 client.24821.objecter resend_mon_ops 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4f4003c80 con 0x7fc50002dd70 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 10 monclient: dump: 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:00.889 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fc4f4003810 con 0x7fc50002dd70 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:00.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002dd70 msgr2=0x7fc50002e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002dd70 0x7fc50002e180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4000fc0 tx=0x7fc4f4000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002dd70 msgr2=0x7fc50002e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc4f1ffb640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc50002dd70 0x7fc50002e180 secure :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4000fc0 tx=0x7fc4f4000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:00.891 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 676661, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.892 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.893 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:00.895 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:00.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:00.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:00.902 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.905 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.906 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:00.909 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:00.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:00.913 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.914 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:00.915 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 10 librados: watch_flush enter 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 10 librados: watch_flush exit 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 20 client.24821.objecter shutdown clearing up homeless session... 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 10 client.24821.objecter successfully canceled tick 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4e403df40 msgr2=0x7fc4e4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc4e403df40 0x7fc4e4040400 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7fc500125cc0 tx=0x7fc4fc011040 comp rx=0 tx=0).stop 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 10 monclient: shutdown 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:00.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5000a4180 msgr2=0x7fc5001245e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 --2- 172.21.15.98:0/1948585538 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc5000a4180 0x7fc5001245e0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fc4f80b0040 tx=0x7fc4f8019680 comp rx=0 tx=0).stop 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc505df1640 1 -- 172.21.15.98:0/1948585538 reap_dead start 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc505df1640 1 -- 172.21.15.98:0/1948585538 reap_dead start 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 shutdown_connections 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 >> 172.21.15.98:0/1948585538 conn(0x7fc5000a4780 msgr2=0x7fc500072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 shutdown_connections 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 -- 172.21.15.98:0/1948585538 wait complete. 2024-09-11T03:01:00.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:00.876+0000 7fc507854640 1 librados: shutdown 2024-09-11T03:01:00.918 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:00.919 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T03:01:00.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:01:01.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient: get_monmap_and_config 2024-09-11T03:01:01.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient: monmap: 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:01.038561+0000 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:01.038561+0000 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:01.037 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:01.038 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 Processor -- start 2024-09-11T03:01:01.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 -- start start 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient: init 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient: _add_conns ranks=[4,6,7,8,0,9,3,2,5,10,1] 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5440a7a10 0x7fb5440724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb5440a7a10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a7df0 0x7fb5440a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): picked mon.noname-d con 0x7fb5440a7df0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 0x7fb544139c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): picked mon.noname-i con 0x7fb5441397c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:01.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb544073d30 con 0x7fb5440a7a10 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb544071ab0 con 0x7fb5440a7df0 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb544071210 con 0x7fb5441397c0 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb54a5a8640 10 monclient(hunting): authenticate will time out at 1614.918771s 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5440a7a10 0x7fb5440724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5440a7a10 0x7fb5440724c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34644/0 (socket says 172.21.15.98:34644) 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 -- 172.21.15.98:0/2170713612 learned_addr learned my addr 172.21.15.98:0/2170713612 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): get_auth_request con 0x7fb5440a7a10 auth_method 0 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a7df0 0x7fb5440a81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 10 monclient(hunting): get_auth_request con 0x7fb5440a7df0 auth_method 0 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb548b45640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 0x7fb544139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb548b45640 10 monclient(hunting): get_auth_request con 0x7fb5441397c0 auth_method 0 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb548b45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb548b45640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb548b45640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient(hunting): handle_auth_done global_id 24313 payload 274 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 -- 172.21.15.98:0/2170713612 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 msgr2=0x7fb544139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 0x7fb544139c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 -- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a7df0 msgr2=0x7fb5440a81d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a7df0 0x7fb5440a81d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 monclient: found mon.noname-c 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:01:01.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.036+0000 7fb543fff640 1 -- 172.21.15.98:0/2170713612 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5440a98e0 con 0x7fb5440a7a10 2024-09-11T03:01:01.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb543fff640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5440a7a10 0x7fb5440724c0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb5380098a0 tx=0x7fb53801cea0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f961a0670b50e4b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 1 -- 172.21.15.98:0/2170713612 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5380b2070 con 0x7fb5440a7a10 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: dump: 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.055 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 5 monclient: authenticate success, global_id 24313 2024-09-11T03:01:01.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 1 -- 172.21.15.98:0/2170713612 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5380ab7f0 con 0x7fb5440a7a10 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 4 set_mon_vals no callback set 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 1 -- 172.21.15.98:0/2170713612 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5380abe50 con 0x7fb5440a7a10 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb542ffd640 10 monclient: dump: 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.057 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 10 monclient: shutdown 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 -- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5440a7a10 msgr2=0x7fb5440724c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5440a7a10 0x7fb5440724c0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb5380098a0 tx=0x7fb53801cea0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 -- 172.21.15.98:0/2170713612 shutdown_connections 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 0x7fb544139c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a7df0 0x7fb5440a81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/2170713612 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb5440a7a10 0x7fb5440724c0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 -- 172.21.15.98:0/2170713612 >> 172.21.15.98:0/2170713612 conn(0x7fb5440a8b00 msgr2=0x7fb5440a8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 -- 172.21.15.98:0/2170713612 shutdown_connections 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 -- 172.21.15.98:0/2170713612 wait complete. 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 10 monclient: monmap: 2024-09-11T03:01:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.060 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 librados: starting msgr at 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.040+0000 7fb54a5a8640 1 librados: starting objecter 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 Processor -- start 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 -- start start 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 librados: setting wanted keys 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 librados: calling monclient init 2024-09-11T03:01:01.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient: init 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient: _add_conns ranks=[2,0,7,3,8,4,5,6,1] 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a7a10 0x7fb5441365b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb5440a7a10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5440a7df0 0x7fb544136af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb5440a7df0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 0x7fb54413bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): picked mon.noname-i con 0x7fb5441397c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb5440a8890 con 0x7fb5440a7df0 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb5440a8710 con 0x7fb5440a7a10 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb5441434b0 con 0x7fb5441397c0 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): authenticate will time out at 1614.924694s 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a7a10 0x7fb5441365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5440a7df0 0x7fb544136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a7a10 0x7fb5441365b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58196/0 (socket says 172.21.15.98:58196) 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 -- 172.21.15.98:0/4030280924 learned_addr learned my addr 172.21.15.98:0/4030280924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): get_auth_request con 0x7fb5440a7a10 auth_method 0 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): get_auth_request con 0x7fb5440a7df0 auth_method 0 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient(hunting): handle_auth_done global_id 64478 payload 824 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb548b45640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 0x7fb54413bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 msgr2=0x7fb54413bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb5441397c0 0x7fb54413bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5440a7df0 msgr2=0x7fb544136af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5440a7df0 0x7fb544136af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 monclient: found mon.noname-b 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 -- 172.21.15.98:0/4030280924 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54413f700 con 0x7fb5440a7a10 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5440a7df0 0x7fb544136af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb543fff640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a7a10 0x7fb5441365b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb5380b0040 tx=0x7fb5380191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 client.?.objecter ms_handle_connect 0x7fb5440a7a10 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5380bd070 con 0x7fb5440a7a10 2024-09-11T03:01:01.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: dump: 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.066 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 5 monclient: authenticate success, global_id 64478 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5380abe10 con 0x7fb5440a7a10 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient: _renew_subs 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54413d680 con 0x7fb5440a7a10 2024-09-11T03:01:01.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5380b2070 con 0x7fb5440a7a10 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 client.64478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5427fc640 4 set_mon_vals no callback set 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: dump: 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.068 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient: _renew_subs 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:01.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54413e1e0 con 0x7fb5440a7a10 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 librados: init done 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413f940 0x7fb54413c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb54413f310 con 0x7fb54413f940 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413f940 0x7fb54413c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient: get_auth_request con 0x7fb54413f940 auth_method 0 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb538004250 con 0x7fb5440a7a10 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb53003df40 0x7fb530040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 10 monclient(hunting): handle_auth_done global_id 44628 payload 824 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 1 -- 172.21.15.98:0/4030280924 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5440a81d0 con 0x7fb54413f940 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(435..435 src has 1..435) ==== 5817+0+0 (secure 0 0 0) 0x7fb5380e8e70 con 0x7fb5440a7a10 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 client.64478.objecter ms_dispatch 0x7fb544000ba0 osd_map(435..435 src has 1..435) 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 3 client.64478.objecter handle_osd_map got epochs [435,435] > 0 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 3 client.64478.objecter handle_osd_map decoding full epoch 435 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 20 client.64478.objecter dump_active .. 0 homeless 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 client.64478.objecter ms_handle_connect 0x7fb54413f940 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 client.64478.objecter resend_mon_ops 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb5437fe640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413f940 0x7fb54413c0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb534009fd0 tx=0x7fb53400f610 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb548b45640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb53003df40 0x7fb530040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5340148b0 con 0x7fb54413f940 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb548b45640 10 monclient: get_auth_request con 0x7fb53003df40 auth_method 0 2024-09-11T03:01:01.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: dump: 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.072 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb534014440 con 0x7fb54413f940 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413f940 msgr2=0x7fb54413c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413f940 0x7fb54413c0d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb534009fd0 tx=0x7fb53400f610 comp rx=0 tx=0).stop 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413f940 msgr2=0x7fb54413c0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413f940 0x7fb54413c0d0 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb534009fd0 tx=0x7fb53400f610 comp rx=0 tx=0).stop 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb540ff9640 10 client.64478.objecter ms_handle_connect 0x7fb53003df40 2024-09-11T03:01:01.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.044+0000 7fb548b45640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb53003df40 0x7fb530040400 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7fb5440a66c0 tx=0x7fb53c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413c610 0x7fb54413c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb54406ca10 con 0x7fb54413c610 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413c610 0x7fb54413c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient: get_auth_request con 0x7fb54413c610 auth_method 0 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 10 monclient(hunting): handle_auth_done global_id 44637 payload 824 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 1 -- 172.21.15.98:0/4030280924 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb54406f9e0 con 0x7fb54413c610 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 10 client.64478.objecter ms_handle_connect 0x7fb54413c610 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 10 client.64478.objecter resend_mon_ops 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb5437fe640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413c610 0x7fb54413c9f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb534000c10 tx=0x7fb5340046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb534014a60 con 0x7fb54413c610 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 10 monclient: dump: 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.075 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fb53401d400 con 0x7fb54413c610 2024-09-11T03:01:01.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413c610 msgr2=0x7fb54413c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413c610 0x7fb54413c9f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb534000c10 tx=0x7fb5340046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413c610 msgr2=0x7fb54413c9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb540ff9640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb54413c610 0x7fb54413c9f0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb534000c10 tx=0x7fb5340046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:01.077 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 287480, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.078 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.079 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.082 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.088 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:01.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:01.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:01.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.095 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:01.096 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.097 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.098 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.099 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.100 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:01.101 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 10 librados: watch_flush enter 2024-09-11T03:01:01.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 10 librados: watch_flush exit 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 20 client.64478.objecter shutdown clearing up homeless session... 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 10 client.64478.objecter successfully canceled tick 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb53003df40 msgr2=0x7fb530040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb53003df40 0x7fb530040400 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7fb5440a66c0 tx=0x7fb53c00b040 comp rx=0 tx=0).stop 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 10 monclient: shutdown 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a7a10 msgr2=0x7fb5441365b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 --2- 172.21.15.98:0/4030280924 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a7a10 0x7fb5441365b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb5380b0040 tx=0x7fb5380191e0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb548b45640 1 -- 172.21.15.98:0/4030280924 reap_dead start 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb548b45640 1 -- 172.21.15.98:0/4030280924 reap_dead start 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 shutdown_connections 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 >> 172.21.15.98:0/4030280924 conn(0x7fb5440a8b00 msgr2=0x7fb5440a9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 shutdown_connections 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 -- 172.21.15.98:0/4030280924 wait complete. 2024-09-11T03:01:01.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.052+0000 7fb54a5a8640 1 librados: shutdown 2024-09-11T03:01:01.104 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:01.104 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T03:01:01.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient: get_monmap_and_config 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient: monmap: 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:01.217958+0000 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:01.217958+0000 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:01.217 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 1 Processor -- start 2024-09-11T03:01:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 1 -- start start 2024-09-11T03:01:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient: init 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient: _add_conns ranks=[1,9,4,10,6,3,5,2,8,7,0] 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f71dc1397d0 addr v1:172.21.15.190:6789/0 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71dc09d4c0 0x7f71dc13e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient(hunting): picked mon.noname-a con 0x7f71dc09d4c0 addr v2:172.21.15.98:3300/0 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc13ec90 0x7f71dc141080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc13ec90 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71dc09de80 con 0x7f71dc1397d0 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f71dc09cbe0 con 0x7f71dc09d4c0 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f71dc09e160 con 0x7f71dc13ec90 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71e0f01640 10 monclient(hunting): authenticate will time out at 1615.097934s 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71dc09d4c0 0x7f71dc13e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f71dc09d4c0 0x7f71dc13e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:33276/0 (socket says 172.21.15.98:33276) 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc13ec90 0x7f71dc141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1398761122 learned_addr learned my addr 172.21.15.98:0/1398761122 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request con 0x7f71dc09d4c0 auth_method 0 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 10 monclient(hunting): get_auth_request con 0x7f71dc13ec90 auth_method 0 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_done global_id 44793 payload 274 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc13ec90 msgr2=0x7f71dc141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 --2- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc13ec90 0x7f71dc141080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1398761122 >> v1:172.21.15.190:6789/0 conn(0x7f71dc1397d0 legacy=0x7f71dc139bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 monclient: found mon.noname-a 2024-09-11T03:01:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1398761122 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc09e440 con 0x7f71dc09d4c0 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.216+0000 7f71dad9d640 1 --2- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc13ec90 0x7f71dc141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 --2- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71dc09d4c0 0x7f71dc13e750 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009e80 tx=0x7f71cc01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307e77d5bbe19941 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 1 -- 172.21.15.98:0/1398761122 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71cc0090d0 con 0x7f71dc09d4c0 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: dump: 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 1 -- 172.21.15.98:0/1398761122 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71cc0a5bf0 con 0x7f71dc09d4c0 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 5 monclient: authenticate success, global_id 44793 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 1 -- 172.21.15.98:0/1398761122 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71cc0a5da0 con 0x7f71dc09d4c0 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 4 set_mon_vals no callback set 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d959a640 10 monclient: dump: 2024-09-11T03:01:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: shutdown 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71dc09d4c0 msgr2=0x7f71dc13e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 --2- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71dc09d4c0 0x7f71dc13e750 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f71cc009e80 tx=0x7f71cc01d300 comp rx=0 tx=0).stop 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- 172.21.15.98:0/1398761122 shutdown_connections 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 --2- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc13ec90 0x7f71dc141080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 --2- 172.21.15.98:0/1398761122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f71dc09d4c0 0x7f71dc13e750 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- 172.21.15.98:0/1398761122 >> 172.21.15.98:0/1398761122 conn(0x7f71dc0a4780 msgr2=0x7f71dc0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- 172.21.15.98:0/1398761122 shutdown_connections 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- 172.21.15.98:0/1398761122 wait complete. 2024-09-11T03:01:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: monmap: 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 librados: starting msgr at 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 librados: starting objecter 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 Processor -- start 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- start start 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 librados: setting wanted keys 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 librados: calling monclient init 2024-09-11T03:01:01.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: init 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: _add_conns ranks=[7,4,3,2,6,5,8,1,0] 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f71dc09d4c0 0x7f71dc121620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): picked mon.noname-i con 0x7f71dc09d4c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc1397d0 0x7f71dc138490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): picked mon.noname-c con 0x7f71dc1397d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71dc13ec90 0x7f71dc11f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): picked mon.noname-g con 0x7f71dc13ec90 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f71dc143f60 con 0x7f71dc13ec90 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f71dc143de0 con 0x7f71dc1397d0 2024-09-11T03:01:01.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f71dc1440e0 con 0x7f71dc09d4c0 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient(hunting): authenticate will time out at 1615.102862s 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc1397d0 0x7f71dc138490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc1397d0 0x7f71dc138490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34658/0 (socket says 172.21.15.98:34658) 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1994188426 learned_addr learned my addr 172.21.15.98:0/1994188426 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request con 0x7f71dc1397d0 auth_method 0 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71dad9d640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71dc13ec90 0x7f71dc11f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71da59c640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f71dc09d4c0 0x7f71dc121620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71dad9d640 10 monclient(hunting): get_auth_request con 0x7f71dc13ec90 auth_method 0 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_done global_id 24331 payload 824 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f71dc09d4c0 msgr2=0x7f71dc121620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f71dc09d4c0 0x7f71dc121620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71dc13ec90 msgr2=0x7f71dc11f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f71dc13ec90 0x7f71dc11f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 monclient: found mon.noname-c 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 -- 172.21.15.98:0/1994188426 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc1d1e90 con 0x7f71dc1397d0 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71da59c640 10 monclient: get_auth_request con 0x7f71dc09d4c0 auth_method 0 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71da59c640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f71dc09d4c0 0x7f71dc121620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d9d9b640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc1397d0 0x7f71dc138490 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f71cc01a4d0 tx=0x7f71cc019640 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 client.?.objecter ms_handle_connect 0x7f71dc1397d0 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71cc0bc020 con 0x7f71dc1397d0 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: dump: 2024-09-11T03:01:01.240 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.241 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71cc003e10 con 0x7f71dc1397d0 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 5 monclient: authenticate success, global_id 24331 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: _renew_subs 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71dc122e20 con 0x7f71dc1397d0 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71cc0044b0 con 0x7f71dc1397d0 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71caffd640 10 monclient: dump: 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.243 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71d8d99640 4 set_mon_vals no callback set 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 client.24331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: _renew_subs 2024-09-11T03:01:01.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f71dc0a4140 con 0x7f71dc1397d0 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.220+0000 7f71e0f01640 1 librados: init done 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71e0f01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71e0f01640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc1389d0 0x7f71dc02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71e0f01640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f71dc06ca10 con 0x7f71dc1389d0 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc1389d0 0x7f71dc02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f71cc004650 con 0x7f71dc1397d0 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient: get_auth_request con 0x7f71dc1389d0 auth_method 0 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71bc03df60 0x7f71bc040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 10 monclient(hunting): handle_auth_done global_id 24830 payload 824 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 1 -- 172.21.15.98:0/1994188426 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71dc068b40 con 0x7f71dc1389d0 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(435..435 src has 1..435) ==== 5817+0+0 (secure 0 0 0) 0x7f71cc0ea890 con 0x7f71dc1397d0 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 client.24331.objecter ms_dispatch 0x7f71dc000ba0 osd_map(435..435 src has 1..435) 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 3 client.24331.objecter handle_osd_map got epochs [435,435] > 0 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 3 client.24331.objecter handle_osd_map decoding full epoch 435 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 20 client.24331.objecter dump_active .. 0 homeless 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 client.24331.objecter ms_handle_connect 0x7f71dc1389d0 2024-09-11T03:01:01.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 client.24331.objecter resend_mon_ops 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71da59c640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc1389d0 0x7f71dc02d450 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f71d0001ec0 tx=0x7f71d000d950 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71dad9d640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71bc03df60 0x7f71bc040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71dad9d640 10 monclient: get_auth_request con 0x7f71bc03df60 auth_method 0 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71d0006c50 con 0x7f71dc1389d0 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 monclient: dump: 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.247 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71d0005280 con 0x7f71dc1389d0 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:01.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc1389d0 msgr2=0x7f71dc02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc1389d0 0x7f71dc02d450 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f71d0001ec0 tx=0x7f71d000d950 comp rx=0 tx=0).stop 2024-09-11T03:01:01.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc1389d0 msgr2=0x7f71dc02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc1389d0 0x7f71dc02d450 secure :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f71d0001ec0 tx=0x7f71d000d950 comp rx=0 tx=0).stop 2024-09-11T03:01:01.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71caffd640 10 client.24331.objecter ms_handle_connect 0x7f71bc03df60 2024-09-11T03:01:01.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.224+0000 7f71dad9d640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71bc03df60 0x7f71bc040420 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0a4750 tx=0x7f71d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.228+0000 7f71e0f01640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.228+0000 7f71e0f01640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc02d990 0x7f71dc02dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.228+0000 7f71e0f01640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.228+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f71dc06e870 con 0x7f71dc02d990 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.228+0000 7f71da59c640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc02d990 0x7f71dc02dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient: get_auth_request con 0x7f71dc02d990 auth_method 0 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 10 monclient(hunting): handle_auth_done global_id 24839 payload 824 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 1 -- 172.21.15.98:0/1994188426 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71dc139ee0 con 0x7f71dc02d990 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71da59c640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc02d990 0x7f71dc02dda0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f71d0000fc0 tx=0x7f71d0000ee0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 10 client.24331.objecter ms_handle_connect 0x7f71dc02d990 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 10 client.24331.objecter resend_mon_ops 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71d0007590 con 0x7f71dc02d990 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 10 monclient: dump: 2024-09-11T03:01:01.252 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.253 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9269 (secure 0 0 0) 0x7f71d0007590 con 0x7f71dc02d990 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc02d990 msgr2=0x7f71dc02dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc02d990 0x7f71dc02dda0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f71d0000fc0 tx=0x7f71d0000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc02d990 msgr2=0x7f71dc02dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71caffd640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f71dc02d990 0x7f71dc02dda0 secure :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f71d0000fc0 tx=0x7f71d0000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.254 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 677025, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:01.259 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.260 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.261 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.266 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:01.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:01.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:01.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.275 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.278 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:01:01.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.279 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:01.279 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:01.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 10 librados: watch_flush enter 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 10 librados: watch_flush exit 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 20 client.24331.objecter shutdown clearing up homeless session... 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 10 client.24331.objecter successfully canceled tick 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71bc03df60 msgr2=0x7f71bc040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f71bc03df60 0x7f71bc040420 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f71dc0a4750 tx=0x7f71d4008040 comp rx=0 tx=0).stop 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 10 monclient: shutdown 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc1397d0 msgr2=0x7f71dc138490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 --2- 172.21.15.98:0/1994188426 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f71dc1397d0 0x7f71dc138490 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f71cc01a4d0 tx=0x7f71cc019640 comp rx=0 tx=0).stop 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71dad9d640 1 -- 172.21.15.98:0/1994188426 reap_dead start 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71dad9d640 1 -- 172.21.15.98:0/1994188426 reap_dead start 2024-09-11T03:01:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 shutdown_connections 2024-09-11T03:01:01.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 >> 172.21.15.98:0/1994188426 conn(0x7f71dc0a4780 msgr2=0x7f71dc13f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 shutdown_connections 2024-09-11T03:01:01.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 -- 172.21.15.98:0/1994188426 wait complete. 2024-09-11T03:01:01.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.232+0000 7f71e0f01640 1 librados: shutdown 2024-09-11T03:01:01.282 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:01.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T03:01:01.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.396+0000 7fdb7fdd4640 10 monclient: get_monmap_and_config 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.396+0000 7fdb7fdd4640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.396+0000 7fdb7fdd4640 10 monclient: monmap: 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:01.401079+0000 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:01.401079+0000 2024-09-11T03:01:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:01.400 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 1 Processor -- start 2024-09-11T03:01:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 1 -- start start 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient: init 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient: _add_conns ranks=[10,1,5,9,7,2,8,0,6,4,3] 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdb78139b10 0x7fdb78072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient(hunting): picked mon.noname-f con 0x7fdb78139b10 addr v2:172.21.15.190:3300/0 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdb78139ef0 addr v1:172.21.15.190:6789/0 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdb7813a2d0 0x7fdb780a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient(hunting): picked mon.noname-h con 0x7fdb7813a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb780702b0 con 0x7fdb78139ef0 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fdb78071bf0 con 0x7fdb78139b10 2024-09-11T03:01:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fdb78071350 con 0x7fdb7813a2d0 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7fdd4640 10 monclient(hunting): authenticate will time out at 1615.281378s 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdb78139b10 0x7fdb78072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdb7813a2d0 0x7fdb780a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fdb78139b10 0x7fdb78072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:44804/0 (socket says 172.21.15.98:44804) 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 -- 172.21.15.98:0/913506135 learned_addr learned my addr 172.21.15.98:0/913506135 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): get_auth_request con 0x7fdb78139b10 auth_method 0 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 10 monclient(hunting): get_auth_request con 0x7fdb7813a2d0 auth_method 0 2024-09-11T03:01:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient(hunting): handle_auth_done global_id 25579 payload 274 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 -- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdb7813a2d0 msgr2=0x7fdb780a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 --2- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdb7813a2d0 0x7fdb780a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 -- 172.21.15.98:0/913506135 >> v1:172.21.15.190:6789/0 conn(0x7fdb78139ef0 legacy=0x7fdb780a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 monclient: found mon.noname-f 2024-09-11T03:01:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:01:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7db70640 1 -- 172.21.15.98:0/913506135 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb780a9620 con 0x7fdb78139b10 2024-09-11T03:01:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.400+0000 7fdb7e371640 1 --2- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdb7813a2d0 0x7fdb780a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7db70640 1 --2- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139b10 0x7fdb78072600 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098a0 tx=0x7fdb7401c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=881399c55d5c9911 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 1 -- 172.21.15.98:0/913506135 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb740b1070 con 0x7fdb78139b10 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: dump: 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.414 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 5 monclient: authenticate success, global_id 25579 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 1 -- 172.21.15.98:0/913506135 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb740ab3a0 con 0x7fdb78139b10 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 4 set_mon_vals no callback set 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 1 -- 172.21.15.98:0/913506135 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb740aba40 con 0x7fdb78139b10 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:01:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7cb6e640 10 monclient: dump: 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.416 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient: shutdown 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139b10 msgr2=0x7fdb78072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139b10 0x7fdb78072600 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098a0 tx=0x7fdb7401c490 comp rx=0 tx=0).stop 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/913506135 shutdown_connections 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fdb7813a2d0 0x7fdb780a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/913506135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139b10 0x7fdb78072600 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/913506135 >> 172.21.15.98:0/913506135 conn(0x7fdb780ac790 msgr2=0x7fdb780a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/913506135 shutdown_connections 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/913506135 wait complete. 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient: monmap: 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 librados: starting msgr at 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 librados: starting objecter 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 Processor -- start 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- start start 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 librados: setting wanted keys 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 librados: calling monclient init 2024-09-11T03:01:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient: init 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient: _add_conns ranks=[3,1,6,7,4,0,2,5,8] 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb78139b10 0x7fdb7813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): picked mon.noname-g con 0x7fdb78139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139ef0 0x7fdb7813ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): picked mon.noname-f con 0x7fdb78139ef0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb7813a2d0 0x7fdb7813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): picked mon.noname-d con 0x7fdb7813a2d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb78144470 con 0x7fdb78139ef0 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fdb780a8870 con 0x7fdb78139b10 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fdb781445f0 con 0x7fdb7813a2d0 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7fdd4640 10 monclient(hunting): authenticate will time out at 1615.287185s 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb7813a2d0 0x7fdb7813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb7813a2d0 0x7fdb7813b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:32794/0 (socket says 172.21.15.98:32794) 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 -- 172.21.15.98:0/4048736850 learned_addr learned my addr 172.21.15.98:0/4048736850 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): get_auth_request con 0x7fdb7813a2d0 auth_method 0 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7d36f640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139ef0 0x7fdb7813ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7d36f640 10 monclient(hunting): get_auth_request con 0x7fdb78139ef0 auth_method 0 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7d36f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7d36f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7d36f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7db70640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb78139b10 0x7fdb7813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient(hunting): handle_auth_done global_id 44646 payload 824 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb78139b10 msgr2=0x7fdb7813a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fdb78139b10 0x7fdb7813a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139ef0 msgr2=0x7fdb7813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139ef0 0x7fdb7813ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 monclient: found mon.noname-d 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 -- 172.21.15.98:0/4048736850 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7813d170 con 0x7fdb7813a2d0 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7d36f640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78139ef0 0x7fdb7813ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb7e371640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb7813a2d0 0x7fdb7813b250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000efc0 tx=0x7fdb7000c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb6e7fc640 10 client.?.objecter ms_handle_connect 0x7fdb7813a2d0 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb6e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb70018030 con 0x7fdb7813a2d0 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.404+0000 7fdb6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: dump: 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.424 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb7000cdf0 con 0x7fdb7813a2d0 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 5 monclient: authenticate success, global_id 44646 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 10 monclient: _renew_subs 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb780738c0 con 0x7fdb7813a2d0 2024-09-11T03:01:01.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb70014d60 con 0x7fdb7813a2d0 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: dump: 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.426 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6ffff640 4 set_mon_vals no callback set 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 10 client.44646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 10 monclient: _renew_subs 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb780a56f0 con 0x7fdb7813a2d0 2024-09-11T03:01:01.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 1 librados: init done 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78133760 0x7fdb78133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7813cb10 con 0x7fdb78133760 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fdb70014480 con 0x7fdb7813a2d0 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78133760 0x7fdb78133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient: get_auth_request con 0x7fdb78133760 auth_method 0 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb5c03df40 0x7fdb5c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(435..435 src has 1..435) ==== 5817+0+0 (secure 0 0 0) 0x7fdb70019070 con 0x7fdb7813a2d0 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 client.44646.objecter ms_dispatch 0x7fdb78000ba0 osd_map(435..435 src has 1..435) 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 3 client.44646.objecter handle_osd_map got epochs [435,435] > 0 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 3 client.44646.objecter handle_osd_map decoding full epoch 435 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 20 client.44646.objecter dump_active .. 0 homeless 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 10 monclient(hunting): handle_auth_done global_id 25606 payload 824 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 1 -- 172.21.15.98:0/4048736850 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb7806e680 con 0x7fdb78133760 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7d36f640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb5c03df40 0x7fdb5c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7d36f640 10 monclient: get_auth_request con 0x7fdb5c03df40 auth_method 0 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 client.44646.objecter ms_handle_connect 0x7fdb78133760 2024-09-11T03:01:01.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 client.44646.objecter resend_mon_ops 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7db70640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78133760 0x7fdb78133b40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fdb780a9a70 tx=0x7fdb74019230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb740b9070 con 0x7fdb78133760 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: dump: 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.430 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 client.44646.objecter ms_handle_connect 0x7fdb5c03df40 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb7d36f640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb5c03df40 0x7fdb5c040400 secure :-1 s=READY pgs=3589 cs=0 l=1 rev1=1 crypto rx=0x7fdb780a4ea0 tx=0x7fdb68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb74003620 con 0x7fdb78133760 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78133760 msgr2=0x7fdb78133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78133760 0x7fdb78133b40 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fdb780a9a70 tx=0x7fdb74019230 comp rx=0 tx=0).stop 2024-09-11T03:01:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78133760 msgr2=0x7fdb78133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.408+0000 7fdb6e7fc640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78133760 0x7fdb78133b40 secure :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fdb780a9a70 tx=0x7fdb74019230 comp rx=0 tx=0).stop 2024-09-11T03:01:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78134080 0x7fdb781373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7806ca10 con 0x7fdb78134080 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78134080 0x7fdb781373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient: get_auth_request con 0x7fdb78134080 auth_method 0 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 10 monclient(hunting): handle_auth_done global_id 25615 payload 824 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 1 -- 172.21.15.98:0/4048736850 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdb78072600 con 0x7fdb78134080 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7db70640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78134080 0x7fdb781373b0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098a0 tx=0x7fdb740a9000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 10 client.44646.objecter ms_handle_connect 0x7fdb78134080 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 10 client.44646.objecter resend_mon_ops 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb740b9040 con 0x7fdb78134080 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 10 monclient: dump: 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.435 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fdb740b5690 con 0x7fdb78134080 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:01.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78134080 msgr2=0x7fdb781373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78134080 0x7fdb781373b0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098a0 tx=0x7fdb740a9000 comp rx=0 tx=0).stop 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78134080 msgr2=0x7fdb781373b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb6e7fc640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb78134080 0x7fdb781373b0 secure :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098a0 tx=0x7fdb740a9000 comp rx=0 tx=0).stop 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 677189, 2024-09-11T03:01:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.446 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.448 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:01.449 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:01.452 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.453 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.455 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.456 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:01.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:01.461 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 10 librados: watch_flush enter 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 10 librados: watch_flush exit 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 20 client.44646.objecter shutdown clearing up homeless session... 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 10 client.44646.objecter successfully canceled tick 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb5c03df40 msgr2=0x7fdb5c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fdb5c03df40 0x7fdb5c040400 secure :-1 s=READY pgs=3589 cs=0 l=1 rev1=1 crypto rx=0x7fdb780a4ea0 tx=0x7fdb68009040 comp rx=0 tx=0).stop 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 10 monclient: shutdown 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb7813a2d0 msgr2=0x7fdb7813b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb7813a2d0 0x7fdb7813b250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000efc0 tx=0x7fdb7000c490 comp rx=0 tx=0).stop 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7e371640 1 -- 172.21.15.98:0/4048736850 reap_dead start 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 shutdown_connections 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 --2- 172.21.15.98:0/4048736850 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fdb7813a2d0 0x7fdb7813b250 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 >> 172.21.15.98:0/4048736850 conn(0x7fdb780ac790 msgr2=0x7fdb78134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 shutdown_connections 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 -- 172.21.15.98:0/4048736850 wait complete. 2024-09-11T03:01:01.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.416+0000 7fdb7fdd4640 1 librados: shutdown 2024-09-11T03:01:01.464 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:01.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T03:01:01.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:01:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient: get_monmap_and_config 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient: monmap: 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:01.581667+0000 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:01.581667+0000 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:01.581 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 Processor -- start 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 -- start start 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient: init 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient: _add_conns ranks=[7,10,4,5,6,1,9,3,8,2,0] 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): picked mon.noname-i con 0x7feb7c139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7feb7c06fe50 0x7feb7c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): picked mon.noname-f con 0x7feb7c06fe50 addr v2:172.21.15.190:3300/0 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7feb7c0719c0 con 0x7feb7c06fe50 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7feb7c071c10 con 0x7feb7c072640 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7feb7c0702d0 con 0x7feb7c139b80 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb830f3640 10 monclient(hunting): authenticate will time out at 1615.462073s 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c0a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:34668/0 (socket says 172.21.15.98:34668) 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 -- 172.21.15.98:0/1996568371 learned_addr learned my addr 172.21.15.98:0/1996568371 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): get_auth_request con 0x7feb7c072640 auth_method 0 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb7bfff640 1 --2- 172.21.15.98:0/1996568371 >> v2:172.21.15.190:3300/0 conn(0x7feb7c06fe50 0x7feb7c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb7bfff640 10 monclient(hunting): get_auth_request con 0x7feb7c06fe50 auth_method 0 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb80e8f640 1 --2- 172.21.15.98:0/1996568371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb80e8f640 10 monclient(hunting): get_auth_request con 0x7feb7c139b80 auth_method 0 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb80e8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb80e8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb80e8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient(hunting): handle_auth_done global_id 24340 payload 274 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 -- 172.21.15.98:0/1996568371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 msgr2=0x7feb7c139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 --2- 172.21.15.98:0/1996568371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 -- 172.21.15.98:0/1996568371 >> v2:172.21.15.190:3300/0 conn(0x7feb7c06fe50 msgr2=0x7feb7c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 --2- 172.21.15.98:0/1996568371 >> v2:172.21.15.190:3300/0 conn(0x7feb7c06fe50 0x7feb7c0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 monclient: found mon.noname-c 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.580+0000 7feb81690640 1 -- 172.21.15.98:0/1996568371 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c071370 con 0x7feb7c072640 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb81690640 1 --2- 172.21.15.98:0/1996568371 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c0a7f50 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7feb74009e50 tx=0x7feb7401c8f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=67a9711aabdcf3bc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 1 -- 172.21.15.98:0/1996568371 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb740090d0 con 0x7feb7c072640 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: dump: 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 1 -- 172.21.15.98:0/1996568371 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb740a5ab0 con 0x7feb7c072640 2024-09-11T03:01:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 1 -- 172.21.15.98:0/1996568371 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb740a5ca0 con 0x7feb7c072640 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 5 monclient: authenticate success, global_id 24340 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 4 set_mon_vals no callback set 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7b7fe640 10 monclient: dump: 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient: shutdown 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- 172.21.15.98:0/1996568371 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 msgr2=0x7feb7c0a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 --2- 172.21.15.98:0/1996568371 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c0a7f50 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7feb74009e50 tx=0x7feb7401c8f0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- 172.21.15.98:0/1996568371 shutdown_connections 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 --2- 172.21.15.98:0/1996568371 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c0a7f50 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 --2- 172.21.15.98:0/1996568371 >> v2:172.21.15.190:3300/0 conn(0x7feb7c06fe50 0x7feb7c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 --2- 172.21.15.98:0/1996568371 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- 172.21.15.98:0/1996568371 >> 172.21.15.98:0/1996568371 conn(0x7feb7c0ac730 msgr2=0x7feb7c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- 172.21.15.98:0/1996568371 shutdown_connections 2024-09-11T03:01:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- 172.21.15.98:0/1996568371 wait complete. 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient: monmap: 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 librados: starting msgr at 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 librados: starting objecter 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 Processor -- start 2024-09-11T03:01:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- start start 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 librados: setting wanted keys 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 librados: calling monclient init 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient: init 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient: _add_conns ranks=[6,4,7,5,0,1,2,3,8] 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb7c06fe50 0x7feb7c13a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): picked mon.noname-d con 0x7feb7c06fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c13abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c072640 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c13b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): picked mon.noname-i con 0x7feb7c139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7feb7c1435c0 con 0x7feb7c072640 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7feb7c143440 con 0x7feb7c06fe50 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7feb7c143740 con 0x7feb7c139b80 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb830f3640 10 monclient(hunting): authenticate will time out at 1615.467697s 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb7c06fe50 0x7feb7c13a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c13abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb7c06fe50 0x7feb7c13a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:32800/0 (socket says 172.21.15.98:32800) 2024-09-11T03:01:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 1 -- 172.21.15.98:0/3742327401 learned_addr learned my addr 172.21.15.98:0/3742327401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 10 monclient(hunting): get_auth_request con 0x7feb7c06fe50 auth_method 0 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 10 monclient(hunting): get_auth_request con 0x7feb7c072640 auth_method 0 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb80e8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.584+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb81690640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c13b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 10 monclient(hunting): handle_auth_done global_id 44655 payload 824 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 msgr2=0x7feb7c13b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c13b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 msgr2=0x7feb7c13abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c13abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 1 monclient: found mon.noname-d 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 1 -- 172.21.15.98:0/3742327401 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c0a9f00 con 0x7feb7c06fe50 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb81690640 10 monclient: get_auth_request con 0x7feb7c139b80 auth_method 0 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7feb7c072640 0x7feb7c13abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb81690640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7feb7c139b80 0x7feb7c13b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb80e8f640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb7c06fe50 0x7feb7c13a6b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb6400d620 tx=0x7feb6400dae0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 client.?.objecter ms_handle_connect 0x7feb7c06fe50 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb6400ba90 con 0x7feb7c06fe50 2024-09-11T03:01:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: dump: 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 5 monclient: authenticate success, global_id 44655 2024-09-11T03:01:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb64019070 con 0x7feb7c06fe50 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 10 monclient: _renew_subs 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7c0a5840 con 0x7feb7c06fe50 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb6400bc90 con 0x7feb7c06fe50 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7a7fc640 4 set_mon_vals no callback set 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 10 client.44655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: dump: 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.611 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 10 monclient: _renew_subs 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7c13caa0 con 0x7feb7c06fe50 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 1 librados: init done 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c13bb40 0x7feb7c133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feb7c06ca10 con 0x7feb7c13bb40 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7feb64013070 con 0x7feb7c06fe50 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb5c03df40 0x7feb5c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(435..435 src has 1..435) ==== 5817+0+0 (secure 0 0 0) 0x7feb6400f040 con 0x7feb7c06fe50 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 client.44655.objecter ms_dispatch 0x7feb7c000ba0 osd_map(435..435 src has 1..435) 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 3 client.44655.objecter handle_osd_map got epochs [435,435] > 0 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 3 client.44655.objecter handle_osd_map decoding full epoch 435 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c13bb40 0x7feb7c133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient: get_auth_request con 0x7feb7c13bb40 auth_method 0 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 20 client.44655.objecter dump_active .. 0 homeless 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb81690640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb5c03df40 0x7feb5c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb81690640 10 monclient: get_auth_request con 0x7feb5c03df40 auth_method 0 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 10 monclient(hunting): handle_auth_done global_id 74172 payload 824 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 1 -- 172.21.15.98:0/3742327401 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb7c0aa350 con 0x7feb7c13bb40 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 client.44655.objecter ms_handle_connect 0x7feb5c03df40 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb81690640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb5c03df40 0x7feb5c040400 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7feb740ae000 tx=0x7feb740a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb7bfff640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c13bb40 0x7feb7c133760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0a5eb0 tx=0x7feb6c00e1d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 client.44655.objecter ms_handle_connect 0x7feb7c13bb40 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 client.44655.objecter resend_mon_ops 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb6c018020 con 0x7feb7c13bb40 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: dump: 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb6c00ed10 con 0x7feb7c13bb40 2024-09-11T03:01:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c13bb40 msgr2=0x7feb7c133760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.588+0000 7feb797fa640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c13bb40 0x7feb7c133760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0a5eb0 tx=0x7feb6c00e1d0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.592+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c13bb40 msgr2=0x7feb7c133760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.592+0000 7feb797fa640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c13bb40 0x7feb7c133760 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0a5eb0 tx=0x7feb6c00e1d0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb830f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb830f3640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c133ca0 0x7feb7c1340b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb830f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7feb7c06e870 con 0x7feb7c133ca0 2024-09-11T03:01:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c133ca0 0x7feb7c1340b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient: get_auth_request con 0x7feb7c133ca0 auth_method 0 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 10 monclient(hunting): handle_auth_done global_id 74181 payload 824 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 1 -- 172.21.15.98:0/3742327401 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb7c13cf10 con 0x7feb7c133ca0 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb7bfff640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c133ca0 0x7feb7c1340b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0a5eb0 tx=0x7feb6c0046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb797fa640 10 client.44655.objecter ms_handle_connect 0x7feb7c133ca0 2024-09-11T03:01:01.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb797fa640 10 client.44655.objecter resend_mon_ops 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb6c018040 con 0x7feb7c133ca0 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb797fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.596+0000 7feb797fa640 10 monclient: dump: 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.621 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7feb6c003b10 con 0x7feb7c133ca0 2024-09-11T03:01:01.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb797fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb797fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c133ca0 msgr2=0x7feb7c1340b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb797fa640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c133ca0 0x7feb7c1340b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0a5eb0 tx=0x7feb6c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb797fa640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c133ca0 msgr2=0x7feb7c1340b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb797fa640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7feb7c133ca0 0x7feb7c1340b0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb7c0a5eb0 tx=0x7feb6c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 18527, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.640 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:01.641 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.642 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.644 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.648 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.649 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:01.650 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 10 librados: watch_flush enter 2024-09-11T03:01:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 10 librados: watch_flush exit 2024-09-11T03:01:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 20 client.44655.objecter shutdown clearing up homeless session... 2024-09-11T03:01:01.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 10 client.44655.objecter successfully canceled tick 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb5c03df40 msgr2=0x7feb5c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7feb5c03df40 0x7feb5c040400 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7feb740ae000 tx=0x7feb740a7070 comp rx=0 tx=0).stop 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 10 monclient: shutdown 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb7c06fe50 msgr2=0x7feb7c13a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 --2- 172.21.15.98:0/3742327401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7feb7c06fe50 0x7feb7c13a6b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb6400d620 tx=0x7feb6400dae0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb81690640 1 -- 172.21.15.98:0/3742327401 reap_dead start 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb81690640 1 -- 172.21.15.98:0/3742327401 reap_dead start 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 shutdown_connections 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 >> 172.21.15.98:0/3742327401 conn(0x7feb7c0ac730 msgr2=0x7feb7c0a7910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 shutdown_connections 2024-09-11T03:01:01.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 -- 172.21.15.98:0/3742327401 wait complete. 2024-09-11T03:01:01.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.600+0000 7feb830f3640 1 librados: shutdown 2024-09-11T03:01:01.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:01.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T03:01:01.654 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:01:01.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient: get_monmap_and_config 2024-09-11T03:01:01.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient: monmap: 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:01.781710+0000 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:01.781710+0000 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:01.781 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 1 Processor -- start 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 1 -- start start 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient: init 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient: _add_conns ranks=[9,1,8,6,7,3,5,4,10,0,2] 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40cc139b10 0x7f40cc072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient(hunting): picked mon.noname-a con 0x7f40cc139b10 addr v2:172.21.15.98:3300/0 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f40cc139ef0 addr v1:172.21.15.190:6789/0 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40cc13a2d0 0x7f40cc0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient(hunting): picked mon.noname-e con 0x7f40cc13a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40cc0702b0 con 0x7f40cc139ef0 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f40cc071bf0 con 0x7f40cc139b10 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f40cc071350 con 0x7f40cc13a2d0 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d3368640 10 monclient(hunting): authenticate will time out at 1615.661941s 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40cc13a2d0 0x7f40cc0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40cc13a2d0 0x7f40cc0a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35222/0 (socket says 172.21.15.98:35222) 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 1 -- 172.21.15.98:0/1764718314 learned_addr learned my addr 172.21.15.98:0/1764718314 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 10 monclient(hunting): get_auth_request con 0x7f40cc13a2d0 auth_method 0 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40cc139b10 0x7f40cc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): get_auth_request con 0x7f40cc139b10 auth_method 0 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient(hunting): handle_auth_done global_id 44802 payload 274 2024-09-11T03:01:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 1 -- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40cc13a2d0 msgr2=0x7f40cc0a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 1 --2- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40cc13a2d0 0x7f40cc0a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 1 -- 172.21.15.98:0/1764718314 >> v1:172.21.15.190:6789/0 conn(0x7f40cc139ef0 legacy=0x7f40cc0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 1 monclient: found mon.noname-a 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1104640 1 -- 172.21.15.98:0/1764718314 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc0a9620 con 0x7f40cc139b10 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.780+0000 7f40d1905640 1 --2- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40cc13a2d0 0x7f40cc0a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1104640 1 --2- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc139b10 0x7f40cc072600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009dc0 tx=0x7f40c401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e3d31116df6823 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 1 -- 172.21.15.98:0/1764718314 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40c40b2070 con 0x7f40cc139b10 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: dump: 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 5 monclient: authenticate success, global_id 44802 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 1 -- 172.21.15.98:0/1764718314 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c40a5ae0 con 0x7f40cc139b10 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 4 set_mon_vals no callback set 2024-09-11T03:01:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 1 -- 172.21.15.98:0/1764718314 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40c40a5d10 con 0x7f40cc139b10 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40cbfff640 10 monclient: dump: 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient: shutdown 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc139b10 msgr2=0x7f40cc072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 --2- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc139b10 0x7f40cc072600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f40c4009dc0 tx=0x7f40c401c8a0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- 172.21.15.98:0/1764718314 shutdown_connections 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 --2- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40cc13a2d0 0x7f40cc0a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 --2- 172.21.15.98:0/1764718314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f40cc139b10 0x7f40cc072600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- 172.21.15.98:0/1764718314 >> 172.21.15.98:0/1764718314 conn(0x7f40cc0ac790 msgr2=0x7f40cc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- 172.21.15.98:0/1764718314 shutdown_connections 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- 172.21.15.98:0/1764718314 wait complete. 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient: monmap: 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 librados: starting msgr at 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 librados: starting objecter 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 Processor -- start 2024-09-11T03:01:01.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- start start 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 librados: setting wanted keys 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 librados: calling monclient init 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient: init 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient: _add_conns ranks=[3,5,2,6,8,4,0,7,1] 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40cc139b10 0x7f40cc133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): picked mon.noname-g con 0x7f40cc139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc139ef0 0x7f40cc133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): picked mon.noname-h con 0x7f40cc139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc13a2d0 0x7f40cc1341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): picked mon.noname-b con 0x7f40cc13a2d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc144470 con 0x7f40cc13a2d0 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f40cc0a8870 con 0x7f40cc139b10 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f40cc1445f0 con 0x7f40cc139ef0 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d3368640 10 monclient(hunting): authenticate will time out at 1615.667507s 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc13a2d0 0x7f40cc1341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc13a2d0 0x7f40cc1341e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58200/0 (socket says 172.21.15.98:58200) 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 -- 172.21.15.98:0/4078857392 learned_addr learned my addr 172.21.15.98:0/4078857392 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): get_auth_request con 0x7f40cc13a2d0 auth_method 0 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient(hunting): handle_auth_done global_id 64487 payload 824 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc139ef0 msgr2=0x7f40cc133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1104640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40cc139b10 0x7f40cc133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d0903640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc139ef0 0x7f40cc133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc139ef0 0x7f40cc133ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40cc139b10 msgr2=0x7f40cc133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f40cc139b10 0x7f40cc133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 monclient: found mon.noname-b 2024-09-11T03:01:01.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.784+0000 7f40d1905640 1 -- 172.21.15.98:0/4078857392 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc073930 con 0x7f40cc13a2d0 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1905640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc13a2d0 0x7f40cc1341e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e930 tx=0x7f40bc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 client.?.objecter ms_handle_connect 0x7f40cc13a2d0 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40bc017070 con 0x7f40cc13a2d0 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: dump: 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.804 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40bc00cdb0 con 0x7f40cc13a2d0 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40bc0138e0 con 0x7f40cc13a2d0 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 5 monclient: authenticate success, global_id 64487 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 10 monclient: _renew_subs 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40cc0aa1b0 con 0x7f40cc13a2d0 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40cb7fe640 4 set_mon_vals no callback set 2024-09-11T03:01:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: dump: 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.807 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 10 client.64487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 10 monclient: _renew_subs 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40cc13d260 con 0x7f40cc13a2d0 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 1 librados: init done 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13af90 0x7f40cc02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f40cc13cee0 con 0x7f40cc13af90 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f40bc013470 con 0x7f40cc13a2d0 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13af90 0x7f40cc02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient: get_auth_request con 0x7f40cc13af90 auth_method 0 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40b803df40 0x7f40b8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(435..435 src has 1..435) ==== 5817+0+0 (secure 0 0 0) 0x7f40bc00f040 con 0x7f40cc13a2d0 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 client.64487.objecter ms_dispatch 0x7f40cc000ba0 osd_map(435..435 src has 1..435) 2024-09-11T03:01:01.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 3 client.64487.objecter handle_osd_map got epochs [435,435] > 0 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 3 client.64487.objecter handle_osd_map decoding full epoch 435 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 20 client.64487.objecter dump_active .. 0 homeless 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 10 monclient(hunting): handle_auth_done global_id 74156 payload 824 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 1 -- 172.21.15.98:0/4078857392 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40cc06e680 con 0x7f40cc13af90 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d0903640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40b803df40 0x7f40b8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d0903640 10 monclient: get_auth_request con 0x7f40b803df40 auth_method 0 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 client.64487.objecter ms_handle_connect 0x7f40cc13af90 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d1104640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13af90 0x7f40cc02d450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0a9a70 tx=0x7f40c4019640 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 client.64487.objecter resend_mon_ops 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40c40bc070 con 0x7f40cc13af90 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:01:01.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: dump: 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 client.64487.objecter ms_handle_connect 0x7f40b803df40 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40d0903640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40b803df40 0x7f40b8040400 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7f40c0007fd0 tx=0x7f40c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40c4004040 con 0x7f40cc13af90 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13af90 msgr2=0x7f40cc02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13af90 0x7f40cc02d450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0a9a70 tx=0x7f40c4019640 comp rx=0 tx=0).stop 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13af90 msgr2=0x7f40cc02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.788+0000 7f40c9ffb640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13af90 0x7f40cc02d450 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0a9a70 tx=0x7f40c4019640 comp rx=0 tx=0).stop 2024-09-11T03:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d3368640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d3368640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13b370 0x7f40cc02d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d3368640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f40cc141220 con 0x7f40cc13b370 2024-09-11T03:01:01.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13b370 0x7f40cc02d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient: get_auth_request con 0x7f40cc13b370 auth_method 0 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 10 monclient(hunting): handle_auth_done global_id 74165 payload 824 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 1 -- 172.21.15.98:0/4078857392 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40cc072600 con 0x7f40cc13b370 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40d1104640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13b370 0x7f40cc02d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f40c40b0040 tx=0x7f40c40a7000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 10 client.64487.objecter ms_handle_connect 0x7f40cc13b370 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 10 client.64487.objecter resend_mon_ops 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40c40b2040 con 0x7f40cc13b370 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 10 monclient: dump: 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.816 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f40c4004810 con 0x7f40cc13b370 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:01.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13b370 msgr2=0x7f40cc02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.796+0000 7f40c9ffb640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13b370 0x7f40cc02d990 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f40c40b0040 tx=0x7f40c40a7000 comp rx=0 tx=0).stop 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40c9ffb640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13b370 msgr2=0x7f40cc02d990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40c9ffb640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40cc13b370 0x7f40cc02d990 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 18727, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:01.823 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:01.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.826 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.828 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.829 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.830 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:01.831 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:01.832 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.833 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.834 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.835 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.837 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.838 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:01.840 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:01.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:01.842 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 10 librados: watch_flush enter 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 10 librados: watch_flush exit 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 20 client.64487.objecter shutdown clearing up homeless session... 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 10 client.64487.objecter successfully canceled tick 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40b803df40 msgr2=0x7f40b8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40b803df40 0x7f40b8040400 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7f40c0007fd0 tx=0x7f40c0009040 comp rx=0 tx=0).stop 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 10 monclient: shutdown 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc13a2d0 msgr2=0x7f40cc1341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc13a2d0 0x7f40cc1341e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00e930 tx=0x7f40bc00edf0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d1905640 1 -- 172.21.15.98:0/4078857392 reap_dead start 2024-09-11T03:01:01.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 shutdown_connections 2024-09-11T03:01:01.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 --2- 172.21.15.98:0/4078857392 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f40cc13a2d0 0x7f40cc1341e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 >> 172.21.15.98:0/4078857392 conn(0x7f40cc0ac790 msgr2=0x7f40cc0aa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 shutdown_connections 2024-09-11T03:01:01.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 -- 172.21.15.98:0/4078857392 wait complete. 2024-09-11T03:01:01.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.800+0000 7f40d3368640 1 librados: shutdown 2024-09-11T03:01:01.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:01.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T03:01:01.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:01:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient: get_monmap_and_config 2024-09-11T03:01:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient: monmap: 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:01.973580+0000 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:01.973580+0000 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:01.973 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 Processor -- start 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 -- start start 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient: init 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient: _add_conns ranks=[2,4,9,5,3,7,0,6,8,10,1] 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc580a7aa0 0x7efc580a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): picked mon.noname-b con 0x7efc580a7aa0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc580a12a0 0x7efc580a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): picked mon.noname-c con 0x7efc580a12a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc58139850 0x7efc58139c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): picked mon.noname-a con 0x7efc58139850 addr v2:172.21.15.98:3300/0 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7efc580a2b30 con 0x7efc58139850 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efc580a8900 con 0x7efc580a7aa0 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7efc580a1720 con 0x7efc580a12a0 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5ff6d640 10 monclient(hunting): authenticate will time out at 1615.853773s 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc580a7aa0 0x7efc580a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc580a12a0 0x7efc580a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc580a7aa0 0x7efc580a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:58202/0 (socket says 172.21.15.98:58202) 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7efc58139850 0x7efc58139c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 1 -- 172.21.15.98:0/3900363318 learned_addr learned my addr 172.21.15.98:0/3900363318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 10 monclient(hunting): get_auth_request con 0x7efc580a7aa0 auth_method 0 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 10 monclient(hunting): get_auth_request con 0x7efc580a12a0 auth_method 0 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): get_auth_request con 0x7efc58139850 auth_method 0 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5d508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient(hunting): handle_auth_done global_id 44811 payload 274 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 1 -- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc580a12a0 msgr2=0x7efc580a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc580a12a0 0x7efc580a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 1 -- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc580a7aa0 msgr2=0x7efc580a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc580a7aa0 0x7efc580a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 1 monclient: found mon.noname-a 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5e50a640 1 -- 172.21.15.98:0/3900363318 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc580a27c0 con 0x7efc58139850 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.972+0000 7efc5dd09640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc580a7aa0 0x7efc580a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58139850 0x7efc58139c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efc50009d90 tx=0x7efc5001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f61f2a7f1974ab server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 1 -- 172.21.15.98:0/3900363318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc500b2070 con 0x7efc58139850 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: dump: 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: _finish_auth 0 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: _check_auth_tickets 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 1 -- 172.21.15.98:0/3900363318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc500a5a70 con 0x7efc58139850 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 5 monclient: authenticate success, global_id 44811 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 1 -- 172.21.15.98:0/3900363318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc500a5ca0 con 0x7efc58139850 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 4 set_mon_vals no callback set 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5cd07640 10 monclient: dump: 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:01.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient: shutdown 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58139850 msgr2=0x7efc58139c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58139850 0x7efc58139c70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efc50009d90 tx=0x7efc5001c830 comp rx=0 tx=0).stop 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- 172.21.15.98:0/3900363318 shutdown_connections 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58139850 0x7efc58139c70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7efc580a12a0 0x7efc580a83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/3900363318 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efc580a7aa0 0x7efc580a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- 172.21.15.98:0/3900363318 >> 172.21.15.98:0/3900363318 conn(0x7efc580ac690 msgr2=0x7efc580a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- 172.21.15.98:0/3900363318 shutdown_connections 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- 172.21.15.98:0/3900363318 wait complete. 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient: build_initial_monmap 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient: monmap: 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:01.993 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 librados: starting msgr at 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 librados: starting objecter 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 Processor -- start 2024-09-11T03:01:01.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- start start 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 librados: setting wanted keys 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 librados: calling monclient init 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient: init 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient: _add_conns ranks=[3,7,5,8,4,0,6,1,2] 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580a12a0 0x7efc581d1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): picked mon.noname-g con 0x7efc580a12a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc580a7aa0 0x7efc58138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): picked mon.noname-i con 0x7efc580a7aa0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 0x7efc5813d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): picked mon.noname-h con 0x7efc58139850 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efc580a65d0 con 0x7efc580a12a0 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7efc580a6450 con 0x7efc58139850 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7efc581432c0 con 0x7efc580a7aa0 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5ff6d640 10 monclient(hunting): authenticate will time out at 1615.858888s 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc580a7aa0 0x7efc58138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 0x7efc5813d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc580a7aa0 0x7efc58138910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:49606/0 (socket says 172.21.15.98:49606) 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580a12a0 0x7efc581d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 1 -- 172.21.15.98:0/2665025083 learned_addr learned my addr 172.21.15.98:0/2665025083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:01.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): get_auth_request con 0x7efc580a7aa0 auth_method 0 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 10 monclient(hunting): get_auth_request con 0x7efc58139850 auth_method 0 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 10 monclient(hunting): get_auth_request con 0x7efc580a12a0 auth_method 0 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 msgr2=0x7efc5813d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 msgr2=0x7efc5813d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 0x7efc5813d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5e50a640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 0x7efc5813d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 10 monclient(hunting): handle_auth_done global_id 19897 payload 824 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 msgr2=0x7efc5813d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7efc58139850 0x7efc5813d8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580a12a0 msgr2=0x7efc581d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.976+0000 7efc5d508640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efc580a12a0 0x7efc581d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5d508640 10 monclient: _finish_hunting 0 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5d508640 1 monclient: found mon.noname-i 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5d508640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5d508640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5d508640 1 -- 172.21.15.98:0/2665025083 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc581242b0 con 0x7efc580a7aa0 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5d508640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc580a7aa0 0x7efc58138910 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7efc5400e930 tx=0x7efc5400edf0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 client.?.objecter ms_handle_connect 0x7efc580a7aa0 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc54017070 con 0x7efc580a7aa0 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:01.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: dump: 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.000 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: _finish_auth 0 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc54013470 con 0x7efc580a7aa0 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 5 monclient: authenticate success, global_id 19897 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc54013a90 con 0x7efc580a7aa0 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: dump: 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.002 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4ffff640 4 set_mon_vals no callback set 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 10 monclient: _renew_subs 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc58122c30 con 0x7efc580a7aa0 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 10 client.19897.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 10 monclient: _renew_subs 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7efc580a5230 con 0x7efc580a7aa0 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 1 librados: init done 2024-09-11T03:01:02.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc58138e50 0x7efc5813ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7efc5806ca10 con 0x7efc58138e50 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc58138e50 0x7efc5813ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient: get_auth_request con 0x7efc58138e50 auth_method 0 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7efc54013c30 con 0x7efc580a7aa0 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3c03df40 0x7efc3c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 5 ==== osd_map(436..436 src has 1..436) ==== 6189+0+0 (secure 0 0 0) 0x7efc5400f040 con 0x7efc580a7aa0 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 client.19897.objecter ms_dispatch 0x7efc58000ba0 osd_map(436..436 src has 1..436) 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 10 monclient(hunting): handle_auth_done global_id 19906 payload 824 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 3 client.19897.objecter handle_osd_map got epochs [436,436] > 0 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 3 client.19897.objecter handle_osd_map decoding full epoch 436 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 1 -- 172.21.15.98:0/2665025083 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc580a4de0 con 0x7efc58138e50 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 20 client.19897.objecter dump_active .. 0 homeless 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5e50a640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3c03df40 0x7efc3c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5e50a640 10 monclient: get_auth_request con 0x7efc3c03df40 auth_method 0 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 client.19897.objecter ms_handle_connect 0x7efc58138e50 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 client.19897.objecter resend_mon_ops 2024-09-11T03:01:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5dd09640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc58138e50 0x7efc5813ddf0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7efc480119a0 tx=0x7efc48011e60 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc48014440 con 0x7efc58138e50 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 monclient: dump: 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc4e7fc640 10 client.19897.objecter ms_handle_connect 0x7efc3c03df40 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.980+0000 7efc5e50a640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3c03df40 0x7efc3c040400 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7efc500af000 tx=0x7efc500a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.984+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efc4800fb60 con 0x7efc58138e50 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.984+0000 7efc4e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.984+0000 7efc4e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.984+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc58138e50 msgr2=0x7efc5813ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.984+0000 7efc4e7fc640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc58138e50 0x7efc5813ddf0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7efc480119a0 tx=0x7efc48011e60 comp rx=0 tx=0).stop 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.984+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc58138e50 msgr2=0x7efc5813ddf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.984+0000 7efc4e7fc640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc58138e50 0x7efc5813ddf0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5ff6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc5813e330 0x7efc58071710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5ff6d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7efc5806e870 con 0x7efc5813e330 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc5813e330 0x7efc58071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient: get_auth_request con 0x7efc5813e330 auth_method 0 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 10 monclient(hunting): handle_auth_done global_id 19915 payload 824 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 1 -- 172.21.15.98:0/2665025083 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efc5813e740 con 0x7efc5813e330 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc4e7fc640 10 client.19897.objecter ms_handle_connect 0x7efc5813e330 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc4e7fc640 10 client.19897.objecter resend_mon_ops 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc5dd09640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc5813e330 0x7efc58071710 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7efc480119a0 tx=0x7efc480046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc48014ac0 con 0x7efc5813e330 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc4e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.988+0000 7efc4e7fc640 10 monclient: dump: 2024-09-11T03:01:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7efc4802c420 con 0x7efc5813e330 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc4e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc4e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc5813e330 msgr2=0x7efc58071710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc4e7fc640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc5813e330 0x7efc58071710 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7efc480119a0 tx=0x7efc480046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc4e7fc640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc5813e330 msgr2=0x7efc58071710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc4e7fc640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc5813e330 0x7efc58071710 secure :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7efc480119a0 tx=0x7efc480046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.011 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 76, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 677761, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:02.012 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:02.013 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:02.014 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:02.015 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:02.016 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:02.017 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.018 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:02.020 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:02.021 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.024 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:02.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:02.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:02.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:02.033 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:02.034 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:02.035 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 10 librados: watch_flush enter 2024-09-11T03:01:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 10 librados: watch_flush exit 2024-09-11T03:01:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 20 client.19897.objecter shutdown clearing up homeless session... 2024-09-11T03:01:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 10 client.19897.objecter successfully canceled tick 2024-09-11T03:01:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3c03df40 msgr2=0x7efc3c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efc3c03df40 0x7efc3c040400 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7efc500af000 tx=0x7efc500a7070 comp rx=0 tx=0).stop 2024-09-11T03:01:02.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 10 monclient: shutdown 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc580a7aa0 msgr2=0x7efc58138910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc580a7aa0 0x7efc58138910 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7efc5400e930 tx=0x7efc5400edf0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5e50a640 1 -- 172.21.15.98:0/2665025083 reap_dead start 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 shutdown_connections 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 --2- 172.21.15.98:0/2665025083 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7efc580a7aa0 0x7efc58138910 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 >> 172.21.15.98:0/2665025083 conn(0x7efc580ac690 msgr2=0x7efc580a55d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 shutdown_connections 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 -- 172.21.15.98:0/2665025083 wait complete. 2024-09-11T03:01:02.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:01.992+0000 7efc5ff6d640 1 librados: shutdown 2024-09-11T03:01:02.039 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:01:02.039 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['d'] 2024-09-11T03:01:02.039 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b', 'c', 'e'] 2024-09-11T03:01:02.039 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-09-11T03:01:02.039 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d store 2024-09-11T03:01:02.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d sync_force --yes-i-really-mean-it 2024-09-11T03:01:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient: get_monmap_and_config 2024-09-11T03:01:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient: build_initial_monmap 2024-09-11T03:01:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient: monmap: 2024-09-11T03:01:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:02.163178+0000 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:02.163178+0000 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:02.163 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 1 Processor -- start 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 1 -- start start 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient: init 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient: _add_conns ranks=[6,0,3,5,10,1,2,9,7,4,8] 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c08139b10 0x7f8c080725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient(hunting): picked mon.noname-d con 0x7f8c08139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c0806fe30 addr v1:172.21.15.98:6789/0 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c0813ea00 0x7f8c08140df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient(hunting): picked mon.noname-g con 0x7f8c0813ea00 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c0813a600 con 0x7f8c0806fe30 2024-09-11T03:01:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8c080a9650 con 0x7f8c0813ea00 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8c08071350 con 0x7f8c08139b10 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.160+0000 7f8c0e4c0640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 10 monclient(hunting): authenticate will time out at 1616.044012s 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c08139b10 0x7f8c080725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c077fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8c0806fe30 0x7f8c08139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:54704/0 (socket says 172.21.15.98:54704) 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c077fe640 1 -- 172.21.15.98:0/2651547526 learned_addr learned my addr 172.21.15.98:0/2651547526 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): get_auth_request con 0x7f8c08139b10 auth_method 0 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0ca5d640 1 --2- 172.21.15.98:0/2651547526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c0813ea00 0x7f8c08140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient(hunting): handle_auth_done global_id 44664 payload 274 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 1 -- 172.21.15.98:0/2651547526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c0813ea00 msgr2=0x7f8c08140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 1 --2- 172.21.15.98:0/2651547526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c0813ea00 0x7f8c08140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 1 -- 172.21.15.98:0/2651547526 >> v1:172.21.15.98:6789/0 conn(0x7f8c0806fe30 legacy=0x7f8c08139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 1 monclient: found mon.noname-d 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 1 -- 172.21.15.98:0/2651547526 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1391308859 0 0) 0x7f8c0813a600 con 0x7f8c0806fe30 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:01:02.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 1 -- 172.21.15.98:0/2651547526 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c08071630 con 0x7f8c08139b10 2024-09-11T03:01:02.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0ca5d640 10 monclient: get_auth_request con 0x7f8c0813ea00 auth_method 0 2024-09-11T03:01:02.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:01:02.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0ca5d640 1 --2- 172.21.15.98:0/2651547526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c0813ea00 0x7f8c08140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c07fff640 1 --2- 172.21.15.98:0/2651547526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c08139b10 0x7f8c080725d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009df0 tx=0x7f8bf801c530 comp rx=0 tx=0).ready entity=mon.6 client_cookie=5c5a90660a1354d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 1 -- 172.21.15.98:0/2651547526 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80b1070 con 0x7f8c08139b10 2024-09-11T03:01:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:01:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: dump: 2024-09-11T03:01:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.174 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.175 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: _finish_auth 0 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 5 monclient: authenticate success, global_id 44664 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 1 -- 172.21.15.98:0/2651547526 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bf80a4c40 con 0x7f8c08139b10 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 4 set_mon_vals no callback set 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 1 -- 172.21.15.98:0/2651547526 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80a4df0 con 0x7f8c08139b10 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:02.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c06ffd640 10 monclient: dump: 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.177 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 10 monclient: shutdown 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/2651547526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c08139b10 msgr2=0x7f8c080725d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 --2- 172.21.15.98:0/2651547526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c08139b10 0x7f8c080725d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8009df0 tx=0x7f8bf801c530 comp rx=0 tx=0).stop 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/2651547526 shutdown_connections 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 --2- 172.21.15.98:0/2651547526 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8c0813ea00 0x7f8c08140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 --2- 172.21.15.98:0/2651547526 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c08139b10 0x7f8c080725d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/2651547526 >> 172.21.15.98:0/2651547526 conn(0x7f8c080ac770 msgr2=0x7f8c080a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/2651547526 shutdown_connections 2024-09-11T03:01:02.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.164+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/2651547526 wait complete. 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: build_initial_monmap 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: monmap: 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:02.179 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 librados: starting msgr at 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 librados: starting objecter 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 Processor -- start 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 -- start start 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 librados: setting wanted keys 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 librados: calling monclient init 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: init 2024-09-11T03:01:02.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: _add_conns ranks=[8,5,1,4,6,3,0,7,2] 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c08139b10 0x7f8c080a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): picked mon.noname-e con 0x7f8c08139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0813ea00 0x7f8c081386d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): picked mon.noname-h con 0x7f8c0813ea00 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c080a2c90 0x7f8c080a3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): picked mon.noname-f con 0x7f8c080a2c90 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8c08144700 con 0x7f8c080a2c90 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8c08144580 con 0x7f8c0813ea00 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8c08144880 con 0x7f8c08139b10 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): authenticate will time out at 1616.049117s 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c08139b10 0x7f8c080a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c08139b10 0x7f8c080a52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35230/0 (socket says 172.21.15.98:35230) 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 -- 172.21.15.98:0/3599723668 learned_addr learned my addr 172.21.15.98:0/3599723668 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): get_auth_request con 0x7f8c08139b10 auth_method 0 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:02.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0813ea00 0x7f8c081386d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0ca5d640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c080a2c90 0x7f8c080a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient(hunting): handle_auth_done global_id 24857 payload 824 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0813ea00 msgr2=0x7f8c081386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0813ea00 0x7f8c081386d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c080a2c90 msgr2=0x7f8c080a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c080a2c90 0x7f8c080a3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 monclient: found mon.noname-e 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 -- 172.21.15.98:0/3599723668 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c080a5c70 con 0x7f8c08139b10 2024-09-11T03:01:02.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient: get_auth_request con 0x7f8c0813ea00 auth_method 0 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0ca5d640 10 monclient: get_auth_request con 0x7f8c080a2c90 auth_method 0 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8c0813ea00 0x7f8c081386d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0ca5d640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c080a2c90 0x7f8c080a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c07fff640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c08139b10 0x7f8c080a52e0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f8bf801cdd0 tx=0x7f8bf80195f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 client.?.objecter ms_handle_connect 0x7f8c08139b10 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80ba020 con 0x7f8c08139b10 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: dump: 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.184 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.185 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: _finish_auth 0 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bf80b2040 con 0x7f8c08139b10 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf80b1070 con 0x7f8c08139b10 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 5 monclient: authenticate success, global_id 24857 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: _renew_subs 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c08123060 con 0x7f8c08139b10 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c067fc640 4 set_mon_vals no callback set 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: dump: 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.187 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 client.24857.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: _renew_subs 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c080a8fb0 con 0x7f8c08139b10 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 librados: init done 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c0806fe30 0x7f8c0802e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8c0813e740 con 0x7f8c0806fe30 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c0806fe30 0x7f8c0802e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient: get_auth_request con 0x7f8c0806fe30 auth_method 0 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8bf80ac440 con 0x7f8c08139b10 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 10 monclient(hunting): handle_auth_done global_id 44673 payload 824 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 1 -- 172.21.15.98:0/3599723668 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c0813e3d0 con 0x7f8c0806fe30 2024-09-11T03:01:02.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c077fe640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c0806fe30 0x7f8c0802e5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0000f70 tx=0x7f8bf000d950 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be803df40 0x7f8be8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 client.24857.objecter ms_handle_connect 0x7f8c0806fe30 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 client.24857.objecter resend_mon_ops 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(436..436 src has 1..436) ==== 6189+0+0 (secure 0 0 0) 0x7f8bf80e9db0 con 0x7f8c08139b10 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 client.24857.objecter ms_dispatch 0x7f8c08000ba0 osd_map(436..436 src has 1..436) 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 3 client.24857.objecter handle_osd_map got epochs [436,436] > 0 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 3 client.24857.objecter handle_osd_map decoding full epoch 436 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 20 client.24857.objecter dump_active .. 0 homeless 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf0018070 con 0x7f8c0806fe30 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c04ff9640 10 monclient: dump: 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.168+0000 7f8c0ca5d640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be803df40 0x7f8be8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c0ca5d640 10 monclient: get_auth_request con 0x7f8be803df40 auth_method 0 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8bf00073a0 con 0x7f8c0806fe30 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c0806fe30 msgr2=0x7f8c0802e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c0806fe30 0x7f8c0802e5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0000f70 tx=0x7f8bf000d950 comp rx=0 tx=0).stop 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c0806fe30 msgr2=0x7f8c0802e5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c0806fe30 0x7f8c0802e5d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c04ff9640 10 client.24857.objecter ms_handle_connect 0x7f8be803df40 2024-09-11T03:01:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.172+0000 7f8c0ca5d640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be803df40 0x7f8be8040400 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7f8c08072bc0 tx=0x7f8c00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c0e4c0640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T03:01:02.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c0e4c0640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c080a3650 0x7f8c08063e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:02.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c0e4c0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:02.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8c0806ca10 con 0x7f8c080a3650 2024-09-11T03:01:02.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c080a3650 0x7f8c08063e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient: get_auth_request con 0x7f8c080a3650 auth_method 0 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 10 monclient(hunting): handle_auth_done global_id 44682 payload 824 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 1 -- 172.21.15.98:0/3599723668 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f8c080725d0 con 0x7f8c080a3650 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c077fe640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c080a3650 0x7f8c08063e00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c080a4df0 tx=0x7f8bf0000ee0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c04ff9640 10 client.24857.objecter ms_handle_connect 0x7f8c080a3650 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c04ff9640 10 client.24857.objecter resend_mon_ops 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.176+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8bf0028080 con 0x7f8c080a3650 2024-09-11T03:01:02.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 10 monclient: dump: 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:02.196 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f8bf001a620 con 0x7f8c080a3650 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c080a3650 msgr2=0x7f8c08063e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c080a3650 0x7f8c08063e00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c080a4df0 tx=0x7f8bf0000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c080a3650 msgr2=0x7f8c08063e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c04ff9640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8c080a3650 0x7f8c08063e00 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8c080a4df0 tx=0x7f8bf0000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.198 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:02.198 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-09-11T03:01:02.198 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T03:01:02.198 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:02.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 10 librados: watch_flush enter 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 10 librados: watch_flush exit 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 20 client.24857.objecter shutdown clearing up homeless session... 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 10 client.24857.objecter successfully canceled tick 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be803df40 msgr2=0x7f8be8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8be803df40 0x7f8be8040400 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7f8c08072bc0 tx=0x7f8c00008040 comp rx=0 tx=0).stop 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 10 monclient: shutdown 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c08139b10 msgr2=0x7f8c080a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 --2- 172.21.15.98:0/3599723668 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8c08139b10 0x7f8c080a52e0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f8bf801cdd0 tx=0x7f8bf80195f0 comp rx=0 tx=0).stop 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0ca5d640 1 -- 172.21.15.98:0/3599723668 reap_dead start 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0ca5d640 1 -- 172.21.15.98:0/3599723668 reap_dead start 2024-09-11T03:01:02.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 shutdown_connections 2024-09-11T03:01:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 >> 172.21.15.98:0/3599723668 conn(0x7f8c080ac770 msgr2=0x7f8c0813f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 shutdown_connections 2024-09-11T03:01:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 -- 172.21.15.98:0/3599723668 wait complete. 2024-09-11T03:01:02.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:02.180+0000 7f8c0e4c0640 1 librados: shutdown 2024-09-11T03:01:02.201 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-09-11T03:01:02.201 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-11T03:01:02.201 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:01:02.215 INFO:tasks.ceph.mon.d:Stopped 2024-09-11T03:01:02.215 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T03:01:02.215 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T03:01:02.215 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T03:01:02.215 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T03:01:02.216 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-09-11T03:01:02.216 INFO:tasks.ceph.mon.e:Sent signal 19 2024-09-11T03:01:02.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T03:01:17.217 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T03:01:17.217 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T03:01:17.217 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T03:01:17.218 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T03:01:17.218 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-09-11T03:01:17.218 INFO:tasks.ceph.mon.e:Sent signal 18 2024-09-11T03:01:17.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-09-11T03:01:17.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:01:17.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient: get_monmap_and_config 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient: build_initial_monmap 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient: monmap: 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:17.350649+0000 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:17.350649+0000 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:17.349 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:17.350 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:17.351 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 Processor -- start 2024-09-11T03:01:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 -- start start 2024-09-11T03:01:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient: init 2024-09-11T03:01:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient: _add_conns ranks=[6,2,10,1,5,3,4,9,7,8,0] 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 0x7faa10139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): picked mon.noname-d con 0x7faa101399e0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): picked mon.noname-b con 0x7faa1013a300 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): picked mon.noname-f con 0x7faa10071b60 addr v2:172.21.15.190:3300/0 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7faa1006ed40 con 0x7faa10071b60 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7faa100a7870 con 0x7faa1013a300 2024-09-11T03:01:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7faa100704a0 con 0x7faa101399e0 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa15437640 10 monclient(hunting): authenticate will time out at 1631.230828s 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0effd640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 msgr2=0x7faa10139dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0effd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 0x7faa10139dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:34724/0 (socket says 172.21.15.98:34724) 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 learned_addr learned my addr 172.21.15.98:0/1023365244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request con 0x7faa1013a300 auth_method 0 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7faa10071b60 auth_method 0 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.348+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:17.552 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 msgr2=0x7faa10139dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 0x7faa10139dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request con 0x7faa1013a300 auth_method 0 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:17.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7faa10071b60 auth_method 0 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:17.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.552+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:01:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 msgr2=0x7faa10139dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T03:01:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 0x7faa10139dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7faa10071b60 auth_method 0 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:01:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request con 0x7faa1013a300 auth_method 0 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:17.956+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:01:18.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 msgr2=0x7faa10139dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T03:01:18.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 0x7faa10139dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:01:18.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7faa10071b60 auth_method 0 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request con 0x7faa1013a300 auth_method 0 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:18.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:18.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:18.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:18.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:01:18.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:18.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:18.760+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): tick 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 monclient(hunting): continuing hunt 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 msgr2=0x7faa10139dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7faa101399e0 0x7faa10139dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 msgr2=0x7faa100a58c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7faa1013a300 0x7faa100a58c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 -- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 msgr2=0x7faa100a5e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:01:20.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 --2- 172.21.15.98:0/1023365244 >> v2:172.21.15.190:3300/0 conn(0x7faa10071b60 0x7faa100a5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient: _add_conns ranks=[8,0,4,3,5,9,1,7] 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): picked mon.noname-e con 0x7fa9fc003820 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9fc0061b0 addr v1:172.21.15.98:6789/0 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): picked mon.noname-c con 0x7fa9fc009e60 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 -- 172.21.15.98:0/1023365244 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa1006f080 con 0x7fa9fc0061b0 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 -- 172.21.15.98:0/1023365244 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7faa1006ed40 con 0x7fa9fc009e60 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 1 -- 172.21.15.98:0/1023365244 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7faa100a7870 con 0x7fa9fc003820 2024-09-11T03:01:20.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0cff9640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7fa9fc009e60 auth_method 0 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 10 monclient(hunting): get_auth_request con 0x7fa9fc003820 auth_method 0 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:20.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:20.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:20.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.348+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:20.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.352+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:20.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.352+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7fa9fc009e60 auth_method 0 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 10 monclient(hunting): get_auth_request con 0x7fa9fc003820 auth_method 0 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:20.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:01:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:01:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:20.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.556+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:01:20.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:20.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7fa9fc009e60 auth_method 0 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 10 monclient(hunting): get_auth_request con 0x7fa9fc003820 auth_method 0 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:20.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:20.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:01:20.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:20.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:20.960+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:01:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 10 monclient(hunting): get_auth_request con 0x7fa9fc003820 auth_method 0 2024-09-11T03:01:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7fa9fc009e60 auth_method 0 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:01:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:21.764+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:01:22.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.284+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035048838 0 0) 0x7faa1006f080 con 0x7fa9fc0061b0 2024-09-11T03:01:22.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.284+0000 7faa0dffb640 10 monclient(hunting): _negotiate 2024-09-11T03:01:22.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.284+0000 7faa0dffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.284+0000 7faa0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.284+0000 7faa0dffb640 10 monclient(hunting): my global_id is 44829 2024-09-11T03:01:22.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.284+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9f8003530 con 0x7fa9fc0061b0 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2617401965 0 0) 0x7fa9f8003530 con 0x7fa9fc0061b0 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 msgr2=0x7fa9fc005c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9fc003820 0x7fa9fc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 msgr2=0x7fa9fc00c320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9fc009e60 0x7fa9fc00c320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 10 monclient: _finish_hunting 0 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:01:22.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9fc00dc40 con 0x7fa9fc0061b0 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 10 monclient: _finish_auth 0 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 10 monclient: _check_auth_tickets 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1023365244 reap_dead start 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7faa0400f040 con 0x7fa9fc0061b0 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.296+0000 7faa0dffb640 10 monclient: dump: 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.302 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 >> v1:172.21.15.98:6789/0 conn(0x7fa9fc0061b0 legacy=0x7fa9fc008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient: _add_conns ranks=[8,4,1,7,2,5,3,6,0] 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9f8004210 0x7fa9f8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): picked mon.e con 0x7fa9f8004210 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7fa9f80052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): picked mon.c con 0x7fa9f8004e20 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8009c00 0x7fa9f800c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): picked mon.f con 0x7fa9f8009c00 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa1006ed40 con 0x7fa9f8009c00 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7faa100a7870 con 0x7fa9f8004e20 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa9f800c9a0 con 0x7fa9f8004210 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0dffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7fa9f80052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9f8004210 0x7fa9f8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): get_auth_request con 0x7fa9f8004210 auth_method 0 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request con 0x7fa9f8004e20 auth_method 0 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8009c00 0x7fa9f800c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7fa9f8009c00 auth_method 0 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient(hunting): handle_auth_done global_id 44829 payload 293 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 msgr2=0x7fa9f80052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7fa9f80052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8009c00 msgr2=0x7fa9f800c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8009c00 0x7fa9f800c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient: _finish_hunting 0 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 1 monclient: found mon.e 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 1 -- 172.21.15.98:0/1023365244 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f8003530 con 0x7fa9f8004210 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient: _finish_auth 0 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:22.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa15437640 5 monclient: authenticate success, global_id 44829 2024-09-11T03:01:22.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.300+0000 7faa15437640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0effd640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9f8004210 0x7fa9f8004810 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7faa00009ed0 tx=0x7faa0001d4d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=8bf48dad993317c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa000ab070 con 0x7fa9f8004210 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 10 monclient: dump: 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.313 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa000af870 con 0x7fa9f8004210 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 4 set_mon_vals no callback set 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 1 -- 172.21.15.98:0/1023365244 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa00009ad0 con 0x7fa9f8004210 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa0dffb640 10 monclient: dump: 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.314 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.315 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 10 monclient: shutdown 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 -- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9f8004210 msgr2=0x7fa9f8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9f8004210 0x7fa9f8004810 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7faa00009ed0 tx=0x7faa0001d4d0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 -- 172.21.15.98:0/1023365244 shutdown_connections 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7fa9f80052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8009c00 0x7fa9f800c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 --2- 172.21.15.98:0/1023365244 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa9f8004210 0x7fa9f8004810 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 -- 172.21.15.98:0/1023365244 >> 172.21.15.98:0/1023365244 conn(0x7faa100aa4b0 msgr2=0x7faa100a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 -- 172.21.15.98:0/1023365244 shutdown_connections 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.304+0000 7faa15437640 1 -- 172.21.15.98:0/1023365244 wait complete. 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient: build_initial_monmap 2024-09-11T03:01:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient: monmap: 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:22.317 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 librados: starting msgr at 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 librados: starting objecter 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 Processor -- start 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 -- start start 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 librados: setting wanted keys 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 librados: calling monclient init 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient: init 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient: _add_conns ranks=[1,4,7,8,3,2,0,6,5] 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8004210 0x7faa101d1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): picked mon.noname-f con 0x7fa9f8004210 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7faa10138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): picked mon.noname-c con 0x7fa9f8004e20 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa9f8009c00 0x7faa10138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): picked mon.noname-i con 0x7fa9f8009c00 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7faa100a6460 con 0x7fa9f8004210 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7faa100a62e0 con 0x7fa9f8004e20 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7faa10138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7faa10138880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39888/0 (socket says 172.21.15.98:39888) 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 1 -- 172.21.15.98:0/1393722969 learned_addr learned my addr 172.21.15.98:0/1393722969 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:22.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7faa100a65e0 con 0x7fa9f8009c00 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa15437640 10 monclient(hunting): authenticate will time out at 1636.191197s 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request con 0x7fa9f8004e20 auth_method 0 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8004210 0x7faa101d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 10 monclient(hunting): get_auth_request con 0x7fa9f8004210 auth_method 0 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa9f8009c00 0x7faa10138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request con 0x7fa9f8009c00 auth_method 0 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient(hunting): handle_auth_done global_id 19924 payload 824 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1393722969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 msgr2=0x7faa10138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7faa10138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8004210 msgr2=0x7faa101d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8004210 0x7faa101d1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 1 monclient: found mon.noname-i 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T03:01:22.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.308+0000 7faa0f7fe640 1 -- 172.21.15.98:0/1393722969 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa10126240 con 0x7fa9f8009c00 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa0f7fe640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa9f8009c00 0x7faa10138dc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7faa100a79f0 tx=0x7faa080187f0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 client.?.objecter ms_handle_connect 0x7fa9f8009c00 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 1 -- 172.21.15.98:0/1393722969 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa08028070 con 0x7fa9f8009c00 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: dump: 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.323 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: _finish_auth 0 2024-09-11T03:01:22.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: _check_auth_tickets 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 1 -- 172.21.15.98:0/1393722969 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa08017600 con 0x7fa9f8009c00 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 5 monclient: authenticate success, global_id 19924 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 monclient: _renew_subs 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa10072cf0 con 0x7fa9f8009c00 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 1 -- 172.21.15.98:0/1393722969 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa08018e10 con 0x7fa9f8009c00 2024-09-11T03:01:22.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 monclient: dump: 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.326 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa0d7fa640 4 set_mon_vals no callback set 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 client.19924.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 monclient: _renew_subs 2024-09-11T03:01:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T03:01:22.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7faa10124ef0 con 0x7fa9f8009c00 2024-09-11T03:01:22.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 1 librados: init done 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa100704a0 con 0x7fa9f8009c00 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 1 -- 172.21.15.98:0/1393722969 <== mon.7 v2:172.21.15.190:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7faa080177f0 con 0x7fa9f8009c00 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa9fc03edd0 0x7fa9fc041290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 1 -- 172.21.15.98:0/1393722969 <== mon.7 v2:172.21.15.190:3303/0 5 ==== osd_map(437..437 src has 1..437) ==== 6202+0+0 (secure 0 0 0) 0x7faa08020070 con 0x7fa9f8009c00 2024-09-11T03:01:22.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 client.19924.objecter ms_dispatch 0x7faa10000ba0 osd_map(437..437 src has 1..437) 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 3 client.19924.objecter handle_osd_map got epochs [437,437] > 0 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 3 client.19924.objecter handle_osd_map decoding full epoch 437 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 20 client.19924.objecter dump_active .. 0 homeless 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa0effd640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa9fc03edd0 0x7fa9fc041290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa0effd640 10 monclient: get_auth_request con 0x7fa9fc03edd0 auth_method 0 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7fa9f77fe640 10 client.19924.objecter ms_handle_connect 0x7fa9fc03edd0 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.312+0000 7faa0effd640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa9fc03edd0 0x7fa9fc041290 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7faa000b0000 tx=0x7faa00005d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.324+0000 7fa9f77fe640 1 -- 172.21.15.98:0/1393722969 <== mon.7 v2:172.21.15.190:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7faa08010040 con 0x7fa9f8009c00 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.324+0000 7fa9f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.324+0000 7fa9f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.580+0000 7faa15437640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:01:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.580+0000 7faa15437640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:01:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.580+0000 7faa15437640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.190:3303/0 2024-09-11T03:01:22.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.580+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faa10069020 con 0x7fa9f8009c00 2024-09-11T03:01:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7fa9f77fe640 1 -- 172.21.15.98:0/1393722969 <== mon.7 v2:172.21.15.190:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7faa08016dc0 con 0x7fa9f8009c00 2024-09-11T03:01:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7fa9f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:01:22.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7fa9f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:22.583 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":84,"quorum":[0,1,2,3,4,5,7,8],"quorum_names":["a","f","b","g","c","h","i","e"],"quorum_leader_name":"i","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:01:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 10 librados: watch_flush enter 2024-09-11T03:01:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 10 librados: watch_flush exit 2024-09-11T03:01:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 20 client.19924.objecter shutdown clearing up homeless session... 2024-09-11T03:01:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 10 client.19924.objecter successfully canceled tick 2024-09-11T03:01:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa9fc03edd0 msgr2=0x7fa9fc041290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa9fc03edd0 0x7fa9fc041290 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7faa000b0000 tx=0x7faa00005d50 comp rx=0 tx=0).stop 2024-09-11T03:01:22.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 10 monclient: shutdown 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa9f8009c00 msgr2=0x7faa10138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa9f8009c00 0x7faa10138dc0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7faa100a79f0 tx=0x7faa080187f0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 shutdown_connections 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa9fc03edd0 0x7fa9fc041290 unknown :-1 s=CLOSED pgs=3594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa9f8009c00 0x7faa10138dc0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9f8004e20 0x7faa10138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 --2- 172.21.15.98:0/1393722969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9f8004210 0x7faa101d1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 >> 172.21.15.98:0/1393722969 conn(0x7faa100aa4b0 msgr2=0x7faa100a9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 shutdown_connections 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 -- 172.21.15.98:0/1393722969 wait complete. 2024-09-11T03:01:22.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.584+0000 7faa15437640 1 librados: shutdown 2024-09-11T03:01:22.602 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-09-11T03:01:22.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-09-11T03:01:22.602 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:01:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient: get_monmap_and_config 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient: build_initial_monmap 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient: monmap: 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:22.725683+0000 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:22.725683+0000 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:22.725 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 Processor -- start 2024-09-11T03:01:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 -- start start 2024-09-11T03:01:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient: init 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient: _add_conns ranks=[2,4,3,8,0,9,5,7,6,1,10] 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a74139b80 0x7f9a74139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): picked mon.noname-b con 0x7f9a74139b80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a7406fce0 0x7f9a740a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): picked mon.noname-c con 0x7f9a7406fce0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a740724b0 0x7f9a740a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): picked mon.noname-g con 0x7f9a740724b0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a74071850 con 0x7f9a74139b80 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.730 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9a74071aa0 con 0x7f9a740724b0 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f9a74070160 con 0x7f9a7406fce0 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a7b517640 10 monclient(hunting): authenticate will time out at 1636.605967s 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a74139b80 0x7f9a74139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a7406fce0 0x7f9a740a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a74139b80 0x7f9a74139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:34768/0 (socket says 172.21.15.98:34768) 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 -- 172.21.15.98:0/1533767653 learned_addr learned my addr 172.21.15.98:0/1533767653 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): get_auth_request con 0x7f9a74139b80 auth_method 0 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 10 monclient(hunting): get_auth_request con 0x7f9a7406fce0 auth_method 0 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient(hunting): handle_auth_done global_id 64505 payload 274 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 -- 172.21.15.98:0/1533767653 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a7406fce0 msgr2=0x7f9a740a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:22.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a7406fce0 0x7f9a740a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a79ab4640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a740724b0 0x7f9a740a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 -- 172.21.15.98:0/1533767653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a740724b0 msgr2=0x7f9a740a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a740724b0 0x7f9a740a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient: _finish_hunting 0 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 monclient: found mon.noname-b 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.724+0000 7f9a792b3640 1 -- 172.21.15.98:0/1533767653 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a740a94a0 con 0x7f9a74139b80 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a74139b80 0x7f9a74139f60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a700098a0 tx=0x7f9a7001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e0a10d075ae905 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 1 -- 172.21.15.98:0/1533767653 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a700b2070 con 0x7f9a74139b80 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:01:22.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: dump: 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.734 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: _finish_auth 0 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: _check_auth_tickets 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 1 -- 172.21.15.98:0/1533767653 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a700ab7f0 con 0x7f9a74139b80 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 5 monclient: authenticate success, global_id 64505 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 1 -- 172.21.15.98:0/1533767653 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a700abe10 con 0x7f9a74139b80 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:22.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 4 set_mon_vals no callback set 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a6bfff640 10 monclient: dump: 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.736 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient: shutdown 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- 172.21.15.98:0/1533767653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a74139b80 msgr2=0x7f9a74139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a74139b80 0x7f9a74139f60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a700098a0 tx=0x7f9a7001c440 comp rx=0 tx=0).stop 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- 172.21.15.98:0/1533767653 shutdown_connections 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9a740724b0 0x7f9a740a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f9a7406fce0 0x7f9a740a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 --2- 172.21.15.98:0/1533767653 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a74139b80 0x7f9a74139f60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- 172.21.15.98:0/1533767653 >> 172.21.15.98:0/1533767653 conn(0x7f9a740ac6b0 msgr2=0x7f9a740a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- 172.21.15.98:0/1533767653 shutdown_connections 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- 172.21.15.98:0/1533767653 wait complete. 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient: build_initial_monmap 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient: monmap: 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:22.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 librados: starting msgr at 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 librados: starting objecter 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 Processor -- start 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- start start 2024-09-11T03:01:22.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 librados: setting wanted keys 2024-09-11T03:01:22.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 librados: calling monclient init 2024-09-11T03:01:22.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient: init 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient: _add_conns ranks=[2,7,1,5,0,3,4,6,8] 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a7406fce0 0x7f9a74134ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): picked mon.noname-b con 0x7f9a7406fce0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a740724b0 0x7f9a74135000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): picked mon.noname-i con 0x7f9a740724b0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a74139b80 0x7f9a7413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): picked mon.noname-f con 0x7f9a74139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9a74143440 con 0x7f9a74139b80 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9a740a88a0 con 0x7f9a7406fce0 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f9a741435c0 con 0x7f9a740724b0 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a7b517640 10 monclient(hunting): authenticate will time out at 1636.610985s 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a7406fce0 0x7f9a74134ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a7406fce0 0x7f9a74134ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:34776/0 (socket says 172.21.15.98:34776) 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 -- 172.21.15.98:0/131708827 learned_addr learned my addr 172.21.15.98:0/131708827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): get_auth_request con 0x7f9a7406fce0 auth_method 0 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a79ab4640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a74139b80 0x7f9a7413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a79ab4640 10 monclient(hunting): get_auth_request con 0x7f9a74139b80 auth_method 0 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a79ab4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a79ab4640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a79ab4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient(hunting): handle_auth_done global_id 64514 payload 824 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a740724b0 msgr2=0x7f9a74135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a78ab2640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a740724b0 0x7f9a74135000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9a740724b0 0x7f9a74135000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a74139b80 msgr2=0x7f9a7413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a74139b80 0x7f9a7413a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient: _finish_hunting 0 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 monclient: found mon.noname-b 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 -- 172.21.15.98:0/131708827 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7413de20 con 0x7f9a7406fce0 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a792b3640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a7406fce0 0x7f9a74134ac0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a700b0040 tx=0x7f9a700191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a69ffb640 10 client.?.objecter ms_handle_connect 0x7f9a7406fce0 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a69ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a700b2040 con 0x7f9a7406fce0 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.728+0000 7f9a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: dump: 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.750 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: _finish_auth 0 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a700bd070 con 0x7f9a7406fce0 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 5 monclient: authenticate success, global_id 64514 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 10 monclient: _renew_subs 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:22.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7413e100 con 0x7f9a7406fce0 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a700abe10 con 0x7f9a7406fce0 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: dump: 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a6b7fe640 4 set_mon_vals no callback set 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 10 client.64514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 10 monclient: _renew_subs 2024-09-11T03:01:22.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a74135540 con 0x7f9a7406fce0 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 1 librados: init done 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413cbe0 0x7f9a7413ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7406ca10 con 0x7f9a7413cbe0 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413cbe0 0x7f9a7413ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient: get_auth_request con 0x7f9a7413cbe0 auth_method 0 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9a70004100 con 0x7f9a7406fce0 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_done global_id 44838 payload 824 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 1 -- 172.21.15.98:0/131708827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a7413e500 con 0x7f9a7413cbe0 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a5803df40 0x7f9a58040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(438..438 src has 1..438) ==== 6202+0+0 (secure 0 0 0) 0x7f9a700e9ae0 con 0x7f9a7406fce0 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 client.64514.objecter ms_dispatch 0x7f9a74000ba0 osd_map(438..438 src has 1..438) 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 3 client.64514.objecter handle_osd_map got epochs [438,438] > 0 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 3 client.64514.objecter handle_osd_map decoding full epoch 438 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a78ab2640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413cbe0 0x7f9a7413ad10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9a740a5e60 tx=0x7f9a64010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 20 client.64514.objecter dump_active .. 0 homeless 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 client.64514.objecter ms_handle_connect 0x7f9a7413cbe0 2024-09-11T03:01:22.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 client.64514.objecter resend_mon_ops 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a6400f040 con 0x7f9a7413cbe0 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: dump: 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a79ab4640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a5803df40 0x7f9a58040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a79ab4640 10 monclient: get_auth_request con 0x7f9a5803df40 auth_method 0 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a64010e70 con 0x7f9a7413cbe0 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:22.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413cbe0 msgr2=0x7f9a7413ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413cbe0 0x7f9a7413ad10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9a740a5e60 tx=0x7f9a64010c20 comp rx=0 tx=0).stop 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413cbe0 msgr2=0x7f9a7413ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413cbe0 0x7f9a7413ad10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a69ffb640 10 client.64514.objecter ms_handle_connect 0x7f9a5803df40 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.732+0000 7f9a79ab4640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a5803df40 0x7f9a58040400 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f9a740a9950 tx=0x7f9a6c007490 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.736+0000 7f9a7b517640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.736+0000 7f9a7b517640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413b250 0x7f9a7413b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.736+0000 7f9a7b517640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.736+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7406e870 con 0x7f9a7413b250 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413b250 0x7f9a7413b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient: get_auth_request con 0x7f9a7413b250 auth_method 0 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 10 monclient(hunting): handle_auth_done global_id 44847 payload 824 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 1 -- 172.21.15.98:0/131708827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a74073d90 con 0x7f9a7413b250 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a78ab2640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413b250 0x7f9a7413b660 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009fd0 tx=0x7f9a640046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 10 client.64514.objecter ms_handle_connect 0x7f9a7413b250 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 10 client.64514.objecter resend_mon_ops 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9a64025070 con 0x7f9a7413b250 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:22.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 10 monclient: dump: 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.760 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7f9a64021180 con 0x7f9a7413b250 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413b250 msgr2=0x7f9a7413b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413b250 0x7f9a7413b660 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009fd0 tx=0x7f9a640046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413b250 msgr2=0x7f9a7413b660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:22.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a69ffb640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a7413b250 0x7f9a7413b660 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009fd0 tx=0x7f9a640046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 309180, 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:22.762 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:22.763 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:22.764 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:22.766 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:22.767 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:22.769 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.772 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:22.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.776 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:22.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:22.780 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:22.783 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.784 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:01:22.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.786 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:22.786 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 10 librados: watch_flush enter 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 10 librados: watch_flush exit 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 20 client.64514.objecter shutdown clearing up homeless session... 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 10 client.64514.objecter successfully canceled tick 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a5803df40 msgr2=0x7f9a58040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9a5803df40 0x7f9a58040400 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f9a740a9950 tx=0x7f9a6c007490 comp rx=0 tx=0).stop 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 10 monclient: shutdown 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a7406fce0 msgr2=0x7f9a74134ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 --2- 172.21.15.98:0/131708827 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9a7406fce0 0x7f9a74134ac0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9a700b0040 tx=0x7f9a700191e0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a79ab4640 1 -- 172.21.15.98:0/131708827 reap_dead start 2024-09-11T03:01:22.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a79ab4640 1 -- 172.21.15.98:0/131708827 reap_dead start 2024-09-11T03:01:22.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 shutdown_connections 2024-09-11T03:01:22.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 >> 172.21.15.98:0/131708827 conn(0x7f9a740ac6b0 msgr2=0x7f9a74135ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:22.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 shutdown_connections 2024-09-11T03:01:22.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 -- 172.21.15.98:0/131708827 wait complete. 2024-09-11T03:01:22.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.740+0000 7f9a7b517640 1 librados: shutdown 2024-09-11T03:01:22.789 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:22.789 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-09-11T03:01:22.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient: get_monmap_and_config 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient: build_initial_monmap 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient: monmap: 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:22.905911+0000 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:22.905911+0000 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:22.906 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 1 Processor -- start 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 1 -- start start 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient: init 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient: _add_conns ranks=[2,0,7,5,3,4,6,10,1,8,9] 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e38139b10 0x7f3e38072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient(hunting): picked mon.noname-b con 0x7f3e38139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e38139ef0 addr v1:172.21.15.98:6789/0 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3e3813a2d0 0x7f3e380a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient(hunting): picked mon.noname-i con 0x7f3e3813a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:22.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e380702b0 con 0x7f3e38139ef0 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3e38071bf0 con 0x7f3e38139b10 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3e38071350 con 0x7f3e3813a2d0 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e4040a640 10 monclient(hunting): authenticate will time out at 1636.786782s 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e38139b10 0x7f3e38072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3d9a5640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3e38139ef0 0x7f3e380a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43896/0 (socket says 172.21.15.98:43896) 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): get_auth_request con 0x7f3e38139b10 auth_method 0 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3d9a5640 1 -- 172.21.15.98:0/687977134 learned_addr learned my addr 172.21.15.98:0/687977134 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e9a7640 1 --2- 172.21.15.98:0/687977134 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3e3813a2d0 0x7f3e380a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e9a7640 10 monclient(hunting): get_auth_request con 0x7f3e3813a2d0 auth_method 0 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e9a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e9a7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e9a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_done global_id 64523 payload 274 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 1 -- 172.21.15.98:0/687977134 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3e3813a2d0 msgr2=0x7f3e380a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 1 --2- 172.21.15.98:0/687977134 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3e3813a2d0 0x7f3e380a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 1 -- 172.21.15.98:0/687977134 >> v1:172.21.15.98:6789/0 conn(0x7f3e38139ef0 legacy=0x7f3e380a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient: _finish_hunting 0 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 1 monclient: found mon.noname-b 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.904+0000 7f3e3e1a6640 1 -- 172.21.15.98:0/687977134 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e380a9620 con 0x7f3e38139b10 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3e1a6640 1 --2- 172.21.15.98:0/687977134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e38139b10 0x7f3e38072600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009dc0 tx=0x7f3e3401c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbdf8383aeeb5e0f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 1 -- 172.21.15.98:0/687977134 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e340b2070 con 0x7f3e38139b10 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: dump: 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: _finish_auth 0 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: _check_auth_tickets 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 1 -- 172.21.15.98:0/687977134 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e340a59d0 con 0x7f3e38139b10 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 5 monclient: authenticate success, global_id 64523 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 4 set_mon_vals no callback set 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 1 -- 172.21.15.98:0/687977134 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e340a5bc0 con 0x7f3e38139b10 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e3d1a4640 10 monclient: dump: 2024-09-11T03:01:22.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.915 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.916 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 10 monclient: shutdown 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 -- 172.21.15.98:0/687977134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e38139b10 msgr2=0x7f3e38072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 --2- 172.21.15.98:0/687977134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e38139b10 0x7f3e38072600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009dc0 tx=0x7f3e3401c8e0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 -- 172.21.15.98:0/687977134 shutdown_connections 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 --2- 172.21.15.98:0/687977134 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3e3813a2d0 0x7f3e380a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 --2- 172.21.15.98:0/687977134 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e38139b10 0x7f3e38072600 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 -- 172.21.15.98:0/687977134 >> 172.21.15.98:0/687977134 conn(0x7f3e380ac790 msgr2=0x7f3e380a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 -- 172.21.15.98:0/687977134 shutdown_connections 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 -- 172.21.15.98:0/687977134 wait complete. 2024-09-11T03:01:22.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 10 monclient: build_initial_monmap 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 10 monclient: monmap: 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:22.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 librados: starting msgr at 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 librados: starting objecter 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 Processor -- start 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 -- start start 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 librados: setting wanted keys 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 1 librados: calling monclient init 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.908+0000 7f3e4040a640 10 monclient: init 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient: _add_conns ranks=[4,1,5,0,8,3,2,6,7] 2024-09-11T03:01:22.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e38139b10 0x7f3e381329e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): picked mon.noname-c con 0x7f3e38139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e38139ef0 0x7f3e38132f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): picked mon.noname-f con 0x7f3e38139ef0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e3813a2d0 0x7f3e380a3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): picked mon.noname-h con 0x7f3e3813a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3e38144470 con 0x7f3e38139ef0 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f3e380a8870 con 0x7f3e38139b10 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3e381445f0 con 0x7f3e3813a2d0 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:22.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): authenticate will time out at 1636.792121s 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e38139b10 0x7f3e381329e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e38139b10 0x7f3e381329e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39908/0 (socket says 172.21.15.98:39908) 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 -- 172.21.15.98:0/3146608184 learned_addr learned my addr 172.21.15.98:0/3146608184 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): get_auth_request con 0x7f3e38139b10 auth_method 0 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e38139ef0 0x7f3e38132f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): get_auth_request con 0x7f3e38139ef0 auth_method 0 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e9a7640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e3813a2d0 0x7f3e380a3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient(hunting): handle_auth_done global_id 24385 payload 824 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e3813a2d0 msgr2=0x7f3e380a3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e3813a2d0 0x7f3e380a3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e38139ef0 msgr2=0x7f3e38132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3e38139ef0 0x7f3e38132f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient: _finish_hunting 0 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 monclient: found mon.noname-c 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:01:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 -- 172.21.15.98:0/3146608184 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3813bbe0 con 0x7f3e38139b10 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e9a7640 10 monclient: get_auth_request con 0x7f3e3813a2d0 auth_method 0 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e9a7640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3e3813a2d0 0x7f3e380a3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e1a6640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e38139b10 0x7f3e381329e0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f3e3401d180 tx=0x7f3e34019680 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 client.?.objecter ms_handle_connect 0x7f3e38139b10 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e340b2050 con 0x7f3e38139b10 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: dump: 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.930 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: _finish_auth 0 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:22.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 5 monclient: authenticate success, global_id 24385 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e340be070 con 0x7f3e38139b10 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient: _renew_subs 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3813bec0 con 0x7f3e38139b10 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e34003bb0 con 0x7f3e38139b10 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3c9a3640 4 set_mon_vals no callback set 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 client.24385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: dump: 2024-09-11T03:01:22.932 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.933 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient: _renew_subs 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e380aa330 con 0x7f3e38139b10 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 librados: init done 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a41d0 0x7f3e380a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3e3806ca10 con 0x7f3e380a41d0 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3e34004040 con 0x7f3e38139b10 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a41d0 0x7f3e380a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient: get_auth_request con 0x7f3e380a41d0 auth_method 0 2024-09-11T03:01:22.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e1c03df40 0x7f3e1c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(438..438 src has 1..438) ==== 6202+0+0 (secure 0 0 0) 0x7f3e340ea6c0 con 0x7f3e38139b10 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 client.24385.objecter ms_dispatch 0x7f3e38000ba0 osd_map(438..438 src has 1..438) 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 3 client.24385.objecter handle_osd_map got epochs [438,438] > 0 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 3 client.24385.objecter handle_osd_map decoding full epoch 438 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_done global_id 64532 payload 824 2024-09-11T03:01:22.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 1 -- 172.21.15.98:0/3146608184 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e3806e680 con 0x7f3e380a41d0 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 20 client.24385.objecter dump_active .. 0 homeless 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e9a7640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e1c03df40 0x7f3e1c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 client.24385.objecter ms_handle_connect 0x7f3e380a41d0 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3d9a5640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a41d0 0x7f3e380a45b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3e28000f70 tx=0x7f3e2800e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 client.24385.objecter resend_mon_ops 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e9a7640 10 monclient: get_auth_request con 0x7f3e1c03df40 auth_method 0 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e280070d0 con 0x7f3e380a41d0 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: dump: 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.936 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.937 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e28018070 con 0x7f3e380a41d0 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a41d0 msgr2=0x7f3e380a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a41d0 0x7f3e380a45b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3e28000f70 tx=0x7f3e2800e940 comp rx=0 tx=0).stop 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a41d0 msgr2=0x7f3e380a45b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a41d0 0x7f3e380a45b0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3e28000f70 tx=0x7f3e2800e940 comp rx=0 tx=0).stop 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e3e9a7640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e1c03df40 0x7f3e1c040400 secure :-1 s=READY pgs=3596 cs=0 l=1 rev1=1 crypto rx=0x7f3e380a5df0 tx=0x7f3e30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.912+0000 7f3e2effd640 10 client.24385.objecter ms_handle_connect 0x7f3e1c03df40 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a4af0 0x7f3e38136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:22.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3e38141820 con 0x7f3e380a4af0 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a4af0 0x7f3e38136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient: get_auth_request con 0x7f3e380a4af0 auth_method 0 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 10 monclient(hunting): handle_auth_done global_id 64541 payload 824 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 1 -- 172.21.15.98:0/3146608184 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e38072600 con 0x7f3e380a4af0 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 10 client.24385.objecter ms_handle_connect 0x7f3e380a4af0 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 10 client.24385.objecter resend_mon_ops 2024-09-11T03:01:22.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3d9a5640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a4af0 0x7f3e38136bb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009010 tx=0x7f3e28000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3e28007540 con 0x7f3e380a4af0 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 10 monclient: dump: 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9256 (secure 0 0 0) 0x7f3e280070d0 con 0x7f3e380a4af0 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a4af0 msgr2=0x7f3e38136bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a4af0 0x7f3e38136bb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009010 tx=0x7f3e28000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a4af0 msgr2=0x7f3e38136bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e2effd640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f3e380a4af0 0x7f3e38136bb0 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009010 tx=0x7f3e28000ee0 comp rx=0 tx=0).stop 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 175113, 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:22.942 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:22.943 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:22.944 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:22.945 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:22.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.949 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.950 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.951 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.952 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.953 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.953 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.953 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.953 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:22.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:22.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.957 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.958 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.960 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:22.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:22.963 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:22.964 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:22.965 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:22.965 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:22.965 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:22.965 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:22.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:22.965 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:22.965 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 10 librados: watch_flush enter 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 10 librados: watch_flush exit 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 20 client.24385.objecter shutdown clearing up homeless session... 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 10 client.24385.objecter successfully canceled tick 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e1c03df40 msgr2=0x7f3e1c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3e1c03df40 0x7f3e1c040400 secure :-1 s=READY pgs=3596 cs=0 l=1 rev1=1 crypto rx=0x7f3e380a5df0 tx=0x7f3e30008040 comp rx=0 tx=0).stop 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 10 monclient: shutdown 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e38139b10 msgr2=0x7f3e381329e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 --2- 172.21.15.98:0/3146608184 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3e38139b10 0x7f3e381329e0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f3e3401d180 tx=0x7f3e34019680 comp rx=0 tx=0).stop 2024-09-11T03:01:22.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3e9a7640 1 -- 172.21.15.98:0/3146608184 reap_dead start 2024-09-11T03:01:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e3e9a7640 1 -- 172.21.15.98:0/3146608184 reap_dead start 2024-09-11T03:01:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 shutdown_connections 2024-09-11T03:01:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 >> 172.21.15.98:0/3146608184 conn(0x7f3e380ac790 msgr2=0x7f3e380aa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 shutdown_connections 2024-09-11T03:01:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 -- 172.21.15.98:0/3146608184 wait complete. 2024-09-11T03:01:22.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:22.920+0000 7f3e4040a640 1 librados: shutdown 2024-09-11T03:01:22.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:22.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-09-11T03:01:22.969 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient: get_monmap_and_config 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient: monmap: 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:23.085988+0000 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:23.085988+0000 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.084 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:23.085 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 Processor -- start 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 -- start start 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient: init 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient: _add_conns ranks=[6,3,10,1,0,5,9,8,2,4,7] 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 0x7f6ddc0a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): picked mon.noname-d con 0x7f6ddc0a7aa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ddc0a12a0 0x7f6ddc0a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): picked mon.noname-g con 0x7f6ddc0a12a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6ddc139850 0x7f6ddc139c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): picked mon.noname-f con 0x7f6ddc139850 addr v2:172.21.15.190:3300/0 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f6ddc0a2b30 con 0x7f6ddc139850 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc0a8900 con 0x7f6ddc0a12a0 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6ddc0a1720 con 0x7f6ddc0a7aa0 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6de0b76640 10 monclient(hunting): authenticate will time out at 1636.966248s 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dda59c640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 msgr2=0x7f6ddc0a7e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T03:01:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dda59c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 0x7f6ddc0a7e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6ddc139850 0x7f6ddc139c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ddc0a12a0 0x7f6ddc0a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ddc0a12a0 0x7f6ddc0a83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:40952/0 (socket says 172.21.15.98:40952) 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 1 -- 172.21.15.98:0/1550405383 learned_addr learned my addr 172.21.15.98:0/1550405383 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): get_auth_request con 0x7f6ddc139850 auth_method 0 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 10 monclient(hunting): get_auth_request con 0x7f6ddc0a12a0 auth_method 0 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6dd9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_done global_id 25669 payload 274 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 1 -- 172.21.15.98:0/1550405383 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 msgr2=0x7f6ddc0a7e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 1 --2- 172.21.15.98:0/1550405383 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 0x7f6ddc0a7e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 1 -- 172.21.15.98:0/1550405383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ddc0a12a0 msgr2=0x7f6ddc0a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 1 --2- 172.21.15.98:0/1550405383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ddc0a12a0 0x7f6ddc0a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 1 monclient: found mon.noname-f 2024-09-11T03:01:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:01:23.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.084+0000 7f6ddad9d640 1 -- 172.21.15.98:0/1550405383 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc0a27c0 con 0x7f6ddc139850 2024-09-11T03:01:23.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6ddad9d640 1 --2- 172.21.15.98:0/1550405383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc139c70 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd401c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4251d40b0b2a36ce server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 1 -- 172.21.15.98:0/1550405383 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd40b2070 con 0x7f6ddc139850 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: dump: 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 5 monclient: authenticate success, global_id 25669 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 1 -- 172.21.15.98:0/1550405383 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd40a5620 con 0x7f6ddc139850 2024-09-11T03:01:23.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 4 set_mon_vals no callback set 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 1 -- 172.21.15.98:0/1550405383 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd40a5cc0 con 0x7f6ddc139850 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6dd959a640 10 monclient: dump: 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.101 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 10 monclient: shutdown 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 -- 172.21.15.98:0/1550405383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 msgr2=0x7f6ddc139c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 --2- 172.21.15.98:0/1550405383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc139c70 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4009d90 tx=0x7f6dd401c850 comp rx=0 tx=0).stop 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 -- 172.21.15.98:0/1550405383 shutdown_connections 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 --2- 172.21.15.98:0/1550405383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc139c70 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 --2- 172.21.15.98:0/1550405383 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6ddc0a12a0 0x7f6ddc0a83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 --2- 172.21.15.98:0/1550405383 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 0x7f6ddc0a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 -- 172.21.15.98:0/1550405383 >> 172.21.15.98:0/1550405383 conn(0x7f6ddc0ac690 msgr2=0x7f6ddc0a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 -- 172.21.15.98:0/1550405383 shutdown_connections 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 -- 172.21.15.98:0/1550405383 wait complete. 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 10 monclient: monmap: 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:23.103 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 librados: starting msgr at 2024-09-11T03:01:23.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 librados: starting objecter 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 Processor -- start 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 -- start start 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 librados: setting wanted keys 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 1 librados: calling monclient init 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.088+0000 7f6de0b76640 10 monclient: init 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient: _add_conns ranks=[7,6,1,3,4,2,5,8,0] 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6ddc0a12a0 0x7f6ddc13bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): picked mon.noname-i con 0x7f6ddc0a12a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 0x7f6ddc13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): picked mon.noname-d con 0x7f6ddc0a7aa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): picked mon.noname-f con 0x7f6ddc139850 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc0a65d0 con 0x7f6ddc139850 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f6ddc0a6450 con 0x7f6ddc0a7aa0 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f6ddc1432c0 con 0x7f6ddc0a12a0 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): authenticate will time out at 1636.972355s 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd9d9b640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 msgr2=0x7f6ddc13dfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd9d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 0x7f6ddc13dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc136a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:59320/0 (socket says 172.21.15.98:59320) 2024-09-11T03:01:23.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 -- 172.21.15.98:0/3624550666 learned_addr learned my addr 172.21.15.98:0/3624550666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): get_auth_request con 0x7f6ddc139850 auth_method 0 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6ddc0a12a0 0x7f6ddc13bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): get_auth_request con 0x7f6ddc0a12a0 auth_method 0 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient(hunting): handle_auth_done global_id 25678 payload 824 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6ddc0a12a0 msgr2=0x7f6ddc13bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6ddc0a12a0 0x7f6ddc13bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 msgr2=0x7f6ddc13dfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f6ddc0a7aa0 0x7f6ddc13dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 monclient: found mon.noname-f 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 -- 172.21.15.98:0/3624550666 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc0a8fe0 con 0x7f6ddc139850 2024-09-11T03:01:23.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6ddad9d640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc136a60 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40b0040 tx=0x7f6dd40195f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 client.?.objecter ms_handle_connect 0x7f6ddc139850 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd40b2070 con 0x7f6ddc139850 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: dump: 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.109 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd40bc070 con 0x7f6ddc139850 2024-09-11T03:01:23.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 5 monclient: authenticate success, global_id 25678 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient: _renew_subs 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ddc13d480 con 0x7f6ddc139850 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd40a5e50 con 0x7f6ddc139850 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd8d99640 4 set_mon_vals no callback set 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: dump: 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.111 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 client.25678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient: _renew_subs 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ddc136fa0 con 0x7f6ddc139850 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 librados: init done 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1460 0x7f6ddc1d1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6ddc13f9b0 con 0x7f6ddc1d1460 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1460 0x7f6ddc1d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient: get_auth_request con 0x7f6ddc1d1460 auth_method 0 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 10 monclient(hunting): handle_auth_done global_id 24394 payload 824 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 1 -- 172.21.15.98:0/3624550666 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ddc0a5120 con 0x7f6ddc1d1460 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dda59c640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1460 0x7f6ddc1d1840 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009610 tx=0x7f6dcc009ad0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 client.25678.objecter ms_handle_connect 0x7f6ddc1d1460 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 client.25678.objecter resend_mon_ops 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6dd40a59e0 con 0x7f6ddc139850 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6dbc03e0e0 0x7f6dbc0405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(438..438 src has 1..438) ==== 6202+0+0 (secure 0 0 0) 0x7f6dd40ea110 con 0x7f6ddc139850 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 client.25678.objecter ms_dispatch 0x7f6ddc000ba0 osd_map(438..438 src has 1..438) 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 3 client.25678.objecter handle_osd_map got epochs [438,438] > 0 2024-09-11T03:01:23.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 3 client.25678.objecter handle_osd_map decoding full epoch 438 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 20 client.25678.objecter dump_active .. 0 homeless 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dcc016070 con 0x7f6ddc1d1460 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: dump: 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.115 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6dcc012a70 con 0x7f6ddc1d1460 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1460 msgr2=0x7f6ddc1d1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1460 0x7f6ddc1d1840 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009610 tx=0x7f6dcc009ad0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1460 msgr2=0x7f6ddc1d1840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd2ffd640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1460 0x7f6ddc1d1840 secure :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc009610 tx=0x7f6dcc009ad0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.092+0000 7f6dd9d9b640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6dbc03e0e0 0x7f6dbc0405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.096+0000 7f6dd9d9b640 10 monclient: get_auth_request con 0x7f6dbc03e0e0 auth_method 0 2024-09-11T03:01:23.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.096+0000 7f6dd2ffd640 10 client.25678.objecter ms_handle_connect 0x7f6dbc03e0e0 2024-09-11T03:01:23.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.096+0000 7f6dd9d9b640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6dbc03e0e0 0x7f6dbc0405a0 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc13d1e0 tx=0x7f6dc4009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1d80 0x7f6ddc137a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6ddc06ca10 con 0x7f6ddc1d1d80 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1d80 0x7f6ddc137a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient: get_auth_request con 0x7f6ddc1d1d80 auth_method 0 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 10 monclient(hunting): handle_auth_done global_id 24403 payload 824 2024-09-11T03:01:23.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 1 -- 172.21.15.98:0/3624550666 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ddc0abed0 con 0x7f6ddc1d1d80 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 10 client.25678.objecter ms_handle_connect 0x7f6ddc1d1d80 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 10 client.25678.objecter resend_mon_ops 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dda59c640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1d80 0x7f6ddc137a40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc005900 tx=0x7f6dcc0059e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dcc016040 con 0x7f6ddc1d1d80 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 10 monclient: dump: 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.120 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7f6dcc01c950 con 0x7f6ddc1d1d80 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:23.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1d80 msgr2=0x7f6ddc137a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1d80 0x7f6ddc137a40 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc005900 tx=0x7f6dcc0059e0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1d80 msgr2=0x7f6ddc137a40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6dd2ffd640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6ddc1d1d80 0x7f6ddc137a40 secure :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc005900 tx=0x7f6dcc0059e0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:23.122 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 698885, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:23.123 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:23.124 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:23.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.127 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.128 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.129 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.130 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:23.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.132 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.134 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.135 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.136 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.137 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:23.138 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.141 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.142 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.143 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.144 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:23.145 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:23.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 10 librados: watch_flush enter 2024-09-11T03:01:23.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 10 librados: watch_flush exit 2024-09-11T03:01:23.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 20 client.25678.objecter shutdown clearing up homeless session... 2024-09-11T03:01:23.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 10 client.25678.objecter successfully canceled tick 2024-09-11T03:01:23.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6dbc03e0e0 msgr2=0x7f6dbc0405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6dbc03e0e0 0x7f6dbc0405a0 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc13d1e0 tx=0x7f6dc4009240 comp rx=0 tx=0).stop 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 10 monclient: shutdown 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 msgr2=0x7f6ddc136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 --2- 172.21.15.98:0/3624550666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ddc139850 0x7f6ddc136a60 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f6dd40b0040 tx=0x7f6dd40195f0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6ddad9d640 1 -- 172.21.15.98:0/3624550666 reap_dead start 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6ddad9d640 1 -- 172.21.15.98:0/3624550666 reap_dead start 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 shutdown_connections 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 >> 172.21.15.98:0/3624550666 conn(0x7f6ddc0ac690 msgr2=0x7f6ddc0aa2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 shutdown_connections 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 -- 172.21.15.98:0/3624550666 wait complete. 2024-09-11T03:01:23.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.100+0000 7f6de0b76640 1 librados: shutdown 2024-09-11T03:01:23.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:23.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-09-11T03:01:23.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:01:23.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient: get_monmap_and_config 2024-09-11T03:01:23.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient: monmap: 2024-09-11T03:01:23.275 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.275 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:23.277451+0000 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:23.277451+0000 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.276 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.277 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.277 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:23.277 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:23.277 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 1 Processor -- start 2024-09-11T03:01:23.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 1 -- start start 2024-09-11T03:01:23.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient: init 2024-09-11T03:01:23.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient: _add_conns ranks=[5,1,0,2,6,7,10,8,9,4,3] 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcfd80a3de0 0x7fcfd80a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient(hunting): picked mon.noname-h con 0x7fcfd80a3de0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcfd80a41c0 addr v1:172.21.15.190:6789/0 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcfd813c9f0 addr v1:172.21.15.98:6789/0 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd8068fa0 con 0x7fcfd813c9f0 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd8068b40 con 0x7fcfd80a41c0 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcfd80a0170 con 0x7fcfd80a3de0 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 10 monclient(hunting): authenticate will time out at 1637.157696s 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcfd813c9f0 0x7fcfd813ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43908/0 (socket says 172.21.15.98:43908) 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd7fff640 1 -- 172.21.15.98:0/4178751478 learned_addr learned my addr 172.21.15.98:0/4178751478 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:01:23.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcfd80a3de0 0x7fcfd80a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request con 0x7fcfd80a3de0 auth_method 0 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1020588418 0 0) 0x7fcfd8068fa0 con 0x7fcfd813c9f0 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): _negotiate 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): my global_id is 44865 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfb8003530 con 0x7fcfd813c9f0 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 983075070 0 0) 0x7fcfb8003530 con 0x7fcfd813c9f0 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcfd80a3de0 msgr2=0x7fcfd80a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcfd80a3de0 0x7fcfd80a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 >> v1:172.21.15.190:6789/0 conn(0x7fcfd80a41c0 legacy=0x7fcfd80a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd809eaf0 con 0x7fcfd813c9f0 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fcfd0001cf0 con 0x7fcfd813c9f0 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 5 monclient: authenticate success, global_id 44865 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: dump: 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 >> v1:172.21.15.98:6789/0 conn(0x7fcfd813c9f0 legacy=0x7fcfd813ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient: _add_conns ranks=[1,2,7,8,3,4,0,6,5] 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb8004240 0x7fcfb8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): picked mon.f con 0x7fcfb8004240 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfb8004da0 0x7fcfb8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): picked mon.b con 0x7fcfb8004da0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8009c40 0x7fcfb800c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): picked mon.i con 0x7fcfb8009c40 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd80a0170 con 0x7fcfb8004240 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb800c990 con 0x7fcfb8004da0 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fcfb800cb40 con 0x7fcfb8009c40 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfdda99640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfb8004da0 0x7fcfb8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): get_auth_request con 0x7fcfb8004da0 auth_method 0 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb8004240 0x7fcfb8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request con 0x7fcfb8004240 auth_method 0 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd7fff640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8009c40 0x7fcfb800c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient(hunting): handle_auth_done global_id 44865 payload 293 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 1 -- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8009c40 msgr2=0x7fcfb800c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8009c40 0x7fcfb800c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 1 -- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb8004240 msgr2=0x7fcfb8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfb8004240 0x7fcfb8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 1 monclient: found mon.b 2024-09-11T03:01:23.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 1 -- 172.21.15.98:0/4178751478 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb8003530 con 0x7fcfb8004da0 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.276+0000 7fcfd7fff640 1 -- 172.21.15.98:0/4178751478 reap_dead start 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfcffff640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfb8004da0 0x7fcfb8005220 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0001810 tx=0x7fcfc001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a04cc6a71fdb92e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc00b0070 con 0x7fcfb8004da0 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 10 monclient: dump: 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.295 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.296 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc00aa810 con 0x7fcfb8004da0 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 4 set_mon_vals no callback set 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 1 -- 172.21.15.98:0/4178751478 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc0001460 con 0x7fcfb8004da0 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd6ffd640 10 monclient: dump: 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.297 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.298 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient: shutdown 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- 172.21.15.98:0/4178751478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfb8004da0 msgr2=0x7fcfb8005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfb8004da0 0x7fcfb8005220 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0001810 tx=0x7fcfc001c440 comp rx=0 tx=0).stop 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- 172.21.15.98:0/4178751478 shutdown_connections 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 --2- 172.21.15.98:0/4178751478 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcfb8004da0 0x7fcfb8005220 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- 172.21.15.98:0/4178751478 >> 172.21.15.98:0/4178751478 conn(0x7fcfd80a4ed0 msgr2=0x7fcfd80a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- 172.21.15.98:0/4178751478 shutdown_connections 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- 172.21.15.98:0/4178751478 wait complete. 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient: monmap: 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:23.299 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.300 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 librados: starting msgr at 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 librados: starting objecter 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 Processor -- start 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- start start 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 librados: setting wanted keys 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 librados: calling monclient init 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient: init 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient: _add_conns ranks=[7,3,0,5,1,2,4,6,8] 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8004da0 0x7fcfd80727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): picked mon.noname-i con 0x7fcfb8004da0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:23.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfd8072d00 0x7fcfd8073120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): picked mon.noname-g con 0x7fcfd8072d00 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81d0f50 0x7fcfd81d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): picked mon.noname-a con 0x7fcfd81d0f50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd809b5d0 con 0x7fcfd81d0f50 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd809b450 con 0x7fcfd8072d00 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fcfd809b750 con 0x7fcfb8004da0 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfdda99640 10 monclient(hunting): authenticate will time out at 1637.163575s 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81d0f50 0x7fcfd81d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81d0f50 0x7fcfd81d13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39100/0 (socket says 172.21.15.98:39100) 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 1 -- 172.21.15.98:0/850297568 learned_addr learned my addr 172.21.15.98:0/850297568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 10 monclient(hunting): get_auth_request con 0x7fcfd81d0f50 auth_method 0 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8004da0 0x7fcfd80727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request con 0x7fcfb8004da0 auth_method 0 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfcffff640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfd8072d00 0x7fcfd8073120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfcffff640 10 monclient(hunting): get_auth_request con 0x7fcfd8072d00 auth_method 0 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfcffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.280+0000 7fcfcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 10 monclient(hunting): handle_auth_done global_id 44874 payload 824 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8004da0 msgr2=0x7fcfd80727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8004da0 0x7fcfd80727c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfd8072d00 msgr2=0x7fcfd8073120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcfd8072d00 0x7fcfd8073120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 1 monclient: found mon.noname-a 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 1 -- 172.21.15.98:0/850297568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd8097dd0 con 0x7fcfd81d0f50 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcfb8004da0 0x7fcfd80727c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd7fff640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81d0f50 0x7fcfd81d13d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80a6260 tx=0x7fcfd000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 client.?.objecter ms_handle_connect 0x7fcfd81d0f50 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfd000bb80 con 0x7fcfd81d0f50 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: dump: 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd000bd20 con 0x7fcfd81d0f50 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 5 monclient: authenticate success, global_id 44874 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 10 monclient: _renew_subs 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd809a310 con 0x7fcfd81d0f50 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfd0007580 con 0x7fcfd81d0f50 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:23.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: dump: 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.308 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 10 client.44874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 10 monclient: _renew_subs 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd67fc640 4 set_mon_vals no callback set 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd809bf60 con 0x7fcfd81d0f50 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 1 librados: init done 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd81d1910 0x7fcfd80a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcfd809c850 con 0x7fcfd81d1910 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd81d1910 0x7fcfd80a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient: get_auth_request con 0x7fcfd81d1910 auth_method 0 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fcfd0007110 con 0x7fcfd81d0f50 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_done global_id 24866 payload 824 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcfbc03df70 0x7fcfbc040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 1 -- 172.21.15.98:0/850297568 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfd80a3590 con 0x7fcfd81d1910 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(438..438 src has 1..438) ==== 6202+0+0 (secure 0 0 0) 0x7fcfd004fd40 con 0x7fcfd81d0f50 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 client.44874.objecter ms_dispatch 0x7fcfd8000ba0 osd_map(438..438 src has 1..438) 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 3 client.44874.objecter handle_osd_map got epochs [438,438] > 0 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 3 client.44874.objecter handle_osd_map decoding full epoch 438 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd81d1910 0x7fcfd80a6290 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80138b0 tx=0x7fcfc8013d70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 20 client.44874.objecter dump_active .. 0 homeless 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 client.44874.objecter ms_handle_connect 0x7fcfd81d1910 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 client.44874.objecter resend_mon_ops 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc801c070 con 0x7fcfd81d1910 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfcffff640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcfbc03df70 0x7fcfbc040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:23.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: dump: 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.312 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfcffff640 10 monclient: get_auth_request con 0x7fcfbc03df70 auth_method 0 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcfc8009e00 con 0x7fcfd81d1910 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd81d1910 msgr2=0x7fcfd80a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd81d1910 0x7fcfd80a6290 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80138b0 tx=0x7fcfc8013d70 comp rx=0 tx=0).stop 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd81d1910 msgr2=0x7fcfd80a6290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd81d1910 0x7fcfd80a6290 secure :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80138b0 tx=0x7fcfc8013d70 comp rx=0 tx=0).stop 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfcffff640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcfbc03df70 0x7fcfbc040430 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00ab000 tx=0x7fcfc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.284+0000 7fcfd4ff9640 10 client.44874.objecter ms_handle_connect 0x7fcfbc03df70 2024-09-11T03:01:23.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:23.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd80a67d0 0x7fcfd80a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcfd806ca10 con 0x7fcfd80a67d0 2024-09-11T03:01:23.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd80a67d0 0x7fcfd80a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient: get_auth_request con 0x7fcfd80a67d0 auth_method 0 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 10 monclient(hunting): handle_auth_done global_id 24875 payload 824 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 1 -- 172.21.15.98:0/850297568 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcfd809bae0 con 0x7fcfd80a67d0 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd77fe640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd80a67d0 0x7fcfd80a6be0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000c10 tx=0x7fcfc8004740 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 10 client.44874.objecter ms_handle_connect 0x7fcfd80a67d0 2024-09-11T03:01:23.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 10 client.44874.objecter resend_mon_ops 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcfc8006800 con 0x7fcfd80a67d0 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 10 monclient: dump: 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.317 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7fcfc8006390 con 0x7fcfd80a67d0 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd80a67d0 msgr2=0x7fcfd80a6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd80a67d0 0x7fcfd80a6be0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000c10 tx=0x7fcfc8004740 comp rx=0 tx=0).stop 2024-09-11T03:01:23.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd80a67d0 msgr2=0x7fcfd80a6be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd4ff9640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcfd80a67d0 0x7fcfd80a6be0 secure :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8000c10 tx=0x7fcfc8004740 comp rx=0 tx=0).stop 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 699085, 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:23.319 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.320 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.321 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:23.322 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:23.323 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.324 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:23.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:23.328 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.331 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:23.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.336 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:23.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.340 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:23.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:23.342 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 10 librados: watch_flush enter 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 10 librados: watch_flush exit 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 20 client.44874.objecter shutdown clearing up homeless session... 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 10 client.44874.objecter successfully canceled tick 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcfbc03df70 msgr2=0x7fcfbc040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcfbc03df70 0x7fcfbc040430 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7fcfc00ab000 tx=0x7fcfc0009040 comp rx=0 tx=0).stop 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 10 monclient: shutdown 2024-09-11T03:01:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81d0f50 msgr2=0x7fcfd81d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 --2- 172.21.15.98:0/850297568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd81d0f50 0x7fcfd81d13d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcfd80a6260 tx=0x7fcfd000b4a0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd7fff640 1 -- 172.21.15.98:0/850297568 reap_dead start 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfd7fff640 1 -- 172.21.15.98:0/850297568 reap_dead start 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 shutdown_connections 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 >> 172.21.15.98:0/850297568 conn(0x7fcfd80a4ed0 msgr2=0x7fcfd8098750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 shutdown_connections 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 -- 172.21.15.98:0/850297568 wait complete. 2024-09-11T03:01:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.292+0000 7fcfdda99640 1 librados: shutdown 2024-09-11T03:01:23.345 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:23.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-09-11T03:01:23.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient: get_monmap_and_config 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient: monmap: 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:23.469863+0000 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:23.469863+0000 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:23.468 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:23.469 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 Processor -- start 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 -- start start 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient: init 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient: _add_conns ranks=[4,5,9,8,10,2,3,0,6,1,7] 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd044139b10 0x7fd044072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd044139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd044139ef0 0x7fd0440a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): picked mon.noname-h con 0x7fd044139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd04413a2d0 0x7fd0440a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): picked mon.noname-a con 0x7fd04413a2d0 addr v2:172.21.15.98:3300/0 2024-09-11T03:01:23.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fd0440702b0 con 0x7fd04413a2d0 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd044071bf0 con 0x7fd044139b10 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd044071350 con 0x7fd044139ef0 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04c6f1640 10 monclient(hunting): authenticate will time out at 1637.350229s 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd044139b10 0x7fd044072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd04413a2d0 0x7fd0440a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fd04413a2d0 0x7fd0440a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39106/0 (socket says 172.21.15.98:39106) 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd044139b10 0x7fd044072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39926/0 (socket says 172.21.15.98:39926) 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 -- 172.21.15.98:0/2303918241 learned_addr learned my addr 172.21.15.98:0/2303918241 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): get_auth_request con 0x7fd04413a2d0 auth_method 0 2024-09-11T03:01:23.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 10 monclient(hunting): get_auth_request con 0x7fd044139b10 auth_method 0 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient(hunting): handle_auth_done global_id 44883 payload 274 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 -- 172.21.15.98:0/2303918241 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd044139ef0 msgr2=0x7fd0440a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 --2- 172.21.15.98:0/2303918241 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd044139ef0 0x7fd0440a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 -- 172.21.15.98:0/2303918241 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd044139b10 msgr2=0x7fd044072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 --2- 172.21.15.98:0/2303918241 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd044139b10 0x7fd044072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 monclient: found mon.noname-a 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.468+0000 7fd04ac8e640 1 -- 172.21.15.98:0/2303918241 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0440a9620 con 0x7fd04413a2d0 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 1 --2- 172.21.15.98:0/2303918241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04413a2d0 0x7fd0440a7f50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009e80 tx=0x7fd03c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f61ba0cb496fc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 1 -- 172.21.15.98:0/2303918241 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd03c0090d0 con 0x7fd04413a2d0 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: dump: 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.477 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 1 -- 172.21.15.98:0/2303918241 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd03c0a5980 con 0x7fd04413a2d0 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 5 monclient: authenticate success, global_id 44883 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 1 -- 172.21.15.98:0/2303918241 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd03c0ad3f0 con 0x7fd04413a2d0 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 4 set_mon_vals no callback set 2024-09-11T03:01:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04948b640 10 monclient: dump: 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient: shutdown 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2303918241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04413a2d0 msgr2=0x7fd0440a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2303918241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04413a2d0 0x7fd0440a7f50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009e80 tx=0x7fd03c01c8d0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2303918241 shutdown_connections 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2303918241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd04413a2d0 0x7fd0440a7f50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2303918241 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd044139ef0 0x7fd0440a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2303918241 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd044139b10 0x7fd044072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2303918241 >> 172.21.15.98:0/2303918241 conn(0x7fd0440ac790 msgr2=0x7fd0440a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2303918241 shutdown_connections 2024-09-11T03:01:23.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2303918241 wait complete. 2024-09-11T03:01:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient: monmap: 2024-09-11T03:01:23.488 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.489 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 librados: starting msgr at 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 librados: starting objecter 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 Processor -- start 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- start start 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 librados: setting wanted keys 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 librados: calling monclient init 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient: init 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient: _add_conns ranks=[7,8,4,0,5,2,1,3,6] 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd044139b10 0x7fd0441319d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): picked mon.noname-i con 0x7fd044139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd044139ef0 0x7fd044131f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): picked mon.noname-e con 0x7fd044139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd04413a2d0 0x7fd0440a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): picked mon.noname-c con 0x7fd04413a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd044143460 con 0x7fd04413a2d0 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fd0440a88a0 con 0x7fd044139b10 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd0441435e0 con 0x7fd044139ef0 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04c6f1640 10 monclient(hunting): authenticate will time out at 1637.355227s 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd04413a2d0 0x7fd0440a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd044139ef0 0x7fd044131f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd04413a2d0 0x7fd0440a2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:39938/0 (socket says 172.21.15.98:39938) 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 1 -- 172.21.15.98:0/2771560804 learned_addr learned my addr 172.21.15.98:0/2771560804 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd044139ef0 0x7fd044131f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:36144/0 (socket says 172.21.15.98:36144) 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 10 monclient(hunting): get_auth_request con 0x7fd04413a2d0 auth_method 0 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): get_auth_request con 0x7fd044139ef0 auth_method 0 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04a48d640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd044139b10 0x7fd0441319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04a48d640 10 monclient(hunting): get_auth_request con 0x7fd044139b10 auth_method 0 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04a48d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04a48d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04a48d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient(hunting): handle_auth_done global_id 24884 payload 824 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd044139b10 msgr2=0x7fd0441319d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd044139b10 0x7fd0441319d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd04413a2d0 msgr2=0x7fd0440a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd04413a2d0 0x7fd0440a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 monclient: found mon.noname-e 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 -- 172.21.15.98:0/2771560804 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04413ad10 con 0x7fd044139ef0 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd04ac8e640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd04413a2d0 0x7fd0440a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd049c8c640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd044139ef0 0x7fd044131f10 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fd03400b3f0 tx=0x7fd03400b8b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd03affd640 10 client.?.objecter ms_handle_connect 0x7fd044139ef0 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.472+0000 7fd03affd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:23.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd034015020 con 0x7fd044139ef0 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: dump: 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.495 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd034011aa0 con 0x7fd044139ef0 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 5 monclient: authenticate success, global_id 24884 2024-09-11T03:01:23.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd034011c50 con 0x7fd044139ef0 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: dump: 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd048c8a640 4 set_mon_vals no callback set 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 10 monclient: _renew_subs 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04413aff0 con 0x7fd044139ef0 2024-09-11T03:01:23.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 10 client.24884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 10 monclient: _renew_subs 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0440a9f70 con 0x7fd044139ef0 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 1 librados: init done 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3050 0x7fd0440a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd04406ca10 con 0x7fd0440a3050 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd034011630 con 0x7fd044139ef0 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3050 0x7fd0440a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd02803df40 0x7fd028040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient: get_auth_request con 0x7fd0440a3050 auth_method 0 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(439..439 src has 1..439) ==== 6202+0+0 (secure 0 0 0) 0x7fd03404f790 con 0x7fd044139ef0 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 client.24884.objecter ms_dispatch 0x7fd044000ba0 osd_map(439..439 src has 1..439) 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 3 client.24884.objecter handle_osd_map got epochs [439,439] > 0 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 3 client.24884.objecter handle_osd_map decoding full epoch 439 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 20 client.24884.objecter dump_active .. 0 homeless 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04ac8e640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd02803df40 0x7fd028040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04ac8e640 10 monclient: get_auth_request con 0x7fd02803df40 auth_method 0 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 10 monclient(hunting): handle_auth_done global_id 25696 payload 824 2024-09-11T03:01:23.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 1 -- 172.21.15.98:0/2771560804 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd04406e680 con 0x7fd0440a3050 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 client.24884.objecter ms_handle_connect 0x7fd0440a3050 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 client.24884.objecter resend_mon_ops 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04a48d640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3050 0x7fd0440a3430 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fd040011600 tx=0x7fd040011ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd0400044a0 con 0x7fd0440a3050 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: dump: 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.501 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 client.24884.objecter ms_handle_connect 0x7fd02803df40 2024-09-11T03:01:23.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd04ac8e640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd02803df40 0x7fd028040400 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7fd03c01d170 tx=0x7fd03c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd040004030 con 0x7fd0440a3050 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3050 msgr2=0x7fd0440a3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3050 0x7fd0440a3430 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fd040011600 tx=0x7fd040011ac0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3050 msgr2=0x7fd0440a3430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.476+0000 7fd03affd640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3050 0x7fd0440a3430 secure :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fd040011600 tx=0x7fd040011ac0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04c6f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3970 0x7fd044137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04c6f1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd04406e870 con 0x7fd0440a3970 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3970 0x7fd044137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient: get_auth_request con 0x7fd0440a3970 auth_method 0 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 10 monclient(hunting): handle_auth_done global_id 25705 payload 824 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 1 -- 172.21.15.98:0/2771560804 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd044135960 con 0x7fd0440a3970 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd04a48d640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3970 0x7fd044137b80 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fd040011600 tx=0x7fd0400046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 10 client.24884.objecter ms_handle_connect 0x7fd0440a3970 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 10 client.24884.objecter resend_mon_ops 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd04000fe30 con 0x7fd0440a3970 2024-09-11T03:01:23.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 10 monclient: dump: 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9423 (secure 0 0 0) 0x7fd04000fe30 con 0x7fd0440a3970 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3970 msgr2=0x7fd044137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3970 0x7fd044137b80 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7fd040011600 tx=0x7fd0400046d0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3970 msgr2=0x7fd044137b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.484+0000 7fd03affd640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0440a3970 0x7fd044137b80 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 699257, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:23.507 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.508 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.509 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:23.510 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:23.511 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:23.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:23.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.515 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.516 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:23.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.523 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.525 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.526 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.529 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:23.530 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 10 librados: watch_flush enter 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 10 librados: watch_flush exit 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 20 client.24884.objecter shutdown clearing up homeless session... 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 10 client.24884.objecter successfully canceled tick 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd02803df40 msgr2=0x7fd028040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd02803df40 0x7fd028040400 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7fd03c01d170 tx=0x7fd03c0a7070 comp rx=0 tx=0).stop 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 10 monclient: shutdown 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd044139ef0 msgr2=0x7fd044131f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 --2- 172.21.15.98:0/2771560804 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd044139ef0 0x7fd044131f10 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fd03400b3f0 tx=0x7fd03400b8b0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04ac8e640 1 -- 172.21.15.98:0/2771560804 reap_dead start 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04ac8e640 1 -- 172.21.15.98:0/2771560804 reap_dead start 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 shutdown_connections 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 >> 172.21.15.98:0/2771560804 conn(0x7fd0440ac790 msgr2=0x7fd0440aa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 shutdown_connections 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 -- 172.21.15.98:0/2771560804 wait complete. 2024-09-11T03:01:23.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.488+0000 7fd04c6f1640 1 librados: shutdown 2024-09-11T03:01:23.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:23.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-09-11T03:01:23.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.652+0000 7f0ed7828640 10 monclient: get_monmap_and_config 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.652+0000 7f0ed7828640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.652+0000 7f0ed7828640 10 monclient: monmap: 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:23.657046+0000 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:23.657046+0000 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.655 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:23.656 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 Processor -- start 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 -- start start 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient: init 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient: _add_conns ranks=[5,9,6,10,4,1,8,7,2,3,0] 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed0139b80 0x7f0ed0139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): picked mon.noname-h con 0x7f0ed0139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0ed006fce0 0x7f0ed00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed006fce0 addr v2:172.21.15.98:3300/0 2024-09-11T03:01:23.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ed00724b0 0x7f0ed00a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): picked mon.noname-d con 0x7f0ed00724b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0ed0071850 con 0x7f0ed006fce0 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f0ed0071aa0 con 0x7f0ed0139b80 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0ed0070160 con 0x7f0ed00724b0 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed7828640 10 monclient(hunting): authenticate will time out at 1637.537313s 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed5dc5640 1 -- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ed00724b0 msgr2=0x7f0ed00a7f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3303/0 2024-09-11T03:01:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed5dc5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ed00724b0 0x7f0ed00a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0ed006fce0 0x7f0ed00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0ed006fce0 0x7f0ed00a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39120/0 (socket says 172.21.15.98:39120) 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 -- 172.21.15.98:0/3591124233 learned_addr learned my addr 172.21.15.98:0/3591124233 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): get_auth_request con 0x7f0ed006fce0 auth_method 0 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed55c4640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed0139b80 0x7f0ed0139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed55c4640 10 monclient(hunting): get_auth_request con 0x7f0ed0139b80 auth_method 0 2024-09-11T03:01:23.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed55c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed55c4640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed55c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_done global_id 44892 payload 274 2024-09-11T03:01:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 -- 172.21.15.98:0/3591124233 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ed00724b0 msgr2=0x7f0ed00a7f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:01:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ed00724b0 0x7f0ed00a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 -- 172.21.15.98:0/3591124233 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed0139b80 msgr2=0x7f0ed0139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed0139b80 0x7f0ed0139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 monclient: found mon.noname-a 2024-09-11T03:01:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.656+0000 7f0ed4dc3640 1 -- 172.21.15.98:0/3591124233 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed00a94a0 con 0x7f0ed006fce0 2024-09-11T03:01:23.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed006fce0 0x7f0ed00a7a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1172d52f4e4941c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 1 -- 172.21.15.98:0/3591124233 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ecc0b1070 con 0x7f0ed006fce0 2024-09-11T03:01:23.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: dump: 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.671 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 5 monclient: authenticate success, global_id 44892 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 1 -- 172.21.15.98:0/3591124233 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ecc0ab830 con 0x7f0ed006fce0 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:23.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 4 set_mon_vals no callback set 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 1 -- 172.21.15.98:0/3591124233 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ecc0aa320 con 0x7f0ed006fce0 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ec7fff640 10 monclient: dump: 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.673 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient: shutdown 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- 172.21.15.98:0/3591124233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed006fce0 msgr2=0x7f0ed00a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed006fce0 0x7f0ed00a7a10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc01c420 comp rx=0 tx=0).stop 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- 172.21.15.98:0/3591124233 shutdown_connections 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ed00724b0 0x7f0ed00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ed006fce0 0x7f0ed00a7a10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 --2- 172.21.15.98:0/3591124233 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed0139b80 0x7f0ed0139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- 172.21.15.98:0/3591124233 >> 172.21.15.98:0/3591124233 conn(0x7f0ed00ac6b0 msgr2=0x7f0ed00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- 172.21.15.98:0/3591124233 shutdown_connections 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- 172.21.15.98:0/3591124233 wait complete. 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient: monmap: 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:23.675 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 librados: starting msgr at 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 librados: starting objecter 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 Processor -- start 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- start start 2024-09-11T03:01:23.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 librados: setting wanted keys 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 librados: calling monclient init 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient: init 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient: _add_conns ranks=[3,5,7,6,0,4,1,8,2] 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed006fce0 0x7f0ed00a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): picked mon.noname-g con 0x7f0ed006fce0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed00724b0 0x7f0ed013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): picked mon.noname-h con 0x7f0ed00724b0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ed0139b80 0x7f0ed013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): picked mon.noname-i con 0x7f0ed0139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed0143440 con 0x7f0ed006fce0 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f0ed00a88a0 con 0x7f0ed00724b0 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0ed01435c0 con 0x7f0ed0139b80 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed7828640 10 monclient(hunting): authenticate will time out at 1637.543359s 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ed0139b80 0x7f0ed013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed006fce0 0x7f0ed00a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed4dc3640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed00724b0 0x7f0ed013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed006fce0 0x7f0ed00a48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:40976/0 (socket says 172.21.15.98:40976) 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 1 -- 172.21.15.98:0/61571417 learned_addr learned my addr 172.21.15.98:0/61571417 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 10 monclient(hunting): get_auth_request con 0x7f0ed006fce0 auth_method 0 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 10 monclient(hunting): get_auth_request con 0x7f0ed0139b80 auth_method 0 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed4dc3640 10 monclient(hunting): get_auth_request con 0x7f0ed00724b0 auth_method 0 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed4dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed55c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.660+0000 7f0ed5dc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 10 monclient(hunting): handle_auth_done global_id 74217 payload 824 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ed0139b80 msgr2=0x7f0ed013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ed0139b80 0x7f0ed013ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed00724b0 msgr2=0x7f0ed013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed00724b0 0x7f0ed013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 1 monclient: found mon.noname-g 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 1 -- 172.21.15.98:0/61571417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed00a5200 con 0x7f0ed006fce0 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0ed00724b0 0x7f0ed013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed55c4640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed006fce0 0x7f0ed00a48b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000d5f0 tx=0x7f0ec000dab0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 client.?.objecter ms_handle_connect 0x7f0ed006fce0 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec0012a60 con 0x7f0ed006fce0 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: dump: 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.681 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec0019070 con 0x7f0ed006fce0 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 5 monclient: authenticate success, global_id 74217 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 10 monclient: _renew_subs 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed0135a10 con 0x7f0ed006fce0 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec0012c20 con 0x7f0ed006fce0 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec6ffd640 4 set_mon_vals no callback set 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 10 client.74217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:23.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: dump: 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.684 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 10 monclient: _renew_subs 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed00a7890 con 0x7f0ed006fce0 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 1 librados: init done 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed013cb40 0x7f0ed002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed006ca10 con 0x7f0ed013cb40 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed013cb40 0x7f0ed002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient: get_auth_request con 0x7f0ed013cb40 auth_method 0 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0ec0013070 con 0x7f0ed006fce0 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0eb403df40 0x7f0eb4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_done global_id 74226 payload 824 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(439..439 src has 1..439) ==== 6202+0+0 (secure 0 0 0) 0x7f0ec000f040 con 0x7f0ed006fce0 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 client.74217.objecter ms_dispatch 0x7f0ed0000ba0 osd_map(439..439 src has 1..439) 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 3 client.74217.objecter handle_osd_map got epochs [439,439] > 0 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 3 client.74217.objecter handle_osd_map decoding full epoch 439 2024-09-11T03:01:23.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 1 -- 172.21.15.98:0/61571417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ed013c7b0 con 0x7f0ed013cb40 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 20 client.74217.objecter dump_active .. 0 homeless 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed5dc5640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0eb403df40 0x7f0eb4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 client.74217.objecter ms_handle_connect 0x7f0ed013cb40 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed013cb40 0x7f0ed002d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0ed006f640 tx=0x7f0ecc0191c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 client.74217.objecter resend_mon_ops 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ed5dc5640 10 monclient: get_auth_request con 0x7f0eb403df40 auth_method 0 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ecc0b1040 con 0x7f0ed013cb40 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.664+0000 7f0ec5ffb640 10 monclient: dump: 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 10 client.74217.objecter ms_handle_connect 0x7f0eb403df40 2024-09-11T03:01:23.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ed5dc5640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0eb403df40 0x7f0eb4040400 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00a5ca0 tx=0x7f0ec800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ecc0ab770 con 0x7f0ed013cb40 2024-09-11T03:01:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed013cb40 msgr2=0x7f0ed002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed013cb40 0x7f0ed002d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0ed006f640 tx=0x7f0ecc0191c0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed013cb40 msgr2=0x7f0ed002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.668+0000 7f0ec5ffb640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed013cb40 0x7f0ed002d450 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0ed006f640 tx=0x7f0ecc0191c0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed7828640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed7828640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed002d990 0x7f0ed002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed7828640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed006e870 con 0x7f0ed002d990 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed002d990 0x7f0ed002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient: get_auth_request con 0x7f0ed002d990 auth_method 0 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 10 monclient(hunting): handle_auth_done global_id 74235 payload 824 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 1 -- 172.21.15.98:0/61571417 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ed0073630 con 0x7f0ed002d990 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ed4dc3640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed002d990 0x7f0ed002dd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc0a9000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ec5ffb640 10 client.74217.objecter ms_handle_connect 0x7f0ed002d990 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.672+0000 7f0ec5ffb640 10 client.74217.objecter resend_mon_ops 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ecc0c6070 con 0x7f0ed002d990 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:01:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 10 monclient: dump: 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9255 (secure 0 0 0) 0x7f0ecc003bd0 con 0x7f0ed002d990 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed002d990 msgr2=0x7f0ed002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed002d990 0x7f0ed002dd90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc0a9000 comp rx=0 tx=0).stop 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed002d990 msgr2=0x7f0ed002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ec5ffb640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed002d990 0x7f0ed002dd90 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0098a0 tx=0x7f0ecc0a9000 comp rx=0 tx=0).stop 2024-09-11T03:01:23.694 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 40603, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:23.695 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.696 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.697 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:23.698 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.699 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.700 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:23.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.705 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.707 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:23.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:23.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:23.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:23.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:23.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.716 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:23.717 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 10 librados: watch_flush enter 2024-09-11T03:01:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 10 librados: watch_flush exit 2024-09-11T03:01:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 20 client.74217.objecter shutdown clearing up homeless session... 2024-09-11T03:01:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 10 client.74217.objecter successfully canceled tick 2024-09-11T03:01:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0eb403df40 msgr2=0x7f0eb4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0eb403df40 0x7f0eb4040400 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00a5ca0 tx=0x7f0ec800d040 comp rx=0 tx=0).stop 2024-09-11T03:01:23.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 10 monclient: shutdown 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed006fce0 msgr2=0x7f0ed00a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 --2- 172.21.15.98:0/61571417 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ed006fce0 0x7f0ed00a48b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000d5f0 tx=0x7f0ec000dab0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed5dc5640 1 -- 172.21.15.98:0/61571417 reap_dead start 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed5dc5640 1 -- 172.21.15.98:0/61571417 reap_dead start 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 shutdown_connections 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 >> 172.21.15.98:0/61571417 conn(0x7f0ed00ac6b0 msgr2=0x7f0ed00aa210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 shutdown_connections 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 -- 172.21.15.98:0/61571417 wait complete. 2024-09-11T03:01:23.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.676+0000 7f0ed7828640 1 librados: shutdown 2024-09-11T03:01:23.721 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:23.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-09-11T03:01:23.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:01:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient: get_monmap_and_config 2024-09-11T03:01:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient: monmap: 2024-09-11T03:01:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.839 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:23.841327+0000 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:23.841327+0000 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.840 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:23.841 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 1 Processor -- start 2024-09-11T03:01:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 1 -- start start 2024-09-11T03:01:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient: init 2024-09-11T03:01:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient: _add_conns ranks=[9,0,10,6,3,2,8,5,1,4,7] 2024-09-11T03:01:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27f8139b10 0x7f27f8072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f27f8139b10 addr v2:172.21.15.98:3300/0 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27f8139ef0 addr v1:172.21.15.98:6789/0 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f27f813a2d0 0x7f27f80a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient(hunting): picked mon.noname-f con 0x7f27f813a2d0 addr v2:172.21.15.190:3300/0 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f80702b0 con 0x7f27f8139ef0 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f27f8071bf0 con 0x7f27f8139b10 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f27f8071350 con 0x7f27f813a2d0 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27fcb5d640 10 monclient(hunting): authenticate will time out at 1637.721670s 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27f8139b10 0x7f27f8072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f27f8139b10 0x7f27f8072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39134/0 (socket says 172.21.15.98:39134) 2024-09-11T03:01:23.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 -- 172.21.15.98:0/1513075155 learned_addr learned my addr 172.21.15.98:0/1513075155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27edd9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f27f8139ef0 0x7f27f80a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43912/0 (socket says 172.21.15.98:43912) 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): get_auth_request con 0x7f27f8139b10 auth_method 0 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f6d9d640 1 --2- 172.21.15.98:0/1513075155 >> v2:172.21.15.190:3300/0 conn(0x7f27f813a2d0 0x7f27f80a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f6d9d640 10 monclient(hunting): get_auth_request con 0x7f27f813a2d0 auth_method 0 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient(hunting): handle_auth_done global_id 44901 payload 274 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 -- 172.21.15.98:0/1513075155 >> v2:172.21.15.190:3300/0 conn(0x7f27f813a2d0 msgr2=0x7f27f80a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 --2- 172.21.15.98:0/1513075155 >> v2:172.21.15.190:3300/0 conn(0x7f27f813a2d0 0x7f27f80a8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 -- 172.21.15.98:0/1513075155 >> v1:172.21.15.98:6789/0 conn(0x7f27f8139ef0 legacy=0x7f27f80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 monclient: found mon.noname-a 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f5d9b640 1 -- 172.21.15.98:0/1513075155 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108121538 0 0) 0x7f27e0001120 con 0x7f27f8139ef0 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f659c640 1 -- 172.21.15.98:0/1513075155 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f80a9620 con 0x7f27f8139b10 2024-09-11T03:01:23.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.840+0000 7f27f5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:01:23.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f659c640 1 --2- 172.21.15.98:0/1513075155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8139b10 0x7f27f8072600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f27f80a9e90 tx=0x7f27e801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a81f7df4b9f0af server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 1 -- 172.21.15.98:0/1513075155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27e80b0070 con 0x7f27f8139b10 2024-09-11T03:01:23.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:23.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: dump: 2024-09-11T03:01:23.853 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.853 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.854 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 1 -- 172.21.15.98:0/1513075155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e80aa810 con 0x7f27f8139b10 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 5 monclient: authenticate success, global_id 44901 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 4 set_mon_vals no callback set 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 1 -- 172.21.15.98:0/1513075155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27e80aa9c0 con 0x7f27f8139b10 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f5d9b640 10 monclient: dump: 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.856 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient: shutdown 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1513075155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8139b10 msgr2=0x7f27f8072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 --2- 172.21.15.98:0/1513075155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8139b10 0x7f27f8072600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f27f80a9e90 tx=0x7f27e801c470 comp rx=0 tx=0).stop 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1513075155 shutdown_connections 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 --2- 172.21.15.98:0/1513075155 >> v2:172.21.15.190:3300/0 conn(0x7f27f813a2d0 0x7f27f80a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 --2- 172.21.15.98:0/1513075155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f8139b10 0x7f27f8072600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1513075155 >> 172.21.15.98:0/1513075155 conn(0x7f27f80ac790 msgr2=0x7f27f80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1513075155 shutdown_connections 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1513075155 wait complete. 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient: build_initial_monmap 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient: monmap: 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:23.858 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 librados: starting msgr at 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 librados: starting objecter 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 Processor -- start 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- start start 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 librados: setting wanted keys 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 librados: calling monclient init 2024-09-11T03:01:23.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient: init 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient: _add_conns ranks=[3,8,0,7,6,4,2,1,5] 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27f8139b10 0x7f27f813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): picked mon.noname-g con 0x7f27f8139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27f8139ef0 0x7f27f813ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): picked mon.noname-e con 0x7f27f8139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f813a2d0 0x7f27f813b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): picked mon.noname-a con 0x7f27f813a2d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27f8144470 con 0x7f27f813a2d0 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f27f80a8870 con 0x7f27f8139b10 2024-09-11T03:01:23.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f27f81445f0 con 0x7f27f8139ef0 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27fcb5d640 10 monclient(hunting): authenticate will time out at 1637.727561s 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f813a2d0 0x7f27f813b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27f8139ef0 0x7f27f813ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f813a2d0 0x7f27f813b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39140/0 (socket says 172.21.15.98:39140) 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 1 -- 172.21.15.98:0/1047680396 learned_addr learned my addr 172.21.15.98:0/1047680396 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 10 monclient(hunting): get_auth_request con 0x7f27f813a2d0 auth_method 0 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 10 monclient(hunting): get_auth_request con 0x7f27f8139ef0 auth_method 0 2024-09-11T03:01:23.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.844+0000 7f27f659c640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27f8139b10 0x7f27f813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 10 monclient(hunting): handle_auth_done global_id 44919 payload 824 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27f8139ef0 msgr2=0x7f27f813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27f8139ef0 0x7f27f813ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27f8139b10 msgr2=0x7f27f813a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27f8139b10 0x7f27f813a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 10 monclient: _finish_hunting 0 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 1 monclient: found mon.noname-a 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 1 -- 172.21.15.98:0/1047680396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f8140fc0 con 0x7f27f813a2d0 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient: get_auth_request con 0x7f27f8139b10 auth_method 0 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f27f8139b10 0x7f27f813a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27edd9b640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f27f8139ef0 0x7f27f813ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f6d9d640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f813a2d0 0x7f27f813b250 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f27f0007ec0 tx=0x7f27f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 client.?.objecter ms_handle_connect 0x7f27f813a2d0 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27f000ee90 con 0x7f27f813a2d0 2024-09-11T03:01:23.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: dump: 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.864 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: _finish_auth 0 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27f0019070 con 0x7f27f813a2d0 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27f0014d00 con 0x7f27f813a2d0 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:23.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: dump: 2024-09-11T03:01:23.866 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.866 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.866 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.866 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.866 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f559a640 4 set_mon_vals no callback set 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 5 monclient: authenticate success, global_id 44919 2024-09-11T03:01:23.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 10 monclient: _renew_subs 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f813b970 con 0x7f27f813a2d0 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 10 client.44919.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 10 monclient: _renew_subs 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f813cbb0 con 0x7f27f813a2d0 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 1 librados: init done 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8133760 0x7f27f8133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f27f806ca10 con 0x7f27f8133760 2024-09-11T03:01:23.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f27f0014420 con 0x7f27f813a2d0 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f27d803e0e0 0x7f27d80405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8133760 0x7f27f8133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(439..439 src has 1..439) ==== 6202+0+0 (secure 0 0 0) 0x7f27f004fe70 con 0x7f27f813a2d0 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 client.44919.objecter ms_dispatch 0x7f27f8000ba0 osd_map(439..439 src has 1..439) 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 3 client.44919.objecter handle_osd_map got epochs [439,439] > 0 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 3 client.44919.objecter handle_osd_map decoding full epoch 439 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient: get_auth_request con 0x7f27f8133760 auth_method 0 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 20 client.44919.objecter dump_active .. 0 homeless 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27edd9b640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f27d803e0e0 0x7f27d80405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27edd9b640 10 monclient: get_auth_request con 0x7f27d803e0e0 auth_method 0 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 10 monclient(hunting): handle_auth_done global_id 74210 payload 824 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 1 -- 172.21.15.98:0/1047680396 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27f80a8200 con 0x7f27f8133760 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27f659c640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8133760 0x7f27f8133b40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f27f80a9a70 tx=0x7f27e8019210 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 client.44919.objecter ms_handle_connect 0x7f27f8133760 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 client.44919.objecter resend_mon_ops 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 client.44919.objecter ms_handle_connect 0x7f27d803e0e0 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27edd9b640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f27d803e0e0 0x7f27d80405a0 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000f70 tx=0x7f27e0001920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27e80b0040 con 0x7f27f8133760 2024-09-11T03:01:23.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.848+0000 7f27ef7fe640 10 monclient: dump: 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.872 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.852+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f27e8003620 con 0x7f27f8133760 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.852+0000 7f27ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.852+0000 7f27ef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.852+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8133760 msgr2=0x7f27f8133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.852+0000 7f27ef7fe640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8133760 0x7f27f8133b40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f27f80a9a70 tx=0x7f27e8019210 comp rx=0 tx=0).stop 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.852+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8133760 msgr2=0x7f27f8133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.852+0000 7f27ef7fe640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8133760 0x7f27f8133b40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27fcb5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:23.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27fcb5d640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8134080 0x7f27f80a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:23.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27fcb5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:23.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f27f806e870 con 0x7f27f8134080 2024-09-11T03:01:23.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8134080 0x7f27f80a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:23.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient: get_auth_request con 0x7f27f8134080 auth_method 0 2024-09-11T03:01:23.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 10 monclient(hunting): handle_auth_done global_id 74219 payload 824 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 1 -- 172.21.15.98:0/1047680396 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f27f806e680 con 0x7f27f8134080 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27f659c640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8134080 0x7f27f80a0e60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f27e80097c0 tx=0x7f27e80a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27ef7fe640 10 client.44919.objecter ms_handle_connect 0x7f27f8134080 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.856+0000 7f27ef7fe640 10 client.44919.objecter resend_mon_ops 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27e80b0040 con 0x7f27f8134080 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 10 monclient: dump: 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9255 (secure 0 0 0) 0x7f27e80b69b0 con 0x7f27f8134080 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8134080 msgr2=0x7f27f80a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8134080 0x7f27f80a0e60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f27e80097c0 tx=0x7f27e80a8000 comp rx=0 tx=0).stop 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8134080 msgr2=0x7f27f80a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27ef7fe640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f27f8134080 0x7f27f80a0e60 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f27e80097c0 tx=0x7f27e80a8000 comp rx=0 tx=0).stop 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 40787, 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:23.879 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.880 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:23.882 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:23.883 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:23.884 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:23.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.891 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:23.899 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:23.900 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:23.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:23.902 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:23.902 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:23.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 10 librados: watch_flush enter 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 10 librados: watch_flush exit 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 20 client.44919.objecter shutdown clearing up homeless session... 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 10 client.44919.objecter successfully canceled tick 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f27d803e0e0 msgr2=0x7f27d80405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f27d803e0e0 0x7f27d80405a0 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7f27e0000f70 tx=0x7f27e0001920 comp rx=0 tx=0).stop 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 10 monclient: shutdown 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f813a2d0 msgr2=0x7f27f813b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 --2- 172.21.15.98:0/1047680396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f813a2d0 0x7f27f813b250 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f27f0007ec0 tx=0x7f27f000e4b0 comp rx=0 tx=0).stop 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27f6d9d640 1 -- 172.21.15.98:0/1047680396 reap_dead start 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27f6d9d640 1 -- 172.21.15.98:0/1047680396 reap_dead start 2024-09-11T03:01:23.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 shutdown_connections 2024-09-11T03:01:23.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 >> 172.21.15.98:0/1047680396 conn(0x7f27f80ac790 msgr2=0x7f27f80aa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:23.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 shutdown_connections 2024-09-11T03:01:23.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 -- 172.21.15.98:0/1047680396 wait complete. 2024-09-11T03:01:23.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:23.860+0000 7f27fcb5d640 1 librados: shutdown 2024-09-11T03:01:23.905 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:23.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-09-11T03:01:23.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:01:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.020+0000 7f903dde8640 10 monclient: get_monmap_and_config 2024-09-11T03:01:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.020+0000 7f903dde8640 10 monclient: build_initial_monmap 2024-09-11T03:01:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient: monmap: 2024-09-11T03:01:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:24.023 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:01:24.025269+0000 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:01:24.025269+0000 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:24.024 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:24.025 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:01:24.025 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:01:24.025 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 1 Processor -- start 2024-09-11T03:01:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 1 -- start start 2024-09-11T03:01:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient: init 2024-09-11T03:01:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:24.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient: _add_conns ranks=[5,0,4,6,10,9,3,7,2,8,1] 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90380a52a0 0x7f903809ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient(hunting): picked mon.noname-h con 0x7f90380a52a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9038073c20 addr v1:172.21.15.98:6789/0 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f903813b480 0x7f903813d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient(hunting): picked mon.noname-c con 0x7f903813b480 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90380a5680 con 0x7f9038073c20 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f903809dbf0 con 0x7f903813b480 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f903809f600 con 0x7f90380a52a0 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f903dde8640 10 monclient(hunting): authenticate will time out at 1637.905551s 2024-09-11T03:01:24.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f903813b480 0x7f903813d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9036ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9038073c20 0x7f903809cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:43928/0 (socket says 172.21.15.98:43928) 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9036ffd640 1 -- 172.21.15.98:0/3334640799 learned_addr learned my addr 172.21.15.98:0/3334640799 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): get_auth_request con 0x7f903813b480 auth_method 0 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90377fe640 1 --2- 172.21.15.98:0/3334640799 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90380a52a0 0x7f903809ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90377fe640 10 monclient(hunting): get_auth_request con 0x7f90380a52a0 auth_method 0 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:24.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90367fc640 1 -- 172.21.15.98:0/3334640799 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 498163001 0 0) 0x7f90380a5680 con 0x7f9038073c20 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90367fc640 10 monclient(hunting): _negotiate 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90367fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90367fc640 10 monclient(hunting): my global_id is 44928 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90367fc640 1 -- 172.21.15.98:0/3334640799 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9024003530 con 0x7f9038073c20 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient(hunting): handle_auth_done global_id 24430 payload 274 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 1 -- 172.21.15.98:0/3334640799 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90380a52a0 msgr2=0x7f903809ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 1 --2- 172.21.15.98:0/3334640799 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90380a52a0 0x7f903809ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 1 -- 172.21.15.98:0/3334640799 >> v1:172.21.15.98:6789/0 conn(0x7f9038073c20 legacy=0x7f903809cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 1 monclient: found mon.noname-c 2024-09-11T03:01:24.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f9037fff640 1 -- 172.21.15.98:0/3334640799 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90380a41a0 con 0x7f903813b480 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.024+0000 7f90377fe640 1 --2- 172.21.15.98:0/3334640799 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90380a52a0 0x7f903809ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 --2- 172.21.15.98:0/3334640799 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f903813b480 0x7f903813d870 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f9030009820 tx=0x7f903001c350 comp rx=0 tx=0).ready entity=mon.4 client_cookie=354397347bea4c9c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 1 -- 172.21.15.98:0/3334640799 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f90300b0070 con 0x7f903813b480 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: dump: 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.032 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: _finish_auth 0 2024-09-11T03:01:24.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: _check_auth_tickets 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 5 monclient: authenticate success, global_id 24430 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 1 -- 172.21.15.98:0/3334640799 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90300aa810 con 0x7f903813b480 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: get_monmap_and_config success 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 4 set_mon_vals no callback set 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 1 -- 172.21.15.98:0/3334640799 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f90300a9300 con 0x7f903813b480 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90367fc640 10 monclient: dump: 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:24.034 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:24.035 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: shutdown 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- 172.21.15.98:0/3334640799 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f903813b480 msgr2=0x7f903813d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 --2- 172.21.15.98:0/3334640799 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f903813b480 0x7f903813d870 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f9030009820 tx=0x7f903001c350 comp rx=0 tx=0).stop 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- 172.21.15.98:0/3334640799 shutdown_connections 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 --2- 172.21.15.98:0/3334640799 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f903813b480 0x7f903813d870 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 --2- 172.21.15.98:0/3334640799 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f90380a52a0 0x7f903809ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- 172.21.15.98:0/3334640799 >> 172.21.15.98:0/3334640799 conn(0x7f90380a4780 msgr2=0x7f90380a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- 172.21.15.98:0/3334640799 shutdown_connections 2024-09-11T03:01:24.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- 172.21.15.98:0/3334640799 wait complete. 2024-09-11T03:01:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: build_initial_monmap 2024-09-11T03:01:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: monmap: 2024-09-11T03:01:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:01:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:01:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:01:24.044 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 librados: starting msgr at 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 librados: starting objecter 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 Processor -- start 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- start start 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 librados: setting wanted keys 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 librados: calling monclient init 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: init 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: _reopen_session rank -1 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: _add_conns ranks=[7,8,0,1,2,4,6,3,5] 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9038073c20 0x7f9038135050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): picked mon.noname-i con 0x7f9038073c20 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90380a52a0 0x7f9038135590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): picked mon.noname-e con 0x7f90380a52a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:01:24.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903813b480 0x7f903811e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): picked mon.noname-a con 0x7f903813b480 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f903809d820 con 0x7f903813b480 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f903809d6a0 con 0x7f9038073c20 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f90381420e0 con 0x7f90380a52a0 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): _renew_subs 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient(hunting): authenticate will time out at 1637.910852s 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903813b480 0x7f903811e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903813b480 0x7f903811e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39142/0 (socket says 172.21.15.98:39142) 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 -- 172.21.15.98:0/94190312 learned_addr learned my addr 172.21.15.98:0/94190312 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:01:24.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): get_auth_request con 0x7f903813b480 auth_method 0 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90380a52a0 0x7f9038135590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 10 monclient(hunting): get_auth_request con 0x7f90380a52a0 auth_method 0 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90377fe640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9038073c20 0x7f9038135050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:24.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90377fe640 10 monclient(hunting): get_auth_request con 0x7f9038073c20 auth_method 0 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient(hunting): handle_auth_done global_id 44937 payload 824 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90380a52a0 msgr2=0x7f9038135590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f90380a52a0 0x7f9038135590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9038073c20 msgr2=0x7f9038135050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9038073c20 0x7f9038135050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient: _finish_hunting 0 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 monclient: found mon.noname-a 2024-09-11T03:01:24.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 -- 172.21.15.98:0/94190312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90381219d0 con 0x7f903813b480 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f90377fe640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9038073c20 0x7f9038135050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9037fff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903813b480 0x7f903811e2d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f90300ae040 tx=0x7f90300190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 client.?.objecter ms_handle_connect 0x7f903813b480 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f90300b0050 con 0x7f903813b480 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: dump: 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:24.050 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:24.051 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: _finish_auth 0 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: _check_auth_tickets 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90300bc070 con 0x7f903813b480 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 5 monclient: authenticate success, global_id 44937 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: _renew_subs 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9038136130 con 0x7f903813b480 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f90300044b0 con 0x7f903813b480 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:24.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f9035ffb640 4 set_mon_vals no callback set 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 client.44937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f902ffff640 10 monclient: dump: 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:24.053 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: _renew_subs 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90380a6040 con 0x7f903813b480 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.028+0000 7f903dde8640 1 librados: init done 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f903dde8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f903dde8640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811e810 0x7f903811ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f903dde8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f903806ca10 con 0x7f903811e810 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9030004040 con 0x7f903813b480 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811e810 0x7f903811ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient: get_auth_request con 0x7f903811e810 auth_method 0 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f901803df40 0x7f9018040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(439..439 src has 1..439) ==== 6202+0+0 (secure 0 0 0) 0x7f90300e9e00 con 0x7f903813b480 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 client.44937.objecter ms_dispatch 0x7f9038000ba0 osd_map(439..439 src has 1..439) 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 3 client.44937.objecter handle_osd_map got epochs [439,439] > 0 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 3 client.44937.objecter handle_osd_map decoding full epoch 439 2024-09-11T03:01:24.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 20 client.44937.objecter dump_active .. 0 homeless 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f9036ffd640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f901803df40 0x7f9018040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f9036ffd640 10 monclient: get_auth_request con 0x7f901803df40 auth_method 0 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 10 monclient(hunting): handle_auth_done global_id 19987 payload 824 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 1 -- 172.21.15.98:0/94190312 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f903806e850 con 0x7f903811e810 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 client.44937.objecter ms_handle_connect 0x7f903811e810 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 client.44937.objecter resend_mon_ops 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f90377fe640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811e810 0x7f903811ebf0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f90280046a0 tx=0x7f9028007700 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9028014880 con 0x7f903811e810 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:24.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 monclient: dump: 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:24.057 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 client.44937.objecter ms_handle_connect 0x7f901803df40 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f9036ffd640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f901803df40 0x7f9018040400 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f9020007a70 tx=0x7f90200079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9028019070 con 0x7f903811e810 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811e810 msgr2=0x7f903811ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811e810 0x7f903811ebf0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f90280046a0 tx=0x7f9028007700 comp rx=0 tx=0).stop 2024-09-11T03:01:24.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811e810 msgr2=0x7f903811ebf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.032+0000 7f902ffff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811e810 0x7f903811ebf0 secure :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f90280046a0 tx=0x7f9028007700 comp rx=0 tx=0).stop 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811f130 0x7f903806f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f9038121d10 con 0x7f903811f130 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811f130 0x7f903806f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient: get_auth_request con 0x7f903811f130 auth_method 0 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:01:24.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 10 monclient(hunting): handle_auth_done global_id 19996 payload 824 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 1 -- 172.21.15.98:0/94190312 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9038068b40 con 0x7f903811f130 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 10 client.44937.objecter ms_handle_connect 0x7f903811f130 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 10 client.44937.objecter resend_mon_ops 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f90377fe640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811f130 0x7f903806f750 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f90280046a0 tx=0x7f9028004a50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9028014c10 con 0x7f903811f130 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 10 monclient: dump: 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:01:24.060 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9425 (secure 0 0 0) 0x7f9028014c10 con 0x7f903811f130 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:01:24.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811f130 msgr2=0x7f903806f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811f130 0x7f903806f750 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f90280046a0 tx=0x7f9028004a50 comp rx=0 tx=0).stop 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811f130 msgr2=0x7f903806f750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f902ffff640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f903811f130 0x7f903806f750 secure :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f90280046a0 tx=0x7f9028004a50 comp rx=0 tx=0).stop 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 84, 2024-09-11T03:01:24.062 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 699813, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:01:24.063 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:01:24.064 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:01:24.066 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:01:24.067 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:01:24.069 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.070 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.071 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:01:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:01:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:01:24.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:01:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:01:24.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:01:24.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.086 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:01:24.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:01:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:01:24.088 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:01:24.088 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:01:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 10 librados: watch_flush enter 2024-09-11T03:01:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 10 librados: watch_flush exit 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 20 client.44937.objecter shutdown clearing up homeless session... 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 10 client.44937.objecter successfully canceled tick 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f901803df40 msgr2=0x7f9018040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f901803df40 0x7f9018040400 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f9020007a70 tx=0x7f90200079c0 comp rx=0 tx=0).stop 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 10 monclient: shutdown 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903813b480 msgr2=0x7f903811e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 --2- 172.21.15.98:0/94190312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f903813b480 0x7f903811e2d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f90300ae040 tx=0x7f90300190f0 comp rx=0 tx=0).stop 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f9037fff640 1 -- 172.21.15.98:0/94190312 reap_dead start 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f9037fff640 1 -- 172.21.15.98:0/94190312 reap_dead start 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 shutdown_connections 2024-09-11T03:01:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 >> 172.21.15.98:0/94190312 conn(0x7f90380a4780 msgr2=0x7f903809fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:01:24.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 shutdown_connections 2024-09-11T03:01:24.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 -- 172.21.15.98:0/94190312 wait complete. 2024-09-11T03:01:24.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:01:24.040+0000 7f903dde8640 1 librados: shutdown 2024-09-11T03:01:24.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:01:24.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T03:01:33.390 INFO:tasks.ceph.osd.3.smithi190.stderr:2024-09-11T03:01:33.393+0000 7efddf8c8640 -1 snap_mapper.add_oid found existing snaps mapped on 104:58dffa4d:test-rados-api-smithi190-28350-97::foo:22, removing 2024-09-11T03:01:33.391 INFO:tasks.ceph.osd.0.smithi098.stderr:2024-09-11T03:01:33.388+0000 7f2bc7665640 -1 snap_mapper.add_oid found existing snaps mapped on 104:58dffa4d:test-rados-api-smithi190-28350-97::foo:22, removing 2024-09-11T03:02:54.092 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-09-11T03:02:54.093 INFO:tasks.ceph.mon.d:Restarting daemon 2024-09-11T03:02:54.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-09-11T03:02:54.097 INFO:tasks.ceph.mon.d:Started 2024-09-11T03:02:54.097 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T03:02:54.098 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T03:02:54.098 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-11T03:02:54.098 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-11T03:02:54.098 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-09-11T03:02:54.098 INFO:tasks.ceph.mon.e:Sent signal 19 2024-09-11T03:02:54.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T03:02:54.210 INFO:tasks.ceph.mon.d.smithi098.stderr:2024-09-11T03:02:54.207+0000 7f2b93a08e00 -1 mon.d@-1(???).log v0 log_external_backlog rewinding external_log_to from 458 -> 0 (sync_force? mon rebuild?) 2024-09-11T03:03:09.099 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T03:03:09.100 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T03:03:09.100 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-11T03:03:09.100 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-11T03:03:09.100 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-09-11T03:03:09.101 INFO:tasks.ceph.mon.e:Sent signal 18 2024-09-11T03:03:09.101 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:03:09.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:03:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient: get_monmap_and_config 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient: build_initial_monmap 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient: monmap: 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:09.246748+0000 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:09.246748+0000 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:09.246 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:09.247 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 Processor -- start 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 -- start start 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient: init 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient: _add_conns ranks=[6,4,9,8,5,3,2,7,0,1,10] 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406f650 0x7f0f94072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): picked mon.noname-d con 0x7f0f9406f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f940a8380 0x7f0f9406fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): picked mon.noname-c con 0x7f0f940a8380 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:09.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0f9406ff70 0x7f0f941397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9406ff70 addr v2:172.21.15.98:3300/0 2024-09-11T03:03:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0f940a8800 con 0x7f0f9406ff70 2024-09-11T03:03:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0f94071a80 con 0x7f0f940a8380 2024-09-11T03:03:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0f94071d60 con 0x7f0f9406f650 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f98cdb640 10 monclient(hunting): authenticate will time out at 1743.127187s 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406f650 0x7f0f94072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406f650 0x7f0f94072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34304/0 (socket says 172.21.15.98:34304) 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 -- 172.21.15.98:0/2021223647 learned_addr learned my addr 172.21.15.98:0/2021223647 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): get_auth_request con 0x7f0f9406f650 auth_method 0 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 1 --2- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f940a8380 0x7f0f9406fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 10 monclient(hunting): get_auth_request con 0x7f0f940a8380 auth_method 0 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 1 --2- 172.21.15.98:0/2021223647 >> v2:172.21.15.98:3300/0 conn(0x7f0f9406ff70 0x7f0f941397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 10 monclient(hunting): get_auth_request con 0x7f0f9406ff70 auth_method 0 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient(hunting): handle_auth_done global_id 84111 payload 274 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 -- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f940a8380 msgr2=0x7f0f9406fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 --2- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f940a8380 0x7f0f9406fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 -- 172.21.15.98:0/2021223647 >> v2:172.21.15.98:3300/0 conn(0x7f0f9406ff70 msgr2=0x7f0f941397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 --2- 172.21.15.98:0/2021223647 >> v2:172.21.15.98:3300/0 conn(0x7f0f9406ff70 0x7f0f941397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:09.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 monclient: found mon.noname-d 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.243+0000 7f0f9259c640 1 -- 172.21.15.98:0/2021223647 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f940714c0 con 0x7f0f9406f650 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9259c640 1 --2- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406f650 0x7f0f94072640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009870 tx=0x7f0f8801cc50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a2fdf92413ea24c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 1 -- 172.21.15.98:0/2021223647 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f880b0070 con 0x7f0f9406f650 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: dump: 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.263 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.264 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: _finish_auth 0 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: _check_auth_tickets 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 1 -- 172.21.15.98:0/2021223647 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f880aa810 con 0x7f0f9406f650 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 5 monclient: authenticate success, global_id 84111 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 1 -- 172.21.15.98:0/2021223647 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f880a9300 con 0x7f0f9406f650 2024-09-11T03:03:09.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 4 set_mon_vals no callback set 2024-09-11T03:03:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f9159a640 10 monclient: dump: 2024-09-11T03:03:09.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.267 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 10 monclient: shutdown 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:09.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 -- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406f650 msgr2=0x7f0f94072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406f650 0x7f0f94072640 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009870 tx=0x7f0f8801cc50 comp rx=0 tx=0).stop 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 -- 172.21.15.98:0/2021223647 shutdown_connections 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/2021223647 >> v2:172.21.15.98:3300/0 conn(0x7f0f9406ff70 0x7f0f941397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0f940a8380 0x7f0f9406fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/2021223647 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406f650 0x7f0f94072640 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 -- 172.21.15.98:0/2021223647 >> 172.21.15.98:0/2021223647 conn(0x7f0f940ac8e0 msgr2=0x7f0f940a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 -- 172.21.15.98:0/2021223647 shutdown_connections 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 -- 172.21.15.98:0/2021223647 wait complete. 2024-09-11T03:03:09.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 10 monclient: build_initial_monmap 2024-09-11T03:03:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 10 monclient: monmap: 2024-09-11T03:03:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:09.270 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:09.271 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 librados: starting msgr at 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 librados: starting objecter 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 Processor -- start 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 -- start start 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 librados: setting wanted keys 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 1 librados: calling monclient init 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.247+0000 7f0f98cdb640 10 monclient: init 2024-09-11T03:03:09.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _add_conns ranks=[8,6,1,5,4,2,3,7,0] 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f9406f650 0x7f0f941d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): picked mon.noname-e con 0x7f0f9406f650 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406ff70 0x7f0f94138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): picked mon.noname-d con 0x7f0f9406ff70 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940a8380 0x7f0f9413da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): picked mon.noname-f con 0x7f0f940a8380 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f94142fc0 con 0x7f0f940a8380 2024-09-11T03:03:09.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0f94142e40 con 0x7f0f9406ff70 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0f94143140 con 0x7f0f9406f650 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient(hunting): authenticate will time out at 1743.132212s 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f9406f650 0x7f0f941d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406ff70 0x7f0f94138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f9406f650 0x7f0f941d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52286/0 (socket says 172.21.15.98:52286) 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 1 -- 172.21.15.98:0/1568560388 learned_addr learned my addr 172.21.15.98:0/1568560388 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient(hunting): get_auth_request con 0x7f0f9406f650 auth_method 0 2024-09-11T03:03:09.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): get_auth_request con 0x7f0f9406ff70 auth_method 0 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient(hunting): handle_auth_done global_id 84120 payload 824 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f92d9d640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940a8380 0x7f0f9413da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 -- 172.21.15.98:0/1568560388 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f9406f650 msgr2=0x7f0f941d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f9406f650 0x7f0f941d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 -- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940a8380 msgr2=0x7f0f9413da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940a8380 0x7f0f9413da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 monclient: found mon.noname-d 2024-09-11T03:03:09.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 -- 172.21.15.98:0/1568560388 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f94140000 con 0x7f0f9406ff70 2024-09-11T03:03:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f92d9d640 10 monclient: get_auth_request con 0x7f0f940a8380 auth_method 0 2024-09-11T03:03:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f92d9d640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940a8380 0x7f0f9413da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f91d9b640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406ff70 0x7f0f94138880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400e870 tx=0x7f0f8400ed30 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 client.?.objecter ms_handle_connect 0x7f0f9406ff70 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 1 -- 172.21.15.98:0/1568560388 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f84017070 con 0x7f0f9406ff70 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: dump: 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.278 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.279 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.280 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:09.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:09.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:09.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 1 -- 172.21.15.98:0/1568560388 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8400ee60 con 0x7f0f9406ff70 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 5 monclient: authenticate success, global_id 84120 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _renew_subs 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f94072940 con 0x7f0f9406ff70 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 1 -- 172.21.15.98:0/1568560388 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0f8400cdb0 con 0x7f0f9406ff70 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 monclient: dump: 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.282 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.282 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.282 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.282 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.282 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.282 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 client.84120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _renew_subs 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f9413f450 con 0x7f0f9406ff70 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f90d99640 4 set_mon_vals no callback set 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 librados: init done 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:09.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 1 -- 172.21.15.98:0/1568560388 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0f84024060 con 0x7f0f9406ff70 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f941397c0 con 0x7f0f9406ff70 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f7403df40 0x7f0f74040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 1 -- 172.21.15.98:0/1568560388 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(523..523 src has 1..523) ==== 6657+0+0 (secure 0 0 0) 0x7f0f8400f040 con 0x7f0f9406ff70 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 client.84120.objecter ms_dispatch 0x7f0f94000ba0 osd_map(523..523 src has 1..523) 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 3 client.84120.objecter handle_osd_map got epochs [523,523] > 0 2024-09-11T03:03:09.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 3 client.84120.objecter handle_osd_map decoding full epoch 523 2024-09-11T03:03:09.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 20 client.84120.objecter dump_active .. 0 homeless 2024-09-11T03:03:09.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f7403df40 0x7f0f74040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 10 monclient: get_auth_request con 0x7f0f7403df40 auth_method 0 2024-09-11T03:03:09.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f82ffd640 10 client.84120.objecter ms_handle_connect 0x7f0f7403df40 2024-09-11T03:03:09.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.251+0000 7f0f9259c640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f7403df40 0x7f0f74040400 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f0f880097c0 tx=0x7f0f88004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.255+0000 7f0f82ffd640 1 -- 172.21.15.98:0/1568560388 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f0f84013470 con 0x7f0f9406ff70 2024-09-11T03:03:09.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.255+0000 7f0f82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:09.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.255+0000 7f0f82ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.503+0000 7f0f98cdb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:03:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.503+0000 7f0f98cdb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.503+0000 7f0f98cdb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:09.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.503+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0f94068f80 con 0x7f0f9406ff70 2024-09-11T03:03:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.503+0000 7f0f82ffd640 1 -- 172.21.15.98:0/1568560388 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f0f840137f0 con 0x7f0f9406ff70 2024-09-11T03:03:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.503+0000 7f0f82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:09.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.503+0000 7f0f82ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:09.505 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":96,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 10 librados: watch_flush enter 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 10 librados: watch_flush exit 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 20 client.84120.objecter shutdown clearing up homeless session... 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 10 client.84120.objecter successfully canceled tick 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f7403df40 msgr2=0x7f0f74040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f7403df40 0x7f0f74040400 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f0f880097c0 tx=0x7f0f88004740 comp rx=0 tx=0).stop 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 10 monclient: shutdown 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406ff70 msgr2=0x7f0f94138880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406ff70 0x7f0f94138880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0f8400e870 tx=0x7f0f8400ed30 comp rx=0 tx=0).stop 2024-09-11T03:03:09.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 shutdown_connections 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0f7403df40 0x7f0f74040400 unknown :-1 s=CLOSED pgs=3611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0f9406ff70 0x7f0f94138880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f940a8380 0x7f0f9413da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 --2- 172.21.15.98:0/1568560388 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0f9406f650 0x7f0f941d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 >> 172.21.15.98:0/1568560388 conn(0x7f0f940ac8e0 msgr2=0x7f0f94072c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 shutdown_connections 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 -- 172.21.15.98:0/1568560388 wait complete. 2024-09-11T03:03:09.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.507+0000 7f0f98cdb640 1 librados: shutdown 2024-09-11T03:03:09.531 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:03:09.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T03:03:09.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:03:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.647+0000 7f1aed2e2640 10 monclient: get_monmap_and_config 2024-09-11T03:03:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.647+0000 7f1aed2e2640 10 monclient: build_initial_monmap 2024-09-11T03:03:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.647+0000 7f1aed2e2640 10 monclient: monmap: 2024-09-11T03:03:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.650 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:09.652205+0000 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:09.652205+0000 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:09.651 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:09.652 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:09.652 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.647+0000 7f1aed2e2640 1 Processor -- start 2024-09-11T03:03:09.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.647+0000 7f1aed2e2640 1 -- start start 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.647+0000 7f1aed2e2640 10 monclient: init 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient: _add_conns ranks=[8,3,1,2,6,7,5,10,0,9,4] 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1ae8139b10 0x7f1ae8072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient(hunting): picked mon.noname-e con 0x7f1ae8139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae8139ef0 0x7f1ae80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient(hunting): picked mon.noname-g con 0x7f1ae8139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1ae813a2d0 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae80702b0 con 0x7f1ae813a2d0 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae8071bf0 con 0x7f1ae8139ef0 2024-09-11T03:03:09.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1ae8071350 con 0x7f1ae8139b10 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient(hunting): authenticate will time out at 1743.532188s 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1ae8139b10 0x7f1ae8072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1ae8139b10 0x7f1ae8072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52312/0 (socket says 172.21.15.98:52312) 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 -- 172.21.15.98:0/327010764 learned_addr learned my addr 172.21.15.98:0/327010764 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ae8139b10 auth_method 0 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae67fc640 1 --2- 172.21.15.98:0/327010764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae8139ef0 0x7f1ae80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_done global_id 24938 payload 274 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 -- 172.21.15.98:0/327010764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae8139ef0 msgr2=0x7f1ae80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 --2- 172.21.15.98:0/327010764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae8139ef0 0x7f1ae80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 -- 172.21.15.98:0/327010764 >> v1:172.21.15.190:6789/0 conn(0x7f1ae813a2d0 legacy=0x7f1ae80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 monclient: found mon.noname-e 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 -- 172.21.15.98:0/327010764 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae80a9620 con 0x7f1ae8139b10 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae6ffd640 1 --2- 172.21.15.98:0/327010764 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1ae8139b10 0x7f1ae8072600 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0098d0 tx=0x7f1adc01c260 comp rx=0 tx=0).ready entity=mon.8 client_cookie=cc9db25d57dec9e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 1 -- 172.21.15.98:0/327010764 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1adc0b0070 con 0x7f1ae8139b10 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: dump: 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.665 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:09.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 1 -- 172.21.15.98:0/327010764 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1adc0aa810 con 0x7f1ae8139b10 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 5 monclient: authenticate success, global_id 24938 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 1 -- 172.21.15.98:0/327010764 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1adc0a9300 con 0x7f1ae8139b10 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 4 set_mon_vals no callback set 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1ae5ffb640 10 monclient: dump: 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.667 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.668 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 10 monclient: shutdown 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- 172.21.15.98:0/327010764 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1ae8139b10 msgr2=0x7f1ae8072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 --2- 172.21.15.98:0/327010764 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1ae8139b10 0x7f1ae8072600 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0098d0 tx=0x7f1adc01c260 comp rx=0 tx=0).stop 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- 172.21.15.98:0/327010764 shutdown_connections 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 --2- 172.21.15.98:0/327010764 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae8139ef0 0x7f1ae80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 --2- 172.21.15.98:0/327010764 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1ae8139b10 0x7f1ae8072600 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- 172.21.15.98:0/327010764 >> 172.21.15.98:0/327010764 conn(0x7f1ae80ac790 msgr2=0x7f1ae80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- 172.21.15.98:0/327010764 shutdown_connections 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.651+0000 7f1aed2e2640 1 -- 172.21.15.98:0/327010764 wait complete. 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: build_initial_monmap 2024-09-11T03:03:09.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: monmap: 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:09.670 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 librados: starting msgr at 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 librados: starting objecter 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 Processor -- start 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 -- start start 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 librados: setting wanted keys 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 librados: calling monclient init 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: init 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: _add_conns ranks=[6,2,3,5,0,1,7,4,8] 2024-09-11T03:03:09.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1ae8139b10 0x7f1ae8133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): picked mon.noname-d con 0x7f1ae8139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ae8139ef0 0x7f1ae8133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): picked mon.noname-b con 0x7f1ae8139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae813a2d0 0x7f1ae81341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): picked mon.noname-g con 0x7f1ae813a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae8144470 con 0x7f1ae8139ef0 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae80a8870 con 0x7f1ae813a2d0 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1ae81445f0 con 0x7f1ae8139b10 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:09.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): authenticate will time out at 1743.537151s 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1ae8139b10 0x7f1ae8133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ae8139ef0 0x7f1ae8133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1ae8139b10 0x7f1ae8133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34314/0 (socket says 172.21.15.98:34314) 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 1 -- 172.21.15.98:0/4161519778 learned_addr learned my addr 172.21.15.98:0/4161519778 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ae8139b10 auth_method 0 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): get_auth_request con 0x7f1ae8139ef0 auth_method 0 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae77fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae813a2d0 0x7f1ae81341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae77fe640 10 monclient(hunting): get_auth_request con 0x7f1ae813a2d0 auth_method 0 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient(hunting): handle_auth_done global_id 64577 payload 824 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1ae8139b10 msgr2=0x7f1ae8133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1ae8139b10 0x7f1ae8133760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae813a2d0 msgr2=0x7f1ae81341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f1ae813a2d0 0x7f1ae81341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient: _finish_hunting 0 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 monclient: found mon.noname-b 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 -- 172.21.15.98:0/4161519778 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae80aa2a0 con 0x7f1ae8139ef0 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1ae8139b10 0x7f1ae8133760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae67fc640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ae8139ef0 0x7f1ae8133ca0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c7f0 tx=0x7f1ad800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ae8139ef0 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:09.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ad8015070 con 0x7f1ae8139ef0 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: dump: 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.676 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad8007a90 con 0x7f1ae8139ef0 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 5 monclient: authenticate success, global_id 64577 2024-09-11T03:03:09.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1ad8007c40 con 0x7f1ae8139ef0 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae57fa640 4 set_mon_vals no callback set 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 monclient: dump: 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.678 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: _renew_subs 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae813ccc0 con 0x7f1ae8139ef0 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 client.64577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: _renew_subs 2024-09-11T03:03:09.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae813c700 con 0x7f1ae8139ef0 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 librados: init done 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a0e60 0x7f1ae80a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae806ca10 con 0x7f1ae80a0e60 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a0e60 0x7f1ae80a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient: get_auth_request con 0x7f1ae80a0e60 auth_method 0 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_done global_id 44973 payload 824 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 1 -- 172.21.15.98:0/4161519778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ae806e680 con 0x7f1ae80a0e60 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1ad8007620 con 0x7f1ae8139ef0 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1ae6ffd640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a0e60 0x7f1ae80a1240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1adc01c090 tx=0x7f1adc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1ad403df40 0x7f1ad4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 client.64577.objecter ms_handle_connect 0x7f1ae80a0e60 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 client.64577.objecter resend_mon_ops 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(524..524 src has 1..524) ==== 6649+0+0 (secure 0 0 0) 0x7f1ad804f8f0 con 0x7f1ae8139ef0 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 10 client.64577.objecter ms_dispatch 0x7f1ae8000ba0 osd_map(524..524 src has 1..524) 2024-09-11T03:03:09.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 3 client.64577.objecter handle_osd_map got epochs [524,524] > 0 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.655+0000 7f1acf7fe640 3 client.64577.objecter handle_osd_map decoding full epoch 524 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 20 client.64577.objecter dump_active .. 0 homeless 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1adc0ba020 con 0x7f1ae80a0e60 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 10 monclient: dump: 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1adc0aa910 con 0x7f1ae80a0e60 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:09.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a0e60 msgr2=0x7f1ae80a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a0e60 0x7f1ae80a1240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1adc01c090 tx=0x7f1adc019000 comp rx=0 tx=0).stop 2024-09-11T03:03:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a0e60 msgr2=0x7f1ae80a1240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1ae77fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1ad403df40 0x7f1ad4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a0e60 0x7f1ae80a1240 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1adc01c090 tx=0x7f1adc019000 comp rx=0 tx=0).stop 2024-09-11T03:03:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1ae77fe640 10 monclient: get_auth_request con 0x7f1ad403df40 auth_method 0 2024-09-11T03:03:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1acf7fe640 10 client.64577.objecter ms_handle_connect 0x7f1ad403df40 2024-09-11T03:03:09.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.659+0000 7f1ae77fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1ad403df40 0x7f1ad4040400 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f1ae813d490 tx=0x7f1ae0009410 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a1780 0x7f1ae813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae806e870 con 0x7f1ae80a1780 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a1780 0x7f1ae813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient: get_auth_request con 0x7f1ae80a1780 auth_method 0 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 10 monclient(hunting): handle_auth_done global_id 44982 payload 824 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 1 -- 172.21.15.98:0/4161519778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ae8072600 con 0x7f1ae80a1780 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 10 client.64577.objecter ms_handle_connect 0x7f1ae80a1780 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 10 client.64577.objecter resend_mon_ops 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae6ffd640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a1780 0x7f1ae813a6b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0b8000 tx=0x7f1adc0a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1adc0ba050 con 0x7f1ae80a1780 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 10 monclient: dump: 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f1adc003620 con 0x7f1ae80a1780 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a1780 msgr2=0x7f1ae813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a1780 0x7f1ae813a6b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0b8000 tx=0x7f1adc0a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a1780 msgr2=0x7f1ae813a6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1acf7fe640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1ae80a1780 0x7f1ae813a6b0 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1adc0b8000 tx=0x7f1adc0a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:09.689 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 416102, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:09.690 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:09.691 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:09.692 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:09.693 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:09.694 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:09.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.697 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.698 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.707 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.709 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:09.711 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:09.713 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 10 librados: watch_flush enter 2024-09-11T03:03:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 10 librados: watch_flush exit 2024-09-11T03:03:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 20 client.64577.objecter shutdown clearing up homeless session... 2024-09-11T03:03:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 10 client.64577.objecter successfully canceled tick 2024-09-11T03:03:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1ad403df40 msgr2=0x7f1ad4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f1ad403df40 0x7f1ad4040400 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f1ae813d490 tx=0x7f1ae0009410 comp rx=0 tx=0).stop 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 10 monclient: shutdown 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ae8139ef0 msgr2=0x7f1ae8133ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 --2- 172.21.15.98:0/4161519778 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1ae8139ef0 0x7f1ae8133ca0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800c7f0 tx=0x7f1ad800ccb0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae77fe640 1 -- 172.21.15.98:0/4161519778 reap_dead start 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1ae77fe640 1 -- 172.21.15.98:0/4161519778 reap_dead start 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 shutdown_connections 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 >> 172.21.15.98:0/4161519778 conn(0x7f1ae80ac790 msgr2=0x7f1ae80ab490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 shutdown_connections 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 -- 172.21.15.98:0/4161519778 wait complete. 2024-09-11T03:03:09.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.663+0000 7f1aed2e2640 1 librados: shutdown 2024-09-11T03:03:09.717 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:09.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T03:03:09.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.831+0000 7f918b9fc640 10 monclient: get_monmap_and_config 2024-09-11T03:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.831+0000 7f918b9fc640 10 monclient: build_initial_monmap 2024-09-11T03:03:09.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient: monmap: 2024-09-11T03:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:09.837256+0000 2024-09-11T03:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:09.837256+0000 2024-09-11T03:03:09.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:09.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:09.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:09.839 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:09.839 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 1 Processor -- start 2024-09-11T03:03:09.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 1 -- start start 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient: init 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient: _add_conns ranks=[2,1,10,3,9,0,5,4,7,8,6] 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9184076b00 0x7f91840a1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient(hunting): picked mon.noname-b con 0x7f9184076b00 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9184076ee0 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f918413eba0 0x7f9184140f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient(hunting): picked mon.noname-f con 0x7f918413eba0 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9184073f90 con 0x7f9184076ee0 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f91840a67e0 con 0x7f918413eba0 2024-09-11T03:03:09.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f91840a0410 con 0x7f9184076b00 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f918b9fc640 10 monclient(hunting): authenticate will time out at 1743.717550s 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9184076b00 0x7f91840a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9184076b00 0x7f91840a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40230/0 (socket says 172.21.15.98:40230) 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 -- 172.21.15.98:0/1615208841 learned_addr learned my addr 172.21.15.98:0/1615208841 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): get_auth_request con 0x7f9184076b00 auth_method 0 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient(hunting): handle_auth_done global_id 64586 payload 274 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189f99640 1 --2- 172.21.15.98:0/1615208841 >> v2:172.21.15.190:3300/0 conn(0x7f918413eba0 0x7f9184140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 -- 172.21.15.98:0/1615208841 >> v2:172.21.15.190:3300/0 conn(0x7f918413eba0 msgr2=0x7f9184140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 --2- 172.21.15.98:0/1615208841 >> v2:172.21.15.190:3300/0 conn(0x7f918413eba0 0x7f9184140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 -- 172.21.15.98:0/1615208841 >> v1:172.21.15.190:6789/0 conn(0x7f9184076ee0 legacy=0x7f9184077300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient: _finish_hunting 0 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 monclient: found mon.noname-b 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:03:09.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189798640 1 -- 172.21.15.98:0/1615208841 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91840a4ad0 con 0x7f9184076b00 2024-09-11T03:03:09.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189f99640 10 monclient: get_auth_request con 0x7f918413eba0 auth_method 0 2024-09-11T03:03:09.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.835+0000 7f9189f99640 1 --2- 172.21.15.98:0/1615208841 >> v2:172.21.15.190:3300/0 conn(0x7f918413eba0 0x7f9184140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:09.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189798640 1 --2- 172.21.15.98:0/1615208841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9184076b00 0x7f91840a1990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9184077bc0 tx=0x7f918001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f980cb5f87507a2b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 1 -- 172.21.15.98:0/1615208841 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f91800b0070 con 0x7f9184076b00 2024-09-11T03:03:09.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:03:09.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: dump: 2024-09-11T03:03:09.863 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.863 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.863 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.866 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.866 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.866 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.866 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.866 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: _finish_auth 0 2024-09-11T03:03:09.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:09.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:09.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 5 monclient: authenticate success, global_id 64586 2024-09-11T03:03:09.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 1 -- 172.21.15.98:0/1615208841 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91800aa810 con 0x7f9184076b00 2024-09-11T03:03:09.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:09.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:09.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:09.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 1 -- 172.21.15.98:0/1615208841 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f91800a9300 con 0x7f9184076b00 2024-09-11T03:03:09.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 4 set_mon_vals no callback set 2024-09-11T03:03:09.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:09.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f917bfff640 10 monclient: dump: 2024-09-11T03:03:09.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.871 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.871 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.871 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.871 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.872 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.872 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.872 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.872 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.872 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient: shutdown 2024-09-11T03:03:09.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:09.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- 172.21.15.98:0/1615208841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9184076b00 msgr2=0x7f91840a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 --2- 172.21.15.98:0/1615208841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9184076b00 0x7f91840a1990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9184077bc0 tx=0x7f918001c400 comp rx=0 tx=0).stop 2024-09-11T03:03:09.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- 172.21.15.98:0/1615208841 shutdown_connections 2024-09-11T03:03:09.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 --2- 172.21.15.98:0/1615208841 >> v2:172.21.15.190:3300/0 conn(0x7f918413eba0 0x7f9184140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 --2- 172.21.15.98:0/1615208841 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f9184076b00 0x7f91840a1990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- 172.21.15.98:0/1615208841 >> 172.21.15.98:0/1615208841 conn(0x7f918406f100 msgr2=0x7f918406f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:09.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- 172.21.15.98:0/1615208841 shutdown_connections 2024-09-11T03:03:09.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- 172.21.15.98:0/1615208841 wait complete. 2024-09-11T03:03:09.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient: build_initial_monmap 2024-09-11T03:03:09.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient: monmap: 2024-09-11T03:03:09.874 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:09.875 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:09.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:09.877 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 librados: starting msgr at 2024-09-11T03:03:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 librados: starting objecter 2024-09-11T03:03:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 Processor -- start 2024-09-11T03:03:09.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- start start 2024-09-11T03:03:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 librados: setting wanted keys 2024-09-11T03:03:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 librados: calling monclient init 2024-09-11T03:03:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient: init 2024-09-11T03:03:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient: _add_conns ranks=[3,1,8,6,2,0,4,5,7] 2024-09-11T03:03:09.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9184076b00 0x7f9184137940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): picked mon.noname-g con 0x7f9184076b00 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9184076ee0 0x7f9184137e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): picked mon.noname-f con 0x7f9184076ee0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f918413eba0 0x7f91841383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): picked mon.noname-e con 0x7f918413eba0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9184144080 con 0x7f9184076ee0 2024-09-11T03:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9184143f00 con 0x7f9184076b00 2024-09-11T03:03:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9184144200 con 0x7f918413eba0 2024-09-11T03:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f918b9fc640 10 monclient(hunting): authenticate will time out at 1743.723061s 2024-09-11T03:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f918413eba0 0x7f91841383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f918413eba0 0x7f91841383c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52328/0 (socket says 172.21.15.98:52328) 2024-09-11T03:03:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 -- 172.21.15.98:0/2967803861 learned_addr learned my addr 172.21.15.98:0/2967803861 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): get_auth_request con 0x7f918413eba0 auth_method 0 2024-09-11T03:03:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9188f97640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9184076ee0 0x7f9184137e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9188f97640 10 monclient(hunting): get_auth_request con 0x7f9184076ee0 auth_method 0 2024-09-11T03:03:09.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9188f97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9188f97640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9188f97640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189798640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9184076b00 0x7f9184137940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189798640 10 monclient(hunting): get_auth_request con 0x7f9184076b00 auth_method 0 2024-09-11T03:03:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189798640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189798640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189798640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient(hunting): handle_auth_done global_id 24947 payload 824 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9184076b00 msgr2=0x7f9184137940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9184076b00 0x7f9184137940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9184076ee0 msgr2=0x7f9184137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9184076ee0 0x7f9184137e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient: _finish_hunting 0 2024-09-11T03:03:09.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 monclient: found mon.noname-e 2024-09-11T03:03:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 -- 172.21.15.98:0/2967803861 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9184072d40 con 0x7f918413eba0 2024-09-11T03:03:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9188f97640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9184076ee0 0x7f9184137e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:03:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9189f99640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f918413eba0 0x7f91841383c0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f917c00cc60 tx=0x7f917c00a650 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 10 client.?.objecter ms_handle_connect 0x7f918413eba0 2024-09-11T03:03:09.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f917c00f8f0 con 0x7f918413eba0 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 10 monclient: dump: 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.887 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.888 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.888 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.888 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.888 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.888 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.888 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.889 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.889 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.889 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.889 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.889 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.839+0000 7f9179ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 5 monclient: authenticate success, global_id 24947 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f917c00faf0 con 0x7f918413eba0 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 10 monclient: _renew_subs 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918413c550 con 0x7f918413eba0 2024-09-11T03:03:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:09.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f917c00fca0 con 0x7f918413eba0 2024-09-11T03:03:09.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f917b7fe640 4 set_mon_vals no callback set 2024-09-11T03:03:09.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:09.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 10 client.24947.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:09.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: dump: 2024-09-11T03:03:09.891 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.892 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.892 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.892 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.892 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.892 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.893 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.893 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.893 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.893 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.893 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 10 monclient: _renew_subs 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f918413b960 con 0x7f918413eba0 2024-09-11T03:03:09.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 1 librados: init done 2024-09-11T03:03:09.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:09.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409ca50 0x7f918409ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f918406ca10 con 0x7f918409ca50 2024-09-11T03:03:09.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f917c00f480 con 0x7f918413eba0 2024-09-11T03:03:09.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409ca50 0x7f918409ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient: get_auth_request con 0x7f918409ca50 auth_method 0 2024-09-11T03:03:09.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f916803df40 0x7f9168040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 10 monclient(hunting): handle_auth_done global_id 64595 payload 824 2024-09-11T03:03:09.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 1 -- 172.21.15.98:0/2967803861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91840a7460 con 0x7f918409ca50 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(524..524 src has 1..524) ==== 6649+0+0 (secure 0 0 0) 0x7f917c050280 con 0x7f918413eba0 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 client.24947.objecter ms_dispatch 0x7f9184000ba0 osd_map(524..524 src has 1..524) 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 3 client.24947.objecter handle_osd_map got epochs [524,524] > 0 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 3 client.24947.objecter handle_osd_map decoding full epoch 524 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 20 client.24947.objecter dump_active .. 0 homeless 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9189798640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409ca50 0x7f918409ce30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f91840a4a40 tx=0x7f91800191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 client.24947.objecter ms_handle_connect 0x7f918409ca50 2024-09-11T03:03:09.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 client.24947.objecter resend_mon_ops 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f91800b9030 con 0x7f918409ca50 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9188f97640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f916803df40 0x7f9168040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: dump: 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.899 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9188f97640 10 monclient: get_auth_request con 0x7f916803df40 auth_method 0 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f91800b1040 con 0x7f918409ca50 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:09.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409ca50 msgr2=0x7f918409ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409ca50 0x7f918409ce30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f91840a4a40 tx=0x7f91800191a0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409ca50 msgr2=0x7f918409ce30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409ca50 0x7f918409ce30 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f91840a4a40 tx=0x7f91800191a0 comp rx=0 tx=0).stop 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9179ffb640 10 client.24947.objecter ms_handle_connect 0x7f916803df40 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.843+0000 7f9188f97640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f916803df40 0x7f9168040400 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7f9184072350 tx=0x7f9174009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409d370 0x7f9184139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f918406e870 con 0x7f918409d370 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409d370 0x7f9184139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient: get_auth_request con 0x7f918409d370 auth_method 0 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:09.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 10 monclient(hunting): handle_auth_done global_id 64604 payload 824 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 1 -- 172.21.15.98:0/2967803861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f91840a6f30 con 0x7f918409d370 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189798640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409d370 0x7f9184139af0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f91800097c0 tx=0x7f91800a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 10 client.24947.objecter ms_handle_connect 0x7f918409d370 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 10 client.24947.objecter resend_mon_ops 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f91800b9040 con 0x7f918409d370 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:09.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 10 monclient: dump: 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:09.903 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f91800be460 con 0x7f918409d370 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409d370 msgr2=0x7f9184139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409d370 0x7f9184139af0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f91800097c0 tx=0x7f91800a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409d370 msgr2=0x7f9184139af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:09.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9179ffb640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f918409d370 0x7f9184139af0 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f91800097c0 tx=0x7f91800a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 282044, 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:09.905 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:09.906 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.907 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:09.908 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:09.909 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:09.910 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.911 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.915 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:09.916 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:09.917 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.918 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.920 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.921 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.922 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.923 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.924 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:09.925 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:09.926 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:09.927 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:09.928 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:09.929 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 10 librados: watch_flush enter 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 10 librados: watch_flush exit 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 20 client.24947.objecter shutdown clearing up homeless session... 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 10 client.24947.objecter successfully canceled tick 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f916803df40 msgr2=0x7f9168040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f916803df40 0x7f9168040400 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7f9184072350 tx=0x7f9174009040 comp rx=0 tx=0).stop 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 10 monclient: shutdown 2024-09-11T03:03:09.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f918413eba0 msgr2=0x7f91841383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 --2- 172.21.15.98:0/2967803861 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f918413eba0 0x7f91841383c0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f917c00cc60 tx=0x7f917c00a650 comp rx=0 tx=0).stop 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189f99640 1 -- 172.21.15.98:0/2967803861 reap_dead start 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f9189f99640 1 -- 172.21.15.98:0/2967803861 reap_dead start 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 shutdown_connections 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 >> 172.21.15.98:0/2967803861 conn(0x7f918406f100 msgr2=0x7f91841393a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 shutdown_connections 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 -- 172.21.15.98:0/2967803861 wait complete. 2024-09-11T03:03:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:09.851+0000 7f918b9fc640 1 librados: shutdown 2024-09-11T03:03:09.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:09.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T03:03:09.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:03:10.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient: get_monmap_and_config 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient: monmap: 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:10.058535+0000 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:10.058535+0000 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.057 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:10.058 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:10.058 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 Processor -- start 2024-09-11T03:03:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 -- start start 2024-09-11T03:03:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient: init 2024-09-11T03:03:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient: _add_conns ranks=[8,6,5,4,10,3,7,2,0,1,9] 2024-09-11T03:03:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16b40a46a0 0x7f16b40a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): picked mon.noname-e con 0x7f16b40a46a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a13a0 0x7f16b40a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): picked mon.noname-d con 0x7f16b40a13a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f16b4136450 0x7f16b41368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): picked mon.noname-h con 0x7f16b4136450 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f16b40a5500 con 0x7f16b4136450 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f16b40a11a0 con 0x7f16b40a13a0 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f16b40a28a0 con 0x7f16b40a46a0 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ba666640 10 monclient(hunting): authenticate will time out at 1743.939441s 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a13a0 0x7f16b40a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16b40a46a0 0x7f16b40a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16b40a46a0 0x7f16b40a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52334/0 (socket says 172.21.15.98:52334) 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 1 -- 172.21.15.98:0/51693709 learned_addr learned my addr 172.21.15.98:0/51693709 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a13a0 0x7f16b40a4fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34330/0 (socket says 172.21.15.98:34330) 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 10 monclient(hunting): get_auth_request con 0x7f16b40a46a0 auth_method 0 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): get_auth_request con 0x7f16b40a13a0 auth_method 0 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b8c03640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f16b4136450 0x7f16b41368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b8c03640 10 monclient(hunting): get_auth_request con 0x7f16b4136450 auth_method 0 2024-09-11T03:03:10.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b8c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b8c03640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b8c03640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient(hunting): handle_auth_done global_id 84138 payload 274 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 -- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16b40a46a0 msgr2=0x7f16b40a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16b40a46a0 0x7f16b40a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 -- 172.21.15.98:0/51693709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f16b4136450 msgr2=0x7f16b41368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f16b4136450 0x7f16b41368b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 monclient: found mon.noname-d 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16ab7fe640 1 -- 172.21.15.98:0/51693709 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b40a25c0 con 0x7f16b40a13a0 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.055+0000 7f16b3fff640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16b40a46a0 0x7f16b40a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ab7fe640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a13a0 0x7f16b40a4fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f169c0097f0 tx=0x7f169c01c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6d677e7915a43271 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 1 -- 172.21.15.98:0/51693709 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f169c0b0070 con 0x7f16b40a13a0 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: dump: 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.073 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 5 monclient: authenticate success, global_id 84138 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 1 -- 172.21.15.98:0/51693709 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f169c0aa810 con 0x7f16b40a13a0 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 1 -- 172.21.15.98:0/51693709 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f169c0a9300 con 0x7f16b40a13a0 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 4 set_mon_vals no callback set 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16b37fe640 10 monclient: dump: 2024-09-11T03:03:10.075 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.076 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 10 monclient: shutdown 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 -- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a13a0 msgr2=0x7f16b40a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a13a0 0x7f16b40a4fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f169c0097f0 tx=0x7f169c01c3a0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 -- 172.21.15.98:0/51693709 shutdown_connections 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f16b4136450 0x7f16b41368b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a13a0 0x7f16b40a4fc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 --2- 172.21.15.98:0/51693709 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f16b40a46a0 0x7f16b40a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 -- 172.21.15.98:0/51693709 >> 172.21.15.98:0/51693709 conn(0x7f16b406ed00 msgr2=0x7f16b406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 -- 172.21.15.98:0/51693709 shutdown_connections 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 -- 172.21.15.98:0/51693709 wait complete. 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 10 monclient: monmap: 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:10.078 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 librados: starting msgr at 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 librados: starting objecter 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 Processor -- start 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 -- start start 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 librados: setting wanted keys 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 1 librados: calling monclient init 2024-09-11T03:03:10.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.059+0000 7f16ba666640 10 monclient: init 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient: _add_conns ranks=[3,6,0,2,8,5,4,1,7] 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16b40a13a0 0x7f16b4134b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): picked mon.noname-g con 0x7f16b40a13a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a46a0 0x7f16b4135070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): picked mon.noname-d con 0x7f16b40a46a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4136450 0x7f16b41355b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): picked mon.noname-a con 0x7f16b4136450 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41402c0 con 0x7f16b4136450 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f16b4140140 con 0x7f16b40a13a0 2024-09-11T03:03:10.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f16b4140440 con 0x7f16b40a46a0 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): authenticate will time out at 1743.944522s 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a46a0 0x7f16b4135070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4136450 0x7f16b41355b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4136450 0x7f16b41355b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32868/0 (socket says 172.21.15.98:32868) 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 -- 172.21.15.98:0/793432768 learned_addr learned my addr 172.21.15.98:0/793432768 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): get_auth_request con 0x7f16b4136450 auth_method 0 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a46a0 0x7f16b4135070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34342/0 (socket says 172.21.15.98:34342) 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient(hunting): get_auth_request con 0x7f16b40a46a0 auth_method 0 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16b40a13a0 0x7f16b4134b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): get_auth_request con 0x7f16b40a13a0 auth_method 0 2024-09-11T03:03:10.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient(hunting): handle_auth_done global_id 44991 payload 824 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a46a0 msgr2=0x7f16b4135070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f16b40a46a0 0x7f16b4135070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16b40a13a0 msgr2=0x7f16b4134b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f16b40a13a0 0x7f16b4134b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 monclient: found mon.noname-a 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 -- 172.21.15.98:0/793432768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b40a86c0 con 0x7f16b4136450 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b8c03640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4136450 0x7f16b41355b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16ac007db0 tx=0x7f16ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 client.?.objecter ms_handle_connect 0x7f16b4136450 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16ac00edd0 con 0x7f16b4136450 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: dump: 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.084 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.085 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ac019070 con 0x7f16b4136450 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 5 monclient: authenticate success, global_id 44991 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient: _renew_subs 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b4123d50 con 0x7f16b4136450 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16ac014d00 con 0x7f16b4136450 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: dump: 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.087 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b2ffd640 4 set_mon_vals no callback set 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 client.44991.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient: _renew_subs 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b4072580 con 0x7f16b4136450 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 librados: init done 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b413b710 0x7f16b41376e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f16b406ca10 con 0x7f16b413b710 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b413b710 0x7f16b41376e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient: get_auth_request con 0x7f16b413b710 auth_method 0 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f16ac014420 con 0x7f16b4136450 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f169403df40 0x7f1694040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 10 monclient(hunting): handle_auth_done global_id 24448 payload 824 2024-09-11T03:03:10.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 1 -- 172.21.15.98:0/793432768 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16b413c940 con 0x7f16b413b710 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(524..524 src has 1..524) ==== 6649+0+0 (secure 0 0 0) 0x7f16ac04ffd0 con 0x7f16b4136450 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 client.44991.objecter ms_dispatch 0x7f16b4000ba0 osd_map(524..524 src has 1..524) 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 3 client.44991.objecter handle_osd_map got epochs [524,524] > 0 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 3 client.44991.objecter handle_osd_map decoding full epoch 524 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 20 client.44991.objecter dump_active .. 0 homeless 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 client.44991.objecter ms_handle_connect 0x7f16b413b710 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 client.44991.objecter resend_mon_ops 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b3fff640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b413b710 0x7f16b41376e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f16a4011600 tx=0x7f16a4011ac0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16a400fba0 con 0x7f16b413b710 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: dump: 2024-09-11T03:03:10.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.091 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f169403df40 0x7f1694040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 10 monclient: get_auth_request con 0x7f169403df40 auth_method 0 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f16a400f730 con 0x7f16b413b710 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b413b710 msgr2=0x7f16b41376e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b413b710 0x7f16b41376e0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f16a4011600 tx=0x7f16a4011ac0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b413b710 msgr2=0x7f16b41376e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b413b710 0x7f16b41376e0 secure :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f16a4011600 tx=0x7f16a4011ac0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16b17fa640 10 client.44991.objecter ms_handle_connect 0x7f169403df40 2024-09-11T03:03:10.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.063+0000 7f16ab7fe640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f169403df40 0x7f1694040400 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7f169c0ad000 tx=0x7f169c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b4137c20 0x7f16b4138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f16b406e870 con 0x7f16b4137c20 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b4137c20 0x7f16b4138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient: get_auth_request con 0x7f16b4137c20 auth_method 0 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 10 monclient(hunting): handle_auth_done global_id 24457 payload 824 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 1 -- 172.21.15.98:0/793432768 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16b4071790 con 0x7f16b4137c20 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b3fff640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b4137c20 0x7f16b4138030 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f16a4000c10 tx=0x7f16a40046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 10 client.44991.objecter ms_handle_connect 0x7f16b4137c20 2024-09-11T03:03:10.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 10 client.44991.objecter resend_mon_ops 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16a40064b0 con 0x7f16b4137c20 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 10 monclient: dump: 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.096 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f16a4006040 con 0x7f16b4137c20 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b4137c20 msgr2=0x7f16b4138030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b4137c20 0x7f16b4138030 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f16a4000c10 tx=0x7f16a40046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b4137c20 msgr2=0x7f16b4138030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b17fa640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f16b4137c20 0x7f16b4138030 secure :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f16a4000c10 tx=0x7f16a40046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 805855, 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:10.098 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:10.099 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:10.100 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.101 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:10.102 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.103 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.104 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.105 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.106 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.107 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:10.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.114 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.118 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.121 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:10.122 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 10 librados: watch_flush enter 2024-09-11T03:03:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 10 librados: watch_flush exit 2024-09-11T03:03:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 20 client.44991.objecter shutdown clearing up homeless session... 2024-09-11T03:03:10.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 10 client.44991.objecter successfully canceled tick 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f169403df40 msgr2=0x7f1694040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f169403df40 0x7f1694040400 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7f169c0ad000 tx=0x7f169c0a6000 comp rx=0 tx=0).stop 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 10 monclient: shutdown 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4136450 msgr2=0x7f16b41355b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16b8c03640 1 -- 172.21.15.98:0/793432768 reap_dead start 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4136450 0x7f16b41355b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16ac007db0 tx=0x7f16ac00e4b0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 shutdown_connections 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 --2- 172.21.15.98:0/793432768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f16b4136450 0x7f16b41355b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 >> 172.21.15.98:0/793432768 conn(0x7f16b406ed00 msgr2=0x7f16b40a7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 shutdown_connections 2024-09-11T03:03:10.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 -- 172.21.15.98:0/793432768 wait complete. 2024-09-11T03:03:10.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.071+0000 7f16ba666640 1 librados: shutdown 2024-09-11T03:03:10.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:10.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T03:03:10.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:03:10.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient: get_monmap_and_config 2024-09-11T03:03:10.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient: monmap: 2024-09-11T03:03:10.240 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:10.242272+0000 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:10.242272+0000 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.241 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.242 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.242 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.242 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:10.242 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:10.242 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 1 Processor -- start 2024-09-11T03:03:10.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 1 -- start start 2024-09-11T03:03:10.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient: init 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient: _add_conns ranks=[1,4,7,3,10,6,0,9,2,8,5] 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0d100a7aa0 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d100a12a0 0x7f0d100a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient(hunting): picked mon.noname-c con 0x7f0d100a12a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1013ea90 0x7f0d10140e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient(hunting): picked mon.noname-i con 0x7f0d1013ea90 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d100a1720 con 0x7f0d100a7aa0 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0d100a2b30 con 0x7f0d100a12a0 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0d100a27c0 con 0x7f0d1013ea90 2024-09-11T03:03:10.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d17766640 10 monclient(hunting): authenticate will time out at 1744.122563s 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d100a12a0 0x7f0d100a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d100a12a0 0x7f0d100a8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42098/0 (socket says 172.21.15.98:42098) 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 -- 172.21.15.98:0/257896665 learned_addr learned my addr 172.21.15.98:0/257896665 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): get_auth_request con 0x7f0d100a12a0 auth_method 0 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d15d03640 1 --2- 172.21.15.98:0/257896665 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1013ea90 0x7f0d10140e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d15d03640 10 monclient(hunting): get_auth_request con 0x7f0d1013ea90 auth_method 0 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d15d03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d15d03640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d15d03640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient(hunting): handle_auth_done global_id 24466 payload 274 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 -- 172.21.15.98:0/257896665 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1013ea90 msgr2=0x7f0d10140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 --2- 172.21.15.98:0/257896665 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1013ea90 0x7f0d10140e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 -- 172.21.15.98:0/257896665 >> v1:172.21.15.190:6789/0 conn(0x7f0d100a7aa0 legacy=0x7f0d100a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 monclient: found mon.noname-c 2024-09-11T03:03:10.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:10.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.239+0000 7f0d14d01640 1 -- 172.21.15.98:0/257896665 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d100a9590 con 0x7f0d100a12a0 2024-09-11T03:03:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 1 --2- 172.21.15.98:0/257896665 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d100a12a0 0x7f0d100a8590 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0d000098a0 tx=0x7f0d0001c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=88af3068286b8dcf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 1 -- 172.21.15.98:0/257896665 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d000b0070 con 0x7f0d100a12a0 2024-09-11T03:03:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: dump: 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.255 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 1 -- 172.21.15.98:0/257896665 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d000aa810 con 0x7f0d100a12a0 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 1 -- 172.21.15.98:0/257896665 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d000a9300 con 0x7f0d100a12a0 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d07fff640 10 monclient: dump: 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.257 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 5 monclient: authenticate success, global_id 24466 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 4 set_mon_vals no callback set 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient: shutdown 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- 172.21.15.98:0/257896665 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d100a12a0 msgr2=0x7f0d100a8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 --2- 172.21.15.98:0/257896665 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d100a12a0 0x7f0d100a8590 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0d000098a0 tx=0x7f0d0001c440 comp rx=0 tx=0).stop 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- 172.21.15.98:0/257896665 shutdown_connections 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 --2- 172.21.15.98:0/257896665 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d1013ea90 0x7f0d10140e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 --2- 172.21.15.98:0/257896665 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d100a12a0 0x7f0d100a8590 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- 172.21.15.98:0/257896665 >> 172.21.15.98:0/257896665 conn(0x7f0d100ac690 msgr2=0x7f0d100a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- 172.21.15.98:0/257896665 shutdown_connections 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- 172.21.15.98:0/257896665 wait complete. 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient: monmap: 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:10.259 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 librados: starting msgr at 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 librados: starting objecter 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 Processor -- start 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- start start 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 librados: setting wanted keys 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 librados: calling monclient init 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient: init 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient: _add_conns ranks=[7,6,2,5,8,4,0,3,1] 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d100a12a0 0x7f0d10126d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): picked mon.noname-i con 0x7f0d100a12a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:10.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d100a7aa0 0x7f0d10124be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): picked mon.noname-d con 0x7f0d100a7aa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d1013ea90 0x7f0d1013dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): picked mon.noname-b con 0x7f0d1013ea90 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0d10143f70 con 0x7f0d1013ea90 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0d10143df0 con 0x7f0d100a7aa0 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0d101440f0 con 0x7f0d100a12a0 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d17766640 10 monclient(hunting): authenticate will time out at 1744.127843s 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d100a7aa0 0x7f0d10124be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d15d03640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d1013ea90 0x7f0d1013dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d100a7aa0 0x7f0d10124be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34346/0 (socket says 172.21.15.98:34346) 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 1 -- 172.21.15.98:0/202499822 learned_addr learned my addr 172.21.15.98:0/202499822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 10 monclient(hunting): get_auth_request con 0x7f0d100a7aa0 auth_method 0 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.243+0000 7f0d14d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient(hunting): get_auth_request con 0x7f0d1013ea90 auth_method 0 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d100a12a0 0x7f0d10126d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): get_auth_request con 0x7f0d100a12a0 auth_method 0 2024-09-11T03:03:10.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 10 monclient(hunting): handle_auth_done global_id 84156 payload 824 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d100a12a0 msgr2=0x7f0d10126d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0d100a12a0 0x7f0d10126d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d1013ea90 msgr2=0x7f0d1013dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d1013ea90 0x7f0d1013dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 1 monclient: found mon.noname-d 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 1 -- 172.21.15.98:0/202499822 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d100a4fa0 con 0x7f0d100a7aa0 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d1013ea90 0x7f0d1013dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d14d01640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d100a7aa0 0x7f0d10124be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d0001aaf0 tx=0x7f0d000191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 client.?.objecter ms_handle_connect 0x7f0d100a7aa0 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d000b9070 con 0x7f0d100a7aa0 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: dump: 2024-09-11T03:03:10.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.267 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.268 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d000044b0 con 0x7f0d100a7aa0 2024-09-11T03:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 5 monclient: authenticate success, global_id 84156 2024-09-11T03:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 10 monclient: _renew_subs 2024-09-11T03:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:10.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d10126210 con 0x7f0d100a7aa0 2024-09-11T03:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d000b0040 con 0x7f0d100a7aa0 2024-09-11T03:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:10.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: dump: 2024-09-11T03:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.272 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.274 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.274 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.274 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.274 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.274 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d077fe640 4 set_mon_vals no callback set 2024-09-11T03:03:10.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 10 client.84156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:10.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 10 monclient: _renew_subs 2024-09-11T03:03:10.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:10.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d1013e120 con 0x7f0d100a7aa0 2024-09-11T03:03:10.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 1 librados: init done 2024-09-11T03:03:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1170 0x7f0d101d1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0d101281f0 con 0x7f0d101d1170 2024-09-11T03:03:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0d00004040 con 0x7f0d100a7aa0 2024-09-11T03:03:10.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1170 0x7f0d101d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient: get_auth_request con 0x7f0d101d1170 auth_method 0 2024-09-11T03:03:10.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0cf403df40 0x7f0cf4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 10 monclient(hunting): handle_auth_done global_id 84165 payload 824 2024-09-11T03:03:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 1 -- 172.21.15.98:0/202499822 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d100a54f0 con 0x7f0d101d1170 2024-09-11T03:03:10.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(524..524 src has 1..524) ==== 6649+0+0 (secure 0 0 0) 0x7f0d000e8ad0 con 0x7f0d100a7aa0 2024-09-11T03:03:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 client.84156.objecter ms_dispatch 0x7f0d10000ba0 osd_map(524..524 src has 1..524) 2024-09-11T03:03:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 3 client.84156.objecter handle_osd_map got epochs [524,524] > 0 2024-09-11T03:03:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 3 client.84156.objecter handle_osd_map decoding full epoch 524 2024-09-11T03:03:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 20 client.84156.objecter dump_active .. 0 homeless 2024-09-11T03:03:10.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 client.84156.objecter ms_handle_connect 0x7f0d101d1170 2024-09-11T03:03:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 client.84156.objecter resend_mon_ops 2024-09-11T03:03:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15502640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1170 0x7f0d101d1550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e500 tx=0x7f0d0c00e9c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d0c0052e0 con 0x7f0d101d1170 2024-09-11T03:03:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0cf403df40 0x7f0cf4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: dump: 2024-09-11T03:03:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.282 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.282 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.282 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.282 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.282 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.283 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 10 monclient: get_auth_request con 0x7f0cf403df40 auth_method 0 2024-09-11T03:03:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d0c018070 con 0x7f0d101d1170 2024-09-11T03:03:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:10.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1170 msgr2=0x7f0d101d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1170 0x7f0d101d1550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e500 tx=0x7f0d0c00e9c0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1170 msgr2=0x7f0d101d1550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1170 0x7f0d101d1550 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e500 tx=0x7f0d0c00e9c0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d15d03640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0cf403df40 0x7f0cf4040400 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f0d100a6400 tx=0x7f0d08009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.247+0000 7f0d05ffb640 10 client.84156.objecter ms_handle_connect 0x7f0cf403df40 2024-09-11T03:03:10.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:10.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1a90 0x7f0d10071710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0d1006ca10 con 0x7f0d101d1a90 2024-09-11T03:03:10.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1a90 0x7f0d10071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient: get_auth_request con 0x7f0d101d1a90 auth_method 0 2024-09-11T03:03:10.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 10 monclient(hunting): handle_auth_done global_id 84174 payload 824 2024-09-11T03:03:10.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 1 -- 172.21.15.98:0/202499822 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d100a7e80 con 0x7f0d101d1a90 2024-09-11T03:03:10.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15502640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1a90 0x7f0d10071710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e500 tx=0x7f0d0c000ee0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 10 client.84156.objecter ms_handle_connect 0x7f0d101d1a90 2024-09-11T03:03:10.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 10 client.84156.objecter resend_mon_ops 2024-09-11T03:03:10.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d0c0076d0 con 0x7f0d101d1a90 2024-09-11T03:03:10.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 10 monclient: dump: 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.293 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f0d0c007260 con 0x7f0d101d1a90 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1a90 msgr2=0x7f0d10071710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1a90 0x7f0d10071710 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e500 tx=0x7f0d0c000ee0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1a90 msgr2=0x7f0d10071710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d05ffb640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d101d1a90 0x7f0d10071710 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00e500 tx=0x7f0d0c000ee0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 16067, 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:10.295 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:10.296 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.297 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:10.299 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.300 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:10.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.307 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.308 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:10.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:10.314 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.315 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.316 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.319 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:10.322 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.324 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:10.324 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:10.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 10 librados: watch_flush enter 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 10 librados: watch_flush exit 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 20 client.84156.objecter shutdown clearing up homeless session... 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 10 client.84156.objecter successfully canceled tick 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0cf403df40 msgr2=0x7f0cf4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0cf403df40 0x7f0cf4040400 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f0d100a6400 tx=0x7f0d08009780 comp rx=0 tx=0).stop 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 10 monclient: shutdown 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d100a7aa0 msgr2=0x7f0d10124be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d100a7aa0 0x7f0d10124be0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d0001aaf0 tx=0x7f0d000191e0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d15d03640 1 -- 172.21.15.98:0/202499822 reap_dead start 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 shutdown_connections 2024-09-11T03:03:10.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 --2- 172.21.15.98:0/202499822 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0d100a7aa0 0x7f0d10124be0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 >> 172.21.15.98:0/202499822 conn(0x7f0d100ac690 msgr2=0x7f0d100a3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 shutdown_connections 2024-09-11T03:03:10.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 -- 172.21.15.98:0/202499822 wait complete. 2024-09-11T03:03:10.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.255+0000 7f0d17766640 1 librados: shutdown 2024-09-11T03:03:10.327 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:10.327 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T03:03:10.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:03:10.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient: get_monmap_and_config 2024-09-11T03:03:10.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient: monmap: 2024-09-11T03:03:10.451 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.451 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:10.453397+0000 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:10.453397+0000 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.452 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.453 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.453 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.453 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:10.453 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:10.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 Processor -- start 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 -- start start 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient: init 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient: _add_conns ranks=[5,7,4,6,1,3,2,8,0,9,10] 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f867c139b10 0x7f867c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): picked mon.noname-h con 0x7f867c139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f867c139ef0 0x7f867c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): picked mon.noname-i con 0x7f867c139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:10.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f867c13a2d0 0x7f867c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): picked mon.noname-c con 0x7f867c13a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f867c0702b0 con 0x7f867c13a2d0 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f867c071bf0 con 0x7f867c139b10 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f867c071350 con 0x7f867c139ef0 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8683c79640 10 monclient(hunting): authenticate will time out at 1744.333694s 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f867c13a2d0 0x7f867c0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f867c13a2d0 0x7f867c0a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42100/0 (socket says 172.21.15.98:42100) 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 -- 172.21.15.98:0/1409274035 learned_addr learned my addr 172.21.15.98:0/1409274035 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): get_auth_request con 0x7f867c13a2d0 auth_method 0 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681a15640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f867c139b10 0x7f867c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681214640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f867c139ef0 0x7f867c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681a15640 10 monclient(hunting): get_auth_request con 0x7f867c139b10 auth_method 0 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681a15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681a15640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681a15640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681214640 10 monclient(hunting): get_auth_request con 0x7f867c139ef0 auth_method 0 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681214640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8681214640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient(hunting): handle_auth_done global_id 24475 payload 274 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 -- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f867c139ef0 msgr2=0x7f867c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f867c139ef0 0x7f867c0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 -- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f867c139b10 msgr2=0x7f867c072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f867c139b10 0x7f867c072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 monclient: found mon.noname-c 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.451+0000 7f8682216640 1 -- 172.21.15.98:0/1409274035 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867c0a9620 con 0x7f867c13a2d0 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f867c13a2d0 0x7f867c0a7f50 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f8674009e80 tx=0x7f867401c910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=fd8eb50851e7842e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 1 -- 172.21.15.98:0/1409274035 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86740090d0 con 0x7f867c13a2d0 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: dump: 2024-09-11T03:03:10.468 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 1 -- 172.21.15.98:0/1409274035 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86740a6970 con 0x7f867c13a2d0 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 1 -- 172.21.15.98:0/1409274035 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86740af480 con 0x7f867c13a2d0 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8680a13640 10 monclient: dump: 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 5 monclient: authenticate success, global_id 24475 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 4 set_mon_vals no callback set 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: shutdown 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- 172.21.15.98:0/1409274035 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f867c13a2d0 msgr2=0x7f867c0a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f867c13a2d0 0x7f867c0a7f50 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f8674009e80 tx=0x7f867401c910 comp rx=0 tx=0).stop 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- 172.21.15.98:0/1409274035 shutdown_connections 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f867c13a2d0 0x7f867c0a7f50 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f867c139ef0 0x7f867c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 --2- 172.21.15.98:0/1409274035 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f867c139b10 0x7f867c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- 172.21.15.98:0/1409274035 >> 172.21.15.98:0/1409274035 conn(0x7f867c0ac790 msgr2=0x7f867c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- 172.21.15.98:0/1409274035 shutdown_connections 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- 172.21.15.98:0/1409274035 wait complete. 2024-09-11T03:03:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: monmap: 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 librados: starting msgr at 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 librados: starting objecter 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 Processor -- start 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- start start 2024-09-11T03:03:10.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 librados: setting wanted keys 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 librados: calling monclient init 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: init 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: _add_conns ranks=[0,3,8,4,6,7,5,2,1] 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f867c139b10 0x7f867c13a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): picked mon.noname-a con 0x7f867c139b10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f867c139ef0 0x7f867c13abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): picked mon.noname-g con 0x7f867c139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c13a2d0 0x7f867c13b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): picked mon.noname-e con 0x7f867c13a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:10.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f867c143460 con 0x7f867c139b10 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f867c0a88a0 con 0x7f867c139ef0 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f867c1435e0 con 0x7f867c13a2d0 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient(hunting): authenticate will time out at 1744.338926s 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f867c139b10 0x7f867c13a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c13a2d0 0x7f867c13b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f867c139b10 0x7f867c13a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32872/0 (socket says 172.21.15.98:32872) 2024-09-11T03:03:10.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): get_auth_request con 0x7f867c13a2d0 auth_method 0 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 1 -- 172.21.15.98:0/1548800248 learned_addr learned my addr 172.21.15.98:0/1548800248 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 10 monclient(hunting): get_auth_request con 0x7f867c139b10 auth_method 0 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681214640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f867c139ef0 0x7f867c13abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681214640 10 monclient(hunting): get_auth_request con 0x7f867c139ef0 auth_method 0 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681214640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681214640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient(hunting): handle_auth_done global_id 24965 payload 824 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f867c139ef0 msgr2=0x7f867c13abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f867c139ef0 0x7f867c13abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f867c139b10 msgr2=0x7f867c13a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f867c139b10 0x7f867c13a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 monclient: found mon.noname-e 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 -- 172.21.15.98:0/1548800248 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f867c1403e0 con 0x7f867c13a2d0 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681a15640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f867c139b10 0x7f867c13a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8681214640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f867c139ef0 0x7f867c13abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:03:10.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8682216640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c13a2d0 0x7f867c13b130 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f867c070430 tx=0x7f86740196b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 client.?.objecter ms_handle_connect 0x7f867c13a2d0 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8674009270 con 0x7f867c13a2d0 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: dump: 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.481 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 5 monclient: authenticate success, global_id 24965 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: _renew_subs 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86740a67b0 con 0x7f867c13a2d0 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f867c0a9f00 con 0x7f867c13a2d0 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8674003e10 con 0x7f867c13a2d0 2024-09-11T03:03:10.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f8673fff640 4 set_mon_vals no callback set 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 10 client.24965.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.455+0000 7f86727fc640 10 monclient: dump: 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.484 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 10 monclient: _renew_subs 2024-09-11T03:03:10.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f867c13cee0 con 0x7f867c13a2d0 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 1 librados: init done 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c133760 0x7f867c133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f867c13c9a0 con 0x7f867c133760 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f86740039a0 con 0x7f867c13a2d0 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c133760 0x7f867c133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient: get_auth_request con 0x7f867c133760 auth_method 0 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 10 monclient(hunting): handle_auth_done global_id 24974 payload 824 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 1 -- 172.21.15.98:0/1548800248 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f867c06e660 con 0x7f867c133760 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f866003de70 0x7f8660040330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(524..524 src has 1..524) ==== 6649+0+0 (secure 0 0 0) 0x7f86740b5030 con 0x7f867c13a2d0 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 client.24965.objecter ms_dispatch 0x7f867c000ba0 osd_map(524..524 src has 1..524) 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 3 client.24965.objecter handle_osd_map got epochs [524,524] > 0 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 3 client.24965.objecter handle_osd_map decoding full epoch 524 2024-09-11T03:03:10.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681a15640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c133760 0x7f867c133b40 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f866c010cb0 tx=0x7f866c00e5a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 20 client.24965.objecter dump_active .. 0 homeless 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 client.24965.objecter ms_handle_connect 0x7f867c133760 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 client.24965.objecter resend_mon_ops 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f866c018070 con 0x7f867c133760 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 monclient: dump: 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.488 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.489 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681214640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f866003de70 0x7f8660040330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681214640 10 monclient: get_auth_request con 0x7f866003de70 auth_method 0 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f866c01d450 con 0x7f867c133760 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c133760 msgr2=0x7f867c133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c133760 0x7f867c133b40 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f866c010cb0 tx=0x7f866c00e5a0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c133760 msgr2=0x7f867c133b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c133760 0x7f867c133b40 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f86727fc640 10 client.24965.objecter ms_handle_connect 0x7f866003de70 2024-09-11T03:03:10.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.459+0000 7f8681214640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f866003de70 0x7f8660040330 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7f867c0a5380 tx=0x7f8678008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8683c79640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8683c79640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c134080 0x7f867c0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8683c79640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f867c06ca10 con 0x7f867c134080 2024-09-11T03:03:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8681a15640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c134080 0x7f867c0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8681a15640 10 monclient: get_auth_request con 0x7f867c134080 auth_method 0 2024-09-11T03:03:10.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8681a15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8681a15640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.463+0000 7f8681a15640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8681a15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8681a15640 10 monclient(hunting): handle_auth_done global_id 24983 payload 824 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8681a15640 1 -- 172.21.15.98:0/1548800248 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f867c072600 con 0x7f867c134080 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 10 client.24965.objecter ms_handle_connect 0x7f867c134080 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 10 client.24965.objecter resend_mon_ops 2024-09-11T03:03:10.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8681a15640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c134080 0x7f867c0a0e60 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f866c010cb0 tx=0x7f866c0046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f866c004530 con 0x7f867c134080 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 10 monclient: dump: 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f866c01f2e0 con 0x7f867c134080 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:10.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c134080 msgr2=0x7f867c0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c134080 0x7f867c0a0e60 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f866c010cb0 tx=0x7f866c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c134080 msgr2=0x7f867c0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f86727fc640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c134080 0x7f867c0a0e60 secure :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f866c010cb0 tx=0x7f866c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:10.497 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 806259, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:10.498 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:10.500 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:10.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.503 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.522 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.523 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.525 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:10.525 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:10.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 10 librados: watch_flush enter 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 10 librados: watch_flush exit 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 20 client.24965.objecter shutdown clearing up homeless session... 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 10 client.24965.objecter successfully canceled tick 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f866003de70 msgr2=0x7f8660040330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f866003de70 0x7f8660040330 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7f867c0a5380 tx=0x7f8678008040 comp rx=0 tx=0).stop 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 10 monclient: shutdown 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c13a2d0 msgr2=0x7f867c13b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8682216640 1 -- 172.21.15.98:0/1548800248 reap_dead start 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c13a2d0 0x7f867c13b130 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f867c070430 tx=0x7f86740196b0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 shutdown_connections 2024-09-11T03:03:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 --2- 172.21.15.98:0/1548800248 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f867c13a2d0 0x7f867c13b130 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 >> 172.21.15.98:0/1548800248 conn(0x7f867c0ac790 msgr2=0x7f867c0aa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 shutdown_connections 2024-09-11T03:03:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 -- 172.21.15.98:0/1548800248 wait complete. 2024-09-11T03:03:10.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.467+0000 7f8683c79640 1 librados: shutdown 2024-09-11T03:03:10.528 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:10.528 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T03:03:10.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.651+0000 7f17fdb40640 10 monclient: get_monmap_and_config 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.651+0000 7f17fdb40640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.651+0000 7f17fdb40640 10 monclient: monmap: 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:10.655984+0000 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:10.655984+0000 2024-09-11T03:03:10.654 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.655 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:10.656 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.651+0000 7f17fdb40640 1 Processor -- start 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.651+0000 7f17fdb40640 1 -- start start 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.651+0000 7f17fdb40640 10 monclient: init 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient: _add_conns ranks=[4,2,3,9,8,5,7,0,6,1,10] 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17f8139b80 0x7f17f8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): picked mon.noname-c con 0x7f17f8139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f806fe50 0x7f17f80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): picked mon.noname-b con 0x7f17f806fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f8072640 0x7f17f80a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): picked mon.noname-g con 0x7f17f8072640 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f17f80719c0 con 0x7f17f806fe50 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f17f8071c10 con 0x7f17f8072640 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f17f80702d0 con 0x7f17f8139b80 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient(hunting): authenticate will time out at 1744.536287s 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f806fe50 0x7f17f80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17f8139b80 0x7f17f8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17f8139b80 0x7f17f8139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42112/0 (socket says 172.21.15.98:42112) 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f806fe50 0x7f17f80a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40272/0 (socket says 172.21.15.98:40272) 2024-09-11T03:03:10.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 -- 172.21.15.98:0/2910111232 learned_addr learned my addr 172.21.15.98:0/2910111232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): get_auth_request con 0x7f17f8139b80 auth_method 0 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 10 monclient(hunting): get_auth_request con 0x7f17f806fe50 auth_method 0 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f7fff640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f8072640 0x7f17f80a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f7fff640 10 monclient(hunting): get_auth_request con 0x7f17f8072640 auth_method 0 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient(hunting): handle_auth_done global_id 24484 payload 274 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 -- 172.21.15.98:0/2910111232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f8072640 msgr2=0x7f17f80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f8072640 0x7f17f80a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 -- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f806fe50 msgr2=0x7f17f80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f806fe50 0x7f17f80a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 monclient: found mon.noname-c 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:10.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 -- 172.21.15.98:0/2910111232 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f8071370 con 0x7f17f8139b80 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17effff640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f806fe50 0x7f17f80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f77fe640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17f8139b80 0x7f17f8139f60 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f17e80098d0 tx=0x7f17e801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=9de0bed724a1de48 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 1 -- 172.21.15.98:0/2910111232 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e80b0070 con 0x7f17f8139b80 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: dump: 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 1 -- 172.21.15.98:0/2910111232 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e80aa810 con 0x7f17f8139b80 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 5 monclient: authenticate success, global_id 24484 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 1 -- 172.21.15.98:0/2910111232 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e80a9300 con 0x7f17f8139b80 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 4 set_mon_vals no callback set 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17f6ffd640 10 monclient: dump: 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 10 monclient: shutdown 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17f8139b80 msgr2=0x7f17f8139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17f8139b80 0x7f17f8139f60 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f17e80098d0 tx=0x7f17e801c440 comp rx=0 tx=0).stop 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- 172.21.15.98:0/2910111232 shutdown_connections 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f17f8072640 0x7f17f80a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f17f806fe50 0x7f17f80a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 --2- 172.21.15.98:0/2910111232 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f17f8139b80 0x7f17f8139f60 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- 172.21.15.98:0/2910111232 >> 172.21.15.98:0/2910111232 conn(0x7f17f80ac730 msgr2=0x7f17f80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- 172.21.15.98:0/2910111232 shutdown_connections 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.655+0000 7f17fdb40640 1 -- 172.21.15.98:0/2910111232 wait complete. 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: monmap: 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.674 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 librados: starting msgr at 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 librados: starting objecter 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 Processor -- start 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 -- start start 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 librados: setting wanted keys 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 librados: calling monclient init 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: init 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: _add_conns ranks=[7,5,0,6,1,8,3,2,4] 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f17f806fe50 0x7f17f8134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): picked mon.noname-i con 0x7f17f806fe50 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f17f8072640 0x7f17f8135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): picked mon.noname-h con 0x7f17f8072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8139b80 0x7f17f813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): picked mon.noname-a con 0x7f17f8139b80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81435c0 con 0x7f17f8139b80 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f17f8143440 con 0x7f17f8072640 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f17f8143740 con 0x7f17f806fe50 2024-09-11T03:03:10.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): authenticate will time out at 1744.541459s 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8139b80 0x7f17f813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8139b80 0x7f17f813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32886/0 (socket says 172.21.15.98:32886) 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 -- 172.21.15.98:0/24655739 learned_addr learned my addr 172.21.15.98:0/24655739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): get_auth_request con 0x7f17f8139b80 auth_method 0 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17effff640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f17f8072640 0x7f17f8135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient(hunting): handle_auth_done global_id 45009 payload 824 2024-09-11T03:03:10.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f17f806fe50 msgr2=0x7f17f8134d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f17f806fe50 0x7f17f8134d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f17f8072640 msgr2=0x7f17f8135260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f17f8072640 0x7f17f8135260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 monclient: found mon.noname-a 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 -- 172.21.15.98:0/24655739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f8135910 con 0x7f17f8139b80 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f7fff640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8139b80 0x7f17f813a6b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f17f0007e30 tx=0x7f17f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 client.?.objecter ms_handle_connect 0x7f17f8139b80 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:10.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17f000ee30 con 0x7f17f8139b80 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: dump: 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.679 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 5 monclient: authenticate success, global_id 45009 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f0019070 con 0x7f17f8139b80 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17f0014d00 con 0x7f17f8139b80 2024-09-11T03:03:10.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f67fc640 4 set_mon_vals no callback set 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 monclient: dump: 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.681 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: _renew_subs 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f80aa410 con 0x7f17f8139b80 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 client.45009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: _renew_subs 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f813eb70 con 0x7f17f8139b80 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 librados: init done 2024-09-11T03:03:10.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813abf0 0x7f17f813afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17f806ca10 con 0x7f17f813abf0 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f17f0014420 con 0x7f17f8139b80 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813abf0 0x7f17f813afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient: get_auth_request con 0x7f17f813abf0 auth_method 0 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f17d803dea0 0x7f17d8040360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(525..525 src has 1..525) ==== 6276+0+0 (secure 0 0 0) 0x7f17f004ff80 con 0x7f17f8139b80 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 10 client.45009.objecter ms_dispatch 0x7f17f8000ba0 osd_map(525..525 src has 1..525) 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 3 client.45009.objecter handle_osd_map got epochs [525,525] > 0 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 3 client.45009.objecter handle_osd_map decoding full epoch 525 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f4ff9640 20 client.45009.objecter dump_active .. 0 homeless 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17effff640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f17d803dea0 0x7f17d8040360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.659+0000 7f17f77fe640 10 monclient(hunting): handle_auth_done global_id 25777 payload 824 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f77fe640 1 -- 172.21.15.98:0/24655739 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17f8139f60 con 0x7f17f813abf0 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17effff640 10 monclient: get_auth_request con 0x7f17d803dea0 auth_method 0 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 client.45009.objecter ms_handle_connect 0x7f17f813abf0 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f77fe640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813abf0 0x7f17f813afd0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f17e80098a0 tx=0x7f17e8009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 client.45009.objecter resend_mon_ops 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e80ba020 con 0x7f17f813abf0 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 monclient: dump: 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 client.45009.objecter ms_handle_connect 0x7f17d803dea0 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17effff640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f17d803dea0 0x7f17d8040360 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009580 tx=0x7f17e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17e8003620 con 0x7f17f813abf0 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813abf0 msgr2=0x7f17f813afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813abf0 0x7f17f813afd0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f17e80098a0 tx=0x7f17e8009040 comp rx=0 tx=0).stop 2024-09-11T03:03:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813abf0 msgr2=0x7f17f813afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.663+0000 7f17f4ff9640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813abf0 0x7f17f813afd0 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.667+0000 7f17fdb40640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.667+0000 7f17fdb40640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813b510 0x7f17f80a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.667+0000 7f17fdb40640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.667+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17f806e870 con 0x7f17f813b510 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.667+0000 7f17f77fe640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813b510 0x7f17f80a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient: get_auth_request con 0x7f17f813b510 auth_method 0 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 10 monclient(hunting): handle_auth_done global_id 25786 payload 824 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 1 -- 172.21.15.98:0/24655739 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17f8069090 con 0x7f17f813b510 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f77fe640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813b510 0x7f17f80a0e60 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c10 tx=0x7f17e80a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 10 client.45009.objecter ms_handle_connect 0x7f17f813b510 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 10 client.45009.objecter resend_mon_ops 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e80ba030 con 0x7f17f813b510 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 10 monclient: dump: 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.690 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f17e80b6b80 con 0x7f17f813b510 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:10.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813b510 msgr2=0x7f17f80a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813b510 0x7f17f80a0e60 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c10 tx=0x7f17e80a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813b510 msgr2=0x7f17f80a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f4ff9640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f813b510 0x7f17f80a0e60 secure :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c10 tx=0x7f17e80a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:10.692 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 806446, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.693 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.694 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.695 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:10.696 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.697 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.698 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.705 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:10.706 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.709 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.710 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:10.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:10.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.712 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:10.716 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.717 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:10.718 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 10 librados: watch_flush enter 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 10 librados: watch_flush exit 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 20 client.45009.objecter shutdown clearing up homeless session... 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 10 client.45009.objecter successfully canceled tick 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f17d803dea0 msgr2=0x7f17d8040360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f17d803dea0 0x7f17d8040360 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009580 tx=0x7f17e0008040 comp rx=0 tx=0).stop 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 10 monclient: shutdown 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8139b80 msgr2=0x7f17f813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 --2- 172.21.15.98:0/24655739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f17f8139b80 0x7f17f813a6b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f17f0007e30 tx=0x7f17f000e4b0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f7fff640 1 -- 172.21.15.98:0/24655739 reap_dead start 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17f7fff640 1 -- 172.21.15.98:0/24655739 reap_dead start 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 shutdown_connections 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 >> 172.21.15.98:0/24655739 conn(0x7f17f80ac730 msgr2=0x7f17f80a7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 shutdown_connections 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 -- 172.21.15.98:0/24655739 wait complete. 2024-09-11T03:03:10.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.671+0000 7f17fdb40640 1 librados: shutdown 2024-09-11T03:03:10.721 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:10.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T03:03:10.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:03:10.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient: get_monmap_and_config 2024-09-11T03:03:10.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient: monmap: 2024-09-11T03:03:10.836 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.836 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:10.838422+0000 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:10.838422+0000 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.837 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.838 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.838 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.838 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.838 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:10.838 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:10.838 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 Processor -- start 2024-09-11T03:03:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 -- start start 2024-09-11T03:03:10.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient: init 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient: _add_conns ranks=[10,7,5,2,1,8,4,0,9,6,3] 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f68b0139b10 0x7f68b0072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): picked mon.noname-f con 0x7f68b0139b10 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f68b0139ef0 0x7f68b00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): picked mon.noname-i con 0x7f68b0139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f68b013a2d0 0x7f68b00a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): picked mon.noname-h con 0x7f68b013a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f68b00702b0 con 0x7f68b0139b10 2024-09-11T03:03:10.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f68b0071bf0 con 0x7f68b013a2d0 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f68b0071350 con 0x7f68b0139ef0 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68b5682640 10 monclient(hunting): authenticate will time out at 1744.719515s 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68aeffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f68b0139b10 0x7f68b0072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68af7fe640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f68b013a2d0 0x7f68b00a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68ae7fc640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f68b0139ef0 0x7f68b00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.835+0000 7f68aeffd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f68b0139b10 0x7f68b0072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:54926/0 (socket says 172.21.15.98:54926) 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 -- 172.21.15.98:0/3450527808 learned_addr learned my addr 172.21.15.98:0/3450527808 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 10 monclient(hunting): get_auth_request con 0x7f68b013a2d0 auth_method 0 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): get_auth_request con 0x7f68b0139b10 auth_method 0 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68ae7fc640 10 monclient(hunting): get_auth_request con 0x7f68b0139ef0 auth_method 0 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient(hunting): handle_auth_done global_id 25795 payload 274 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 -- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f68b0139ef0 msgr2=0x7f68b00a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f68b0139ef0 0x7f68b00a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 -- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f68b013a2d0 msgr2=0x7f68b00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f68b013a2d0 0x7f68b00a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 monclient: found mon.noname-f 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:03:10.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 -- 172.21.15.98:0/3450527808 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b00a9620 con 0x7f68b0139b10 2024-09-11T03:03:10.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68af7fe640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f68b013a2d0 0x7f68b00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:10.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68aeffd640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68b0139b10 0x7f68b0072600 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098a0 tx=0x7f68a001c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74dc897722196183 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 1 -- 172.21.15.98:0/3450527808 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68a00b0070 con 0x7f68b0139b10 2024-09-11T03:03:10.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:03:10.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: dump: 2024-09-11T03:03:10.856 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.857 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 5 monclient: authenticate success, global_id 25795 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 1 -- 172.21.15.98:0/3450527808 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a00aa3a0 con 0x7f68b0139b10 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 4 set_mon_vals no callback set 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 1 -- 172.21.15.98:0/3450527808 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68a00aaa40 con 0x7f68b0139b10 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68adffb640 10 monclient: dump: 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.859 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 10 monclient: shutdown 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 -- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68b0139b10 msgr2=0x7f68b0072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68b0139b10 0x7f68b0072600 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f68a00098a0 tx=0x7f68a001c490 comp rx=0 tx=0).stop 2024-09-11T03:03:10.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 -- 172.21.15.98:0/3450527808 shutdown_connections 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f68b013a2d0 0x7f68b00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f68b0139ef0 0x7f68b00a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 --2- 172.21.15.98:0/3450527808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68b0139b10 0x7f68b0072600 unknown :-1 s=CLOSED pgs=2393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 -- 172.21.15.98:0/3450527808 >> 172.21.15.98:0/3450527808 conn(0x7f68b00ac790 msgr2=0x7f68b00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 -- 172.21.15.98:0/3450527808 shutdown_connections 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.839+0000 7f68b5682640 1 -- 172.21.15.98:0/3450527808 wait complete. 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: build_initial_monmap 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: monmap: 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:10.861 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 librados: starting msgr at 2024-09-11T03:03:10.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 librados: starting objecter 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 Processor -- start 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 -- start start 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 librados: setting wanted keys 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 librados: calling monclient init 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: init 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: _add_conns ranks=[6,0,8,3,2,7,4,5,1] 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f68b0139b10 0x7f68b0133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): picked mon.noname-d con 0x7f68b0139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68b0139ef0 0x7f68b01340f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): picked mon.noname-a con 0x7f68b0139ef0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f68b013a2d0 0x7f68b013a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): picked mon.noname-e con 0x7f68b013a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f68b0143460 con 0x7f68b0139ef0 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f68b00a88a0 con 0x7f68b0139b10 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f68b01435e0 con 0x7f68b013a2d0 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): authenticate will time out at 1744.725460s 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f68b0139b10 0x7f68b0133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68b0139ef0 0x7f68b01340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f68b013a2d0 0x7f68b013a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f68b0139b10 0x7f68b0133bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34366/0 (socket says 172.21.15.98:34366) 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 1 -- 172.21.15.98:0/1755620489 learned_addr learned my addr 172.21.15.98:0/1755620489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): get_auth_request con 0x7f68b013a2d0 auth_method 0 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): get_auth_request con 0x7f68b0139b10 auth_method 0 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68b0139ef0 0x7f68b01340f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32902/0 (socket says 172.21.15.98:32902) 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 10 monclient(hunting): get_auth_request con 0x7f68b0139ef0 auth_method 0 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient(hunting): handle_auth_done global_id 24992 payload 824 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f68b0139b10 msgr2=0x7f68b0133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f68b0139b10 0x7f68b0133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68b0139ef0 msgr2=0x7f68b01340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f68b0139ef0 0x7f68b01340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 monclient: found mon.noname-e 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 -- 172.21.15.98:0/1755620489 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68b0134630 con 0x7f68b013a2d0 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68af7fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f68b013a2d0 0x7f68b013a780 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009db0 tx=0x7f68a800c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 client.?.objecter ms_handle_connect 0x7f68b013a2d0 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:10.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68a8018030 con 0x7f68b013a2d0 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: dump: 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.868 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a800cd30 con 0x7f68b013a2d0 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 5 monclient: authenticate success, global_id 24992 2024-09-11T03:03:10.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68a8014d60 con 0x7f68b013a2d0 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: _renew_subs 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68b013d430 con 0x7f68b013a2d0 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 monclient: dump: 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68ad7fa640 4 set_mon_vals no callback set 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 client.24992.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:10.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: _renew_subs 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f68b00a5a00 con 0x7f68b013a2d0 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 librados: init done 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013bc30 0x7f68b013acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f68b006ca10 con 0x7f68b013bc30 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f68a8014480 con 0x7f68b013a2d0 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f689c03df40 0x7f689c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013bc30 0x7f68b013acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(525..525 src has 1..525) ==== 6276+0+0 (secure 0 0 0) 0x7f68a8019070 con 0x7f68b013a2d0 2024-09-11T03:03:10.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 10 client.24992.objecter ms_dispatch 0x7f68b0000ba0 osd_map(525..525 src has 1..525) 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 3 client.24992.objecter handle_osd_map got epochs [525,525] > 0 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 3 client.24992.objecter handle_osd_map decoding full epoch 525 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient: get_auth_request con 0x7f68b013bc30 auth_method 0 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68977fe640 20 client.24992.objecter dump_active .. 0 homeless 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.843+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68ae7fc640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f689c03df40 0x7f689c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68ae7fc640 10 monclient: get_auth_request con 0x7f689c03df40 auth_method 0 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68aeffd640 10 monclient(hunting): handle_auth_done global_id 74298 payload 824 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68aeffd640 1 -- 172.21.15.98:0/1755620489 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68b006e680 con 0x7f68b013bc30 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68aeffd640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013bc30 0x7f68b013acc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68a001c2c0 tx=0x7f68a0019230 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 client.24992.objecter ms_handle_connect 0x7f68b013bc30 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 client.24992.objecter resend_mon_ops 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 client.24992.objecter ms_handle_connect 0x7f689c03df40 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68ae7fc640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f689c03df40 0x7f689c040400 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7f68b00a57b0 tx=0x7f68a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68a00b0040 con 0x7f68b013bc30 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 monclient: dump: 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.875 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f68a00093d0 con 0x7f68b013bc30 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013bc30 msgr2=0x7f68b013acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013bc30 0x7f68b013acc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68a001c2c0 tx=0x7f68a0019230 comp rx=0 tx=0).stop 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013bc30 msgr2=0x7f68b013acc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.847+0000 7f68977fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013bc30 0x7f68b013acc0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f68a001c2c0 tx=0x7f68a0019230 comp rx=0 tx=0).stop 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.851+0000 7f68b5682640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.851+0000 7f68b5682640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013b200 0x7f68b00a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.851+0000 7f68b5682640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.851+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f68b013b610 con 0x7f68b013b200 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013b200 0x7f68b00a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient: get_auth_request con 0x7f68b013b200 auth_method 0 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:10.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:10.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 10 monclient(hunting): handle_auth_done global_id 74307 payload 824 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 1 -- 172.21.15.98:0/1755620489 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f68b0072600 con 0x7f68b013b200 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68aeffd640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013b200 0x7f68b00a0e60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f68a0000c10 tx=0x7f68a00a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 10 client.24992.objecter ms_handle_connect 0x7f68b013b200 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 10 client.24992.objecter resend_mon_ops 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f68a00b5af0 con 0x7f68b013b200 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 10 monclient: dump: 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:10.879 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:10.880 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f68a00b5680 con 0x7f68b013b200 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013b200 msgr2=0x7f68b00a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013b200 0x7f68b00a0e60 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f68a0000c10 tx=0x7f68a00a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013b200 msgr2=0x7f68b00a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68977fe640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f68b013b200 0x7f68b00a0e60 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f68a0000c10 tx=0x7f68a00a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:10.881 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 147787, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:10.882 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.883 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.884 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.885 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:10.886 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:10.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:10.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.890 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.891 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:10.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.895 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:10.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.899 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.900 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.902 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:10.903 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.904 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.904 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.904 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:10.904 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:10.904 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.905 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.906 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:10.907 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:10.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 10 librados: watch_flush enter 2024-09-11T03:03:10.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 10 librados: watch_flush exit 2024-09-11T03:03:10.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 20 client.24992.objecter shutdown clearing up homeless session... 2024-09-11T03:03:10.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 10 client.24992.objecter successfully canceled tick 2024-09-11T03:03:10.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f689c03df40 msgr2=0x7f689c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f689c03df40 0x7f689c040400 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7f68b00a57b0 tx=0x7f68a4009190 comp rx=0 tx=0).stop 2024-09-11T03:03:10.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 10 monclient: shutdown 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f68b013a2d0 msgr2=0x7f68b013a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 --2- 172.21.15.98:0/1755620489 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f68b013a2d0 0x7f68b013a780 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009db0 tx=0x7f68a800c490 comp rx=0 tx=0).stop 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68af7fe640 1 -- 172.21.15.98:0/1755620489 reap_dead start 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68af7fe640 1 -- 172.21.15.98:0/1755620489 reap_dead start 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 shutdown_connections 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 >> 172.21.15.98:0/1755620489 conn(0x7f68b00ac790 msgr2=0x7f68b00a6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 shutdown_connections 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 -- 172.21.15.98:0/1755620489 wait complete. 2024-09-11T03:03:10.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:10.855+0000 7f68b5682640 1 librados: shutdown 2024-09-11T03:03:10.911 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:10.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T03:03:10.911 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:03:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient: get_monmap_and_config 2024-09-11T03:03:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient: build_initial_monmap 2024-09-11T03:03:11.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient: monmap: 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:11.037509+0000 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:11.037509+0000 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:11.036 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:11.037 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:11.037 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 1 Processor -- start 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 1 -- start start 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient: init 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient: _add_conns ranks=[1,8,0,3,7,6,10,2,5,9,4] 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f83d4139b80 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f83d406fe50 0x7f83d413ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient(hunting): picked mon.noname-e con 0x7f83d406fe50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83d4072640 addr v1:172.21.15.98:6789/0 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d4071370 con 0x7f83d4072640 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d40a9600 con 0x7f83d4139b80 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f83d40719c0 con 0x7f83d406fe50 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d9c7e640 10 monclient(hunting): authenticate will time out at 1744.917813s 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f83d406fe50 0x7f83d413ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f83d4072640 0x7f83d4140f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46592/0 (socket says 172.21.15.98:46592) 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d3fff640 1 -- 172.21.15.98:0/2936952790 learned_addr learned my addr 172.21.15.98:0/2936952790 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): get_auth_request con 0x7f83d406fe50 auth_method 0 2024-09-11T03:03:11.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_done global_id 25001 payload 274 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 1 -- 172.21.15.98:0/2936952790 >> v1:172.21.15.190:6789/0 conn(0x7f83d4139b80 legacy=0x7f83d4139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 1 -- 172.21.15.98:0/2936952790 >> v1:172.21.15.98:6789/0 conn(0x7f83d4072640 legacy=0x7f83d4140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 1 monclient: found mon.noname-e 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:11.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.035+0000 7f83d2ffd640 1 -- 172.21.15.98:0/2936952790 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d4144690 con 0x7f83d406fe50 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d2ffd640 1 --2- 172.21.15.98:0/2936952790 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f83d406fe50 0x7f83d413ea00 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0097f0 tx=0x7f83bc01c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=564d148c63cd353c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 1 -- 172.21.15.98:0/2936952790 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83bc0b0070 con 0x7f83d406fe50 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: dump: 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.050 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: _finish_auth 0 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: _check_auth_tickets 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 1 -- 172.21.15.98:0/2936952790 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc0aa810 con 0x7f83d406fe50 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 5 monclient: authenticate success, global_id 25001 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 1 -- 172.21.15.98:0/2936952790 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83bc0a9300 con 0x7f83d406fe50 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 4 set_mon_vals no callback set 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d27fc640 10 monclient: dump: 2024-09-11T03:03:11.052 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.053 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient: shutdown 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- 172.21.15.98:0/2936952790 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f83d406fe50 msgr2=0x7f83d413ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 --2- 172.21.15.98:0/2936952790 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f83d406fe50 0x7f83d413ea00 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f83bc0097f0 tx=0x7f83bc01c3a0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- 172.21.15.98:0/2936952790 shutdown_connections 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 --2- 172.21.15.98:0/2936952790 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f83d406fe50 0x7f83d413ea00 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- 172.21.15.98:0/2936952790 >> 172.21.15.98:0/2936952790 conn(0x7f83d40ac730 msgr2=0x7f83d40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- 172.21.15.98:0/2936952790 shutdown_connections 2024-09-11T03:03:11.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- 172.21.15.98:0/2936952790 wait complete. 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient: build_initial_monmap 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient: monmap: 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:11.055 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 librados: starting msgr at 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 librados: starting objecter 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 Processor -- start 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- start start 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 librados: setting wanted keys 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 librados: calling monclient init 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient: init 2024-09-11T03:03:11.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient: _add_conns ranks=[0,5,3,4,8,7,2,1,6] 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d406fe50 0x7f83d41d1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): picked mon.noname-a con 0x7f83d406fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4072640 0x7f83d41d1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): picked mon.noname-h con 0x7f83d4072640 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83d4139b80 0x7f83d413d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): picked mon.noname-g con 0x7f83d4139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f83d40702d0 con 0x7f83d406fe50 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f83d4145690 con 0x7f83d4139b80 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f83d4145810 con 0x7f83d4072640 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d9c7e640 10 monclient(hunting): authenticate will time out at 1744.923633s 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d406fe50 0x7f83d41d1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d406fe50 0x7f83d41d1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32914/0 (socket says 172.21.15.98:32914) 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 1 -- 172.21.15.98:0/246336097 learned_addr learned my addr 172.21.15.98:0/246336097 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 10 monclient(hunting): get_auth_request con 0x7f83d406fe50 auth_method 0 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.039+0000 7f83d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4072640 0x7f83d41d1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83d4139b80 0x7f83d413d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): get_auth_request con 0x7f83d4072640 auth_method 0 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 10 monclient(hunting): get_auth_request con 0x7f83d4139b80 auth_method 0 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 10 monclient(hunting): handle_auth_done global_id 45036 payload 824 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4072640 msgr2=0x7f83d41d1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4072640 0x7f83d41d1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83d4139b80 msgr2=0x7f83d413d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f83d4139b80 0x7f83d413d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 10 monclient: _finish_hunting 0 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 1 monclient: found mon.noname-a 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 1 -- 172.21.15.98:0/246336097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d40a6a00 con 0x7f83d406fe50 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d37fe640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d406fe50 0x7f83d41d1a40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f83c400c3f0 tx=0x7f83c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 client.?.objecter ms_handle_connect 0x7f83d406fe50 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83c4017070 con 0x7f83d406fe50 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: dump: 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: _finish_auth 0 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c4003600 con 0x7f83d406fe50 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 5 monclient: authenticate success, global_id 45036 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 10 monclient: _renew_subs 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d4141070 con 0x7f83d406fe50 2024-09-11T03:03:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83c4003c20 con 0x7f83d406fe50 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 10 client.45036.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: dump: 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d1ffb640 4 set_mon_vals no callback set 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 10 monclient: _renew_subs 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d40a7280 con 0x7f83d406fe50 2024-09-11T03:03:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 1 librados: init done 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f83c40037b0 con 0x7f83d406fe50 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4140c40 0x7f83d413e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f83d406ca10 con 0x7f83d4140c40 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f83b403df70 0x7f83b4040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(525..525 src has 1..525) ==== 6276+0+0 (secure 0 0 0) 0x7f83c404e8f0 con 0x7f83d406fe50 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 client.45036.objecter ms_dispatch 0x7f83d4000ba0 osd_map(525..525 src has 1..525) 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 3 client.45036.objecter handle_osd_map got epochs [525,525] > 0 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 3 client.45036.objecter handle_osd_map decoding full epoch 525 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4140c40 0x7f83d413e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient: get_auth_request con 0x7f83d4140c40 auth_method 0 2024-09-11T03:03:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 20 client.45036.objecter dump_active .. 0 homeless 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f83b403df70 0x7f83b4040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 10 monclient: get_auth_request con 0x7f83b403df70 auth_method 0 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_done global_id 74318 payload 824 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 1 -- 172.21.15.98:0/246336097 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83d4139f60 con 0x7f83d4140c40 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 client.45036.objecter ms_handle_connect 0x7f83d4140c40 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d2ffd640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4140c40 0x7f83d413e010 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83d40a9e00 tx=0x7f83bc019140 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 client.45036.objecter resend_mon_ops 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 client.45036.objecter ms_handle_connect 0x7f83b403df70 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83d3fff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f83b403df70 0x7f83b4040430 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0012f0 tx=0x7f83cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83bc0b9070 con 0x7f83d4140c40 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: dump: 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f83bc003ae0 con 0x7f83d4140c40 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:11.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4140c40 msgr2=0x7f83d413e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4140c40 0x7f83d413e010 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83d40a9e00 tx=0x7f83bc019140 comp rx=0 tx=0).stop 2024-09-11T03:03:11.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4140c40 msgr2=0x7f83d413e010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:11.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.043+0000 7f83cbfff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d4140c40 0x7f83d413e010 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f83d40a9e00 tx=0x7f83bc019140 comp rx=0 tx=0).stop 2024-09-11T03:03:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d9c7e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d9c7e640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d413e550 0x7f83d402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d9c7e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f83d406e870 con 0x7f83d413e550 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d413e550 0x7f83d402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient: get_auth_request con 0x7f83d413e550 auth_method 0 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 10 monclient(hunting): handle_auth_done global_id 74327 payload 824 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 1 -- 172.21.15.98:0/246336097 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f83d40a4df0 con 0x7f83d413e550 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83d2ffd640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d413e550 0x7f83d402d450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f83bc018fb0 tx=0x7f83bc0a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83cbfff640 10 client.45036.objecter ms_handle_connect 0x7f83d413e550 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83cbfff640 10 client.45036.objecter resend_mon_ops 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83bc0b9040 con 0x7f83d413e550 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83cbfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.051+0000 7f83cbfff640 10 monclient: dump: 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.072 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f83bc0bd400 con 0x7f83d413e550 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d413e550 msgr2=0x7f83d402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83cbfff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d413e550 0x7f83d402d450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f83bc018fb0 tx=0x7f83bc0a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:11.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83cbfff640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d413e550 msgr2=0x7f83d402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83cbfff640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f83d413e550 0x7f83d402d450 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f83bc018fb0 tx=0x7f83bc0a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 147983, 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:11.074 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:11.075 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.076 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:11.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:11.078 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:11.079 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:11.080 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:11.083 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:11.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:11.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.088 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.089 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:11.090 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:11.091 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.092 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.093 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:11.094 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.095 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:11.096 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.097 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:11.098 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 10 librados: watch_flush enter 2024-09-11T03:03:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 10 librados: watch_flush exit 2024-09-11T03:03:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 20 client.45036.objecter shutdown clearing up homeless session... 2024-09-11T03:03:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 10 client.45036.objecter successfully canceled tick 2024-09-11T03:03:11.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f83b403df70 msgr2=0x7f83b4040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f83b403df70 0x7f83b4040430 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0012f0 tx=0x7f83cc009040 comp rx=0 tx=0).stop 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 10 monclient: shutdown 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d406fe50 msgr2=0x7f83d41d1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 --2- 172.21.15.98:0/246336097 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83d406fe50 0x7f83d41d1a40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f83c400c3f0 tx=0x7f83c400c8b0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d3fff640 1 -- 172.21.15.98:0/246336097 reap_dead start 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d3fff640 1 -- 172.21.15.98:0/246336097 reap_dead start 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 shutdown_connections 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 >> 172.21.15.98:0/246336097 conn(0x7f83d40ac730 msgr2=0x7f83d40a7490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 shutdown_connections 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 -- 172.21.15.98:0/246336097 wait complete. 2024-09-11T03:03:11.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.055+0000 7f83d9c7e640 1 librados: shutdown 2024-09-11T03:03:11.101 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:11.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T03:03:11.101 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient: get_monmap_and_config 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient: build_initial_monmap 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient: monmap: 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:11.218192+0000 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:11.218192+0000 2024-09-11T03:03:11.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:11.217 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:11.218 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 Processor -- start 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 -- start start 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient: init 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient: _add_conns ranks=[7,2,4,6,5,0,8,3,10,9,1] 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f058809bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): picked mon.noname-i con 0x7f058809b8c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f058809df80 0x7f05880a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): picked mon.noname-b con 0x7f058809df80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:11.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05880a4320 0x7f05880a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): picked mon.noname-c con 0x7f05880a4320 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f058809d2c0 con 0x7f058809df80 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f058809d5a0 con 0x7f05880a4320 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f058809d880 con 0x7f058809b8c0 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058ea98640 10 monclient(hunting): authenticate will time out at 1745.099008s 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f058809df80 0x7f05880a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05880a4320 0x7f05880a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f058809df80 0x7f05880a3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40274/0 (socket says 172.21.15.98:40274) 2024-09-11T03:03:11.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05880a4320 0x7f05880a47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42116/0 (socket says 172.21.15.98:42116) 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 -- 172.21.15.98:0/1961604674 learned_addr learned my addr 172.21.15.98:0/1961604674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): get_auth_request con 0x7f058809df80 auth_method 0 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 10 monclient(hunting): get_auth_request con 0x7f05880a4320 auth_method 0 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058c834640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f058809bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058c834640 10 monclient(hunting): get_auth_request con 0x7f058809b8c0 auth_method 0 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058c834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058c834640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058c834640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient(hunting): handle_auth_done global_id 64631 payload 274 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 -- 172.21.15.98:0/1961604674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 msgr2=0x7f058809bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f058809bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 -- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05880a4320 msgr2=0x7f05880a47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05880a4320 0x7f05880a47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 monclient: found mon.noname-b 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f0587fff640 1 -- 172.21.15.98:0/1961604674 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05880a5d10 con 0x7f058809df80 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.215+0000 7f058d035640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05880a4320 0x7f05880a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f0587fff640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f058809df80 0x7f05880a3de0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f05700098a0 tx=0x7f057001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4c8906243df9f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 1 -- 172.21.15.98:0/1961604674 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05700b0070 con 0x7f058809df80 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: dump: 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.232 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.233 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 5 monclient: authenticate success, global_id 64631 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 1 -- 172.21.15.98:0/1961604674 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05700aa810 con 0x7f058809df80 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 1 -- 172.21.15.98:0/1961604674 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05700a9300 con 0x7f058809df80 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 4 set_mon_vals no callback set 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:11.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f05877fe640 10 monclient: dump: 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.235 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 10 monclient: shutdown 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 -- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f058809df80 msgr2=0x7f05880a3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f058809df80 0x7f05880a3de0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f05700098a0 tx=0x7f057001c440 comp rx=0 tx=0).stop 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 -- 172.21.15.98:0/1961604674 shutdown_connections 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05880a4320 0x7f05880a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f058809df80 0x7f05880a3de0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 --2- 172.21.15.98:0/1961604674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f058809bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 -- 172.21.15.98:0/1961604674 >> 172.21.15.98:0/1961604674 conn(0x7f05880a4ed0 msgr2=0x7f05880a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 -- 172.21.15.98:0/1961604674 shutdown_connections 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 -- 172.21.15.98:0/1961604674 wait complete. 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 10 monclient: build_initial_monmap 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 10 monclient: monmap: 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:11.237 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 librados: starting msgr at 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 librados: starting objecter 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 Processor -- start 2024-09-11T03:03:11.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 -- start start 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 librados: setting wanted keys 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 1 librados: calling monclient init 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.219+0000 7f058ea98640 10 monclient: init 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient: _add_conns ranks=[7,1,0,8,5,2,4,6,3] 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f05880738a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): picked mon.noname-i con 0x7f058809b8c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f058809df80 0x7f05881d1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): picked mon.noname-f con 0x7f058809df80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05880a4320 0x7f05881d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): picked mon.noname-a con 0x7f05880a4320 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0588142f30 con 0x7f05880a4320 2024-09-11T03:03:11.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0588142db0 con 0x7f058809df80 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f05881430b0 con 0x7f058809b8c0 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): authenticate will time out at 1745.104324s 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05880a4320 0x7f05881d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05880a4320 0x7f05881d1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32922/0 (socket says 172.21.15.98:32922) 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 -- 172.21.15.98:0/870578228 learned_addr learned my addr 172.21.15.98:0/870578228 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): get_auth_request con 0x7f05880a4320 auth_method 0 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f0587fff640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f058809df80 0x7f05881d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f05880738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient(hunting): handle_auth_done global_id 45045 payload 824 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 msgr2=0x7f05880738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f05880738a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f058809df80 msgr2=0x7f05881d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f058809df80 0x7f05881d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient: _finish_hunting 0 2024-09-11T03:03:11.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 monclient: found mon.noname-a 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 -- 172.21.15.98:0/870578228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0588094dc0 con 0x7f05880a4320 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient: get_auth_request con 0x7f058809b8c0 auth_method 0 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f0587fff640 10 monclient: get_auth_request con 0x7f058809df80 auth_method 0 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f058809b8c0 0x7f05880738a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f0587fff640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f058809df80 0x7f05881d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058d035640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05880a4320 0x7f05881d1b90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f058000ef90 tx=0x7f058000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 client.?.objecter ms_handle_connect 0x7f05880a4320 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0580018020 con 0x7f05880a4320 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:11.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: dump: 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.243 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: _finish_auth 0 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: _check_auth_tickets 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058000cdf0 con 0x7f05880a4320 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 5 monclient: authenticate success, global_id 45045 2024-09-11T03:03:11.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient: _renew_subs 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05880a47a0 con 0x7f05880a4320 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0580014d60 con 0x7f05880a4320 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f0586ffd640 4 set_mon_vals no callback set 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: dump: 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.245 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 client.45045.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:11.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient: _renew_subs 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0588094ae0 con 0x7f05880a4320 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 librados: init done 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05880923f0 0x7f05880927d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f058809bca0 con 0x7f05880923f0 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0580014480 con 0x7f05880a4320 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05880923f0 0x7f05880927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient: get_auth_request con 0x7f05880923f0 auth_method 0 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056803df40 0x7f0568040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(525..525 src has 1..525) ==== 6276+0+0 (secure 0 0 0) 0x7f0580050870 con 0x7f05880a4320 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 client.45045.objecter ms_dispatch 0x7f0588000ba0 osd_map(525..525 src has 1..525) 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 3 client.45045.objecter handle_osd_map got epochs [525,525] > 0 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 3 client.45045.objecter handle_osd_map decoding full epoch 525 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 20 client.45045.objecter dump_active .. 0 homeless 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f0587fff640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056803df40 0x7f0568040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f0587fff640 10 monclient: get_auth_request con 0x7f056803df40 auth_method 0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 10 monclient(hunting): handle_auth_done global_id 20086 payload 824 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 1 -- 172.21.15.98:0/870578228 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05880948e0 con 0x7f05880923f0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 client.45045.objecter ms_handle_connect 0x7f05880923f0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f058c834640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05880923f0 0x7f05880927d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f057800e740 tx=0x7f057800ec00 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 client.45045.objecter resend_mon_ops 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 client.45045.objecter ms_handle_connect 0x7f056803df40 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f0587fff640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056803df40 0x7f0568040400 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7f05700ad000 tx=0x7f05700a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0578017070 con 0x7f05880923f0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:03:11.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.223+0000 7f05857fa640 10 monclient: dump: 2024-09-11T03:03:11.250 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.250 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.250 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.250 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.251 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.227+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f057800ed30 con 0x7f05880923f0 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.227+0000 7f05857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.227+0000 7f05857fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.227+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05880923f0 msgr2=0x7f05880927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.227+0000 7f05857fa640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05880923f0 0x7f05880927d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f057800e740 tx=0x7f057800ec00 comp rx=0 tx=0).stop 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.227+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05880923f0 msgr2=0x7f05880927d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:11.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.227+0000 7f05857fa640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05880923f0 0x7f05880927d0 secure :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f057800e740 tx=0x7f057800ec00 comp rx=0 tx=0).stop 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058ea98640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058ea98640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0588092d10 0x7f05880707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058ea98640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f058806ca10 con 0x7f0588092d10 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0588092d10 0x7f05880707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient: get_auth_request con 0x7f0588092d10 auth_method 0 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 10 monclient(hunting): handle_auth_done global_id 20095 payload 824 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.231+0000 7f058c834640 1 -- 172.21.15.98:0/870578228 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05880a3d60 con 0x7f0588092d10 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 10 client.45045.objecter ms_handle_connect 0x7f0588092d10 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 10 client.45045.objecter resend_mon_ops 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058c834640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0588092d10 0x7f05880707b0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f057800e740 tx=0x7f05780046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f057800f040 con 0x7f0588092d10 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 10 monclient: dump: 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.256 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7f057800ce30 con 0x7f0588092d10 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0588092d10 msgr2=0x7f05880707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0588092d10 0x7f05880707b0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f057800e740 tx=0x7f05780046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0588092d10 msgr2=0x7f05880707b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:11.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f05857fa640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0588092d10 0x7f05880707b0 secure :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f057800e740 tx=0x7f05780046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 807005, 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:11.258 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:11.259 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:11.260 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:11.261 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:11.262 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.263 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:11.267 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.268 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:11.272 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:11.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:11.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:11.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:11.279 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.280 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:11.281 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:11.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:11.282 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:11.282 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:11.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 10 librados: watch_flush enter 2024-09-11T03:03:11.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 10 librados: watch_flush exit 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 20 client.45045.objecter shutdown clearing up homeless session... 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 10 client.45045.objecter successfully canceled tick 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056803df40 msgr2=0x7f0568040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f056803df40 0x7f0568040400 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7f05700ad000 tx=0x7f05700a6000 comp rx=0 tx=0).stop 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 10 monclient: shutdown 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05880a4320 msgr2=0x7f05881d1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 --2- 172.21.15.98:0/870578228 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05880a4320 0x7f05881d1b90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f058000ef90 tx=0x7f058000c490 comp rx=0 tx=0).stop 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058d035640 1 -- 172.21.15.98:0/870578228 reap_dead start 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058d035640 1 -- 172.21.15.98:0/870578228 reap_dead start 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 shutdown_connections 2024-09-11T03:03:11.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 >> 172.21.15.98:0/870578228 conn(0x7f05880a4ed0 msgr2=0x7f05880a0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:11.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 shutdown_connections 2024-09-11T03:03:11.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 -- 172.21.15.98:0/870578228 wait complete. 2024-09-11T03:03:11.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.235+0000 7f058ea98640 1 librados: shutdown 2024-09-11T03:03:11.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:11.285 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T03:03:11.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T03:03:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.399+0000 7fb54a237640 10 monclient: get_monmap_and_config 2024-09-11T03:03:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.399+0000 7fb54a237640 10 monclient: build_initial_monmap 2024-09-11T03:03:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.399+0000 7fb54a237640 10 monclient: monmap: 2024-09-11T03:03:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.402 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:11.404424+0000 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:11.404424+0000 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:11.403 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:11.404 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:11.404 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 Processor -- start 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 -- start start 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient: init 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient: _add_conns ranks=[5,6,2,10,3,7,0,1,9,8,4] 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5440a7aa0 0x7fb5440a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): picked mon.noname-h con 0x7fb5440a7aa0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:11.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a15b0 0x7fb5440a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): picked mon.noname-d con 0x7fb5440a15b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a83c0 0x7fb5441398c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): picked mon.noname-b con 0x7fb5440a83c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fb5440a3120 con 0x7fb5440a83c0 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb5440a1a60 con 0x7fb5440a7aa0 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb5440a2b00 con 0x7fb5440a15b0 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:11.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient(hunting): authenticate will time out at 1745.284861s 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a15b0 0x7fb5440a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a83c0 0x7fb5441398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a15b0 0x7fb5440a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34370/0 (socket says 172.21.15.98:34370) 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 -- 172.21.15.98:0/1512976358 learned_addr learned my addr 172.21.15.98:0/1512976358 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): get_auth_request con 0x7fb5440a15b0 auth_method 0 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 10 monclient(hunting): get_auth_request con 0x7fb5440a83c0 auth_method 0 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb5437fe640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5440a7aa0 0x7fb5440a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb5437fe640 10 monclient(hunting): get_auth_request con 0x7fb5440a7aa0 auth_method 0 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb5437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient(hunting): handle_auth_done global_id 84192 payload 274 2024-09-11T03:03:11.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 -- 172.21.15.98:0/1512976358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5440a7aa0 msgr2=0x7fb5440a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:11.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5440a7aa0 0x7fb5440a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 -- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a83c0 msgr2=0x7fb5441398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:11.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a83c0 0x7fb5441398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient: _finish_hunting 0 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 monclient: found mon.noname-d 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 -- 172.21.15.98:0/1512976358 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5440a96f0 con 0x7fb5440a15b0 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb543fff640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a83c0 0x7fb5441398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb53affd640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a15b0 0x7fb5440a7e80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0098a0 tx=0x7fb52c01c180 comp rx=0 tx=0).ready entity=mon.6 client_cookie=f9c9c010719dc8a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 1 -- 172.21.15.98:0/1512976358 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb52c0b0070 con 0x7fb5440a15b0 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: dump: 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 1 -- 172.21.15.98:0/1512976358 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52c0aa810 con 0x7fb5440a15b0 2024-09-11T03:03:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 5 monclient: authenticate success, global_id 84192 2024-09-11T03:03:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:11.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:11.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:11.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb54a237640 4 set_mon_vals no callback set 2024-09-11T03:03:11.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 1 -- 172.21.15.98:0/1512976358 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb52c0a9300 con 0x7fb5440a15b0 2024-09-11T03:03:11.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.403+0000 7fb542ffd640 10 monclient: dump: 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.418 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: shutdown 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a15b0 msgr2=0x7fb5440a7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a15b0 0x7fb5440a7e80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0098a0 tx=0x7fb52c01c180 comp rx=0 tx=0).stop 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- 172.21.15.98:0/1512976358 shutdown_connections 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fb5440a83c0 0x7fb5441398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb5440a15b0 0x7fb5440a7e80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 --2- 172.21.15.98:0/1512976358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb5440a7aa0 0x7fb5440a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- 172.21.15.98:0/1512976358 >> 172.21.15.98:0/1512976358 conn(0x7fb5440ac7a0 msgr2=0x7fb5440a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- 172.21.15.98:0/1512976358 shutdown_connections 2024-09-11T03:03:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- 172.21.15.98:0/1512976358 wait complete. 2024-09-11T03:03:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: build_initial_monmap 2024-09-11T03:03:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: monmap: 2024-09-11T03:03:11.420 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:11.421 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 librados: starting msgr at 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 librados: starting objecter 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 Processor -- start 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- start start 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 librados: setting wanted keys 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 librados: calling monclient init 2024-09-11T03:03:11.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: init 2024-09-11T03:03:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: _add_conns ranks=[1,3,7,8,0,4,5,2,6] 2024-09-11T03:03:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5440a7aa0 0x7fb5441cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): picked mon.noname-f con 0x7fb5440a7aa0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:11.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5440a83c0 0x7fb5441231c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): picked mon.noname-g con 0x7fb5440a83c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544139000 0x7fb544139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): picked mon.noname-i con 0x7fb544139000 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb544142f20 con 0x7fb5440a7aa0 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb544142da0 con 0x7fb5440a83c0 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb5441430a0 con 0x7fb544139000 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient(hunting): authenticate will time out at 1745.289908s 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5440a7aa0 0x7fb5441cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5440a83c0 0x7fb5441231c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544139000 0x7fb544139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5440a83c0 0x7fb5441231c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:39904/0 (socket says 172.21.15.98:39904) 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5440a7aa0 0x7fb5441cf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:54932/0 (socket says 172.21.15.98:54932) 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 -- 172.21.15.98:0/366834628 learned_addr learned my addr 172.21.15.98:0/366834628 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): get_auth_request con 0x7fb5440a83c0 auth_method 0 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 10 monclient(hunting): get_auth_request con 0x7fb5440a7aa0 auth_method 0 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 10 monclient(hunting): get_auth_request con 0x7fb544139000 auth_method 0 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb5437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient(hunting): handle_auth_done global_id 74325 payload 824 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 -- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544139000 msgr2=0x7fb544139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544139000 0x7fb544139460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 -- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5440a7aa0 msgr2=0x7fb5441cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5440a7aa0 0x7fb5441cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient: _finish_hunting 0 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 monclient: found mon.noname-g 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T03:03:11.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 -- 172.21.15.98:0/366834628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb544122d40 con 0x7fb5440a83c0 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb53affd640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5440a83c0 0x7fb5441231c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0ae040 tx=0x7fb52c018f20 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 client.?.objecter ms_handle_connect 0x7fb5440a83c0 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 1 -- 172.21.15.98:0/366834628 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb52c0b9070 con 0x7fb5440a83c0 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: dump: 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.432 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: _finish_auth 0 2024-09-11T03:03:11.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 5 monclient: authenticate success, global_id 74325 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 1 -- 172.21.15.98:0/366834628 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb52c0044b0 con 0x7fb5440a83c0 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: _renew_subs 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5440ab8b0 con 0x7fb5440a83c0 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 1 -- 172.21.15.98:0/366834628 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb52c0b0040 con 0x7fb5440a83c0 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:11.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb541ffb640 4 set_mon_vals no callback set 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 client.74325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb540ff9640 10 monclient: dump: 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:11.435 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: _renew_subs 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5440a5aa0 con 0x7fb5440a83c0 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.407+0000 7fb54a237640 1 librados: init done 2024-09-11T03:03:11.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb54a237640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb54a237640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb54a237640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5440a38e0 con 0x7fb5440a83c0 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 1 -- 172.21.15.98:0/366834628 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb52c0aacb0 con 0x7fb5440a83c0 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52403df30 0x7fb5240403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 1 -- 172.21.15.98:0/366834628 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(525..525 src has 1..525) ==== 6276+0+0 (secure 0 0 0) 0x7fb52c0e8b20 con 0x7fb5440a83c0 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 10 client.74325.objecter ms_dispatch 0x7fb544000ba0 osd_map(525..525 src has 1..525) 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 3 client.74325.objecter handle_osd_map got epochs [525,525] > 0 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 3 client.74325.objecter handle_osd_map decoding full epoch 525 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 20 client.74325.objecter dump_active .. 0 homeless 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb5437fe640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52403df30 0x7fb5240403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb5437fe640 10 monclient: get_auth_request con 0x7fb52403df30 auth_method 0 2024-09-11T03:03:11.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb540ff9640 10 client.74325.objecter ms_handle_connect 0x7fb52403df30 2024-09-11T03:03:11.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.411+0000 7fb5437fe640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52403df30 0x7fb5240403f0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7fb534006aa0 tx=0x7fb534008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:11.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.423+0000 7fb540ff9640 1 -- 172.21.15.98:0/366834628 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fb5440ab8b0 con 0x7fb5440a83c0 2024-09-11T03:03:11.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.423+0000 7fb540ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:11.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.423+0000 7fb540ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:11.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.667+0000 7fb54a237640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T03:03:11.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.667+0000 7fb54a237640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T03:03:11.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.667+0000 7fb54a237640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:03:11.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.667+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb544069090 con 0x7fb5440a83c0 2024-09-11T03:03:11.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb540ff9640 1 -- 172.21.15.98:0/366834628 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb52c016150 con 0x7fb5440a83c0 2024-09-11T03:03:11.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb540ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T03:03:11.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb540ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.678+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.678+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=7,logm=62} crc {auth=1655696098,config=692619927,health=3281969523,kv=2806819558,logm=845338102}) 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=9,logm=62} crc {auth=3842227035,config=692619927,health=3281969523,kv=3994774503,logm=426422278}) 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=11,kv=10,logm=60} crc {auth=1655696098,config=692619927,health=32989520,kv=3991532845,logm=1787610946}) 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=16,config=2,health=11,kv=9,logm=62} crc {auth=4047774273,config=692619927,health=130301661,kv=2811756501,logm=2205302986}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=2638402113,config=692619927,health=3281969523,kv=3991532845,logm=294363106}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=16,config=2,health=12,kv=10,logm=60} crc {auth=3317658193,config=692619927,health=3281969523,kv=3991532845,logm=3873420126}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=17,config=2,health=11,kv=10,logm=60} crc {auth=1655696098,config=692619927,health=32989520,kv=373919883,logm=1468210119}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=17,config=2,health=11,kv=8,logm=62} crc {auth=1655696098,config=692619927,health=493599810,kv=1009120483,logm=3272329202}) 2024-09-11T03:03:11.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.682+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=4255542836,config=692619927,health=3281969523,kv=3991532845,logm=400694446}) 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 10 librados: watch_flush enter 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 10 librados: watch_flush exit 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 20 client.74325.objecter shutdown clearing up homeless session... 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 10 client.74325.objecter successfully canceled tick 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52403df30 msgr2=0x7fb5240403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52403df30 0x7fb5240403f0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7fb534006aa0 tx=0x7fb534008040 comp rx=0 tx=0).stop 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 10 monclient: shutdown 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:11.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5440a83c0 msgr2=0x7fb5441231c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5440a83c0 0x7fb5441231c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0ae040 tx=0x7fb52c018f20 comp rx=0 tx=0).stop 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 shutdown_connections 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb52403df30 0x7fb5240403f0 unknown :-1 s=CLOSED pgs=3621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb544139000 0x7fb544139460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb5440a83c0 0x7fb5441231c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 --2- 172.21.15.98:0/366834628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5440a7aa0 0x7fb5441cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 >> 172.21.15.98:0/366834628 conn(0x7fb5440ac7a0 msgr2=0x7fb544140200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 shutdown_connections 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 -- 172.21.15.98:0/366834628 wait complete. 2024-09-11T03:03:11.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:11.675+0000 7fb54a237640 1 librados: shutdown 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32872975}) 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544856127}) 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3127146029}) 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2840796193}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2788486889}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=957162196}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=897344964}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=902957337}) 2024-09-11T03:03:11.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.686+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2910116079}) 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466041418}) 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=631006772}) 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040593106}) 2024-09-11T03:03:11.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3567459325}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3858576654}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1610701728}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=287553669}) 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4024958040}) 2024-09-11T03:03:11.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.690+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3154745270}) 2024-09-11T03:03:11.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2196206412}) 2024-09-11T03:03:11.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922859595}) 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518056448}) 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=828276570}) 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2454838724}) 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1454481162}) 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4133874006}) 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473162651}) 2024-09-11T03:03:11.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.694+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2130548187}) 2024-09-11T03:03:11.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757098531}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2719044448}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4232308176}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2730455418}) 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.698+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2546141723}) 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3727437623}) 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3343532037}) 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314514023}) 2024-09-11T03:03:11.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374781907}) 2024-09-11T03:03:11.700 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T03:03:11.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=284364390}) 2024-09-11T03:03:11.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1767552476}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3440767734}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3984511994}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3616460708}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.702+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.706+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=600067782}) 2024-09-11T03:03:11.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.706+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.706+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.706+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=539877870}) 2024-09-11T03:03:11.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.706+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.706+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=625141328}) 2024-09-11T03:03:11.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.706+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3938033446}) 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560799909}) 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2619557778}) 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1383335171}) 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3259923024}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1283863348}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3060663909}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=388169542}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1912925443}) 2024-09-11T03:03:11.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2059307694}) 2024-09-11T03:03:11.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761292520}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3082351174}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213734264}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.710+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063412497}) 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3206426295}) 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3930038889}) 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2310243750}) 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2381841376}) 2024-09-11T03:03:11.710 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=677138117}) 2024-09-11T03:03:11.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940103040}) 2024-09-11T03:03:11.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939745910}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920039235}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1011414796}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3149995333}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=328666716}) 2024-09-11T03:03:11.714 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.714 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.714 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3983532297}) 2024-09-11T03:03:11.714 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.714 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=777410178}) 2024-09-11T03:03:11.714 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.714+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2324120691}) 2024-09-11T03:03:11.715 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407492976}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3658632927}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282815714}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=420507144}) 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.716 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1438388567}) 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1925090187}) 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4022209467}) 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4025121738}) 2024-09-11T03:03:11.717 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.718+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2957654434}) 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178943279}) 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945195121}) 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.720 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.721 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815719696}) 2024-09-11T03:03:11.721 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.721 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.721 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3169846405}) 2024-09-11T03:03:11.721 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.721 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2119361423}) 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2869017630}) 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4286944191}) 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2711831283}) 2024-09-11T03:03:11.722 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.722+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=355217422}) 2024-09-11T03:03:11.724 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.724 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.724 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573653280}) 2024-09-11T03:03:11.724 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.724 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.724 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1774158447}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2794172419}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=474086948}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3314373230}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2849825237}) 2024-09-11T03:03:11.725 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.726 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.726 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2441141298}) 2024-09-11T03:03:11.726 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.726 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2728561962}) 2024-09-11T03:03:11.726 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.726+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4181286304}) 2024-09-11T03:03:11.727 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.727 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2769755428}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1702586589}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3138628433}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=180340266}) 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.728 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1202424767}) 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2971513154}) 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=453145338}) 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604058948}) 2024-09-11T03:03:11.729 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.730+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2173438825}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=318070702}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4035427146}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3829119357}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.732 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3860615449}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1116595696}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3933745618}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1160238431}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147470107}) 2024-09-11T03:03:11.733 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.734+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=931435629}) 2024-09-11T03:03:11.735 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.735 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.735 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2800109497}) 2024-09-11T03:03:11.735 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.735 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.735 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082339019}) 2024-09-11T03:03:11.735 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3704066850}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1214013600}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3369951348}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=745586031}) 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.736 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.737 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1921321791}) 2024-09-11T03:03:11.737 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.737 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236094931}) 2024-09-11T03:03:11.737 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.738+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2949202809}) 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205137457}) 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4229872407}) 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.738 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3379375494}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1984960052}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2393614340}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4256225653}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.739 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3313512302}) 2024-09-11T03:03:11.740 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.740 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3888531900}) 2024-09-11T03:03:11.740 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1937830416}) 2024-09-11T03:03:11.742 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.742 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.742 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3044982892}) 2024-09-11T03:03:11.742 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.742+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.742 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.742 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=69264030}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699983147}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2117815213}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3131720217}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2985460181}) 2024-09-11T03:03:11.743 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.744 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.744 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=208500659}) 2024-09-11T03:03:11.744 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.744 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433556463}) 2024-09-11T03:03:11.744 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=613207960}) 2024-09-11T03:03:11.746 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.746 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.746 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2776122283}) 2024-09-11T03:03:11.746 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.746 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=153749046}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4161036234}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.746+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2843517071}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2694652372}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=942211773}) 2024-09-11T03:03:11.747 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.748 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.748 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3633062679}) 2024-09-11T03:03:11.748 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.748 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1948775228}) 2024-09-11T03:03:11.748 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1989390084}) 2024-09-11T03:03:11.749 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.749 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.749 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444445874}) 2024-09-11T03:03:11.749 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.749 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.749 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952519968}) 2024-09-11T03:03:11.749 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3141616233}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620939554}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3312472903}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=24751947}) 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.750 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.751 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2097786799}) 2024-09-11T03:03:11.751 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.751 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2148685071}) 2024-09-11T03:03:11.751 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.750+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1235268745}) 2024-09-11T03:03:11.754 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.754 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.754 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833538610}) 2024-09-11T03:03:11.754 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.754 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.754 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755150683}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15082867}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3046258435}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=861999637}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3395690895}) 2024-09-11T03:03:11.755 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1808093355}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=517877054}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.754+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1326139863}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101276380}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1707135669}) 2024-09-11T03:03:11.756 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1668879496}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3922519760}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1343730189}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2546811333}) 2024-09-11T03:03:11.757 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.758 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.758 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4076520458}) 2024-09-11T03:03:11.758 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.758 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3859347013}) 2024-09-11T03:03:11.758 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=876508047}) 2024-09-11T03:03:11.764 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.764 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209932625}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.758+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1540923929}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407989400}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1358342648}) 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.765 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2861898835}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3188591047}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=31289689}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1927004450}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2502198001}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.766 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299804488}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691209105}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542917292}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1652947236}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=976382862}) 2024-09-11T03:03:11.767 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=851696385}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=127632372}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1395603594}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.762+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3536371104}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869262821}) 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.768 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=110042181}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3054912108}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2428618622}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2424288384}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.769 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2742036978}) 2024-09-11T03:03:11.770 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.770 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.770 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3863516116}) 2024-09-11T03:03:11.770 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.770 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694317750}) 2024-09-11T03:03:11.770 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.766+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1818959772}) 2024-09-11T03:03:11.776 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.776 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.776 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782417431}) 2024-09-11T03:03:11.776 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.776 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2323137029}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2775734302}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=817000019}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2086081473}) 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.777 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=691762056}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2613215079}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2168426894}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.770+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=751789302}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405223816}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2053540622}) 2024-09-11T03:03:11.778 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954032526}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4183074552}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=657724244}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1208719102}) 2024-09-11T03:03:11.779 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2350350325}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1803719684}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1446846081}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847630698}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868033255}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.780 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1905480993}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042627173}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3324277237}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2428758225}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.781 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1719403857}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1871120976}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.774+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2690557965}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996157557}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2246114614}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.782 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1044000915}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=972897364}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632286420}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1032600827}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1583115785}) 2024-09-11T03:03:11.783 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.784 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=618516175}) 2024-09-11T03:03:11.784 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.778+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1241599779}) 2024-09-11T03:03:11.792 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.792 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.792 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161246898}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3139109712}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931892977}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3486807705}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.793 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1182365054}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2646020748}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=344651718}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780652216}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.782+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=29659747}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251681940}) 2024-09-11T03:03:11.794 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849756478}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855166163}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=867753525}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2028753803}) 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.795 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=414985332}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4208189918}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3768262292}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.786+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4084220498}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=407111104}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.796 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=991843091}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398219002}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4205703829}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2819008355}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3256593246}) 2024-09-11T03:03:11.797 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1532964168}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=931130326}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3290738280}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2097546065}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320364099}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.798 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329074918}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3470900484}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1486569485}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4083833063}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3114528189}) 2024-09-11T03:03:11.799 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=445360465}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.790+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1184849621}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2544807075}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841510513}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1531641382}) 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.800 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600401079}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3744409610}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4193758533}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3490670718}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639711705}) 2024-09-11T03:03:11.801 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=244708729}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394247963}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991588282}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2176659509}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3068210183}) 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.802 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1893301546}) 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3878176078}) 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=133755160}) 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.794+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4221817431}) 2024-09-11T03:03:11.803 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=717624458}) 2024-09-11T03:03:11.818 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.818 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624430883}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553340353}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915540120}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2960971013}) 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.819 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3623937859}) 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2277004488}) 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1699537744}) 2024-09-11T03:03:11.820 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999805054}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.798+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2166136435}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875997541}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3058705817}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896595009}) 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.821 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914591705}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2626547487}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4063317080}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3287946543}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813024450}) 2024-09-11T03:03:11.822 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1549745366}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211897736}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987220334}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=620488048}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1588400310}) 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.823 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=56452166}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1379137214}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1296850049}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1386551063}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.802+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=824530266}) 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.824 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095527517}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462064877}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970789164}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2034175722}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.825 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3352628204}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1443691110}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1442921778}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3846522115}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2393465587}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988784663}) 2024-09-11T03:03:11.826 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489457842}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984206131}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=273306603}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=7133785}) 2024-09-11T03:03:11.827 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4261820678}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1303699403}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56323815}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.806+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=51815717}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079387012}) 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.828 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1674239821}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532271396}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1195797603}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2330894204}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.829 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3628916123}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1748192266}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1454164796}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.810+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3495040659}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97236340}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1925617602}) 2024-09-11T03:03:11.830 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2501034173}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3499542711}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3299629996}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=937797089}) 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.831 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2991882053}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=679970500}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2563859054}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1111172387}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=291843851}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.832 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3027331380}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1956783686}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=637601872}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2998605867}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1549282307}) 2024-09-11T03:03:11.833 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2159844414}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.814+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=944713965}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120402678}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2228454740}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436891100}) 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.834 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2253123030}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2842644585}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2687955551}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=911665748}) 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.835 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2952435776}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3582042188}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810905280}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3790712770}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683342404}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.836 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3128857788}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=386146759}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2358517644}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2485060863}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2105333048}) 2024-09-11T03:03:11.837 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.818+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2336292445}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094887793}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929366407}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2107786756}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2220734534}) 2024-09-11T03:03:11.838 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1177843257}) 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3292569311}) 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=776518581}) 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3057012969}) 2024-09-11T03:03:11.839 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1827825380}) 2024-09-11T03:03:11.978 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.978 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.978 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405664579}) 2024-09-11T03:03:11.978 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011890460}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.822+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824611487}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2383447155}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3866910290}) 2024-09-11T03:03:11.979 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2839028141}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=182945609}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=737056628}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1412295349}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=18466624}) 2024-09-11T03:03:11.980 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683165262}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2176821009}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2281615430}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1654444483}) 2024-09-11T03:03:11.981 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2486926970}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=397678922}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231662505}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.826+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2992013941}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3584546416}) 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.982 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155361970}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651372681}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1016331345}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4038066780}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.983 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3332572427}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2581368874}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=193752936}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3315959858}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=399870699}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.984 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=392706605}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1203628421}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4177312217}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1367183651}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2964312479}) 2024-09-11T03:03:11.985 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3593928638}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1980217292}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.830+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=562631720}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541280308}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872912450}) 2024-09-11T03:03:11.986 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239701919}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3190394793}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=273917433}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3847360975}) 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.987 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=973655433}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1519422992}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2786259275}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3911876107}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089670224}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.988 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2002969607}) 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2281398415}) 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1276611493}) 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3515251839}) 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.989 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=89220402}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=604656011}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.834+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2561015933}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1740582521}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3379531110}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.990 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=915658782}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3328691829}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2144879207}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2137340008}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3228198499}) 2024-09-11T03:03:11.991 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3004860135}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2102855111}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592250044}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.838+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3356747731}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299018859}) 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.992 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1598229695}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2933079600}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2002181382}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2188729941}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3728317878}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=264948117}) 2024-09-11T03:03:11.993 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=689290766}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281582142}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595208750}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4174656906}) 2024-09-11T03:03:11.994 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=223031250}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2682422386}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1858222917}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2004711316}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.842+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3995974481}) 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.995 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332811162}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966771556}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792382701}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1932068275}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.996 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1881456965}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=324994128}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=93333129}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1629852912}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3422198196}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349819243}) 2024-09-11T03:03:11.997 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3659329670}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915841278}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=501840528}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3443216648}) 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.998 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3053977155}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1739155149}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817946181}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.846+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4107684264}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100704178}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:11.999 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703547093}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243208227}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=990874174}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1639124780}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3621108684}) 2024-09-11T03:03:12.000 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2196733068}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2165178790}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3157977808}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870490319}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2330595355}) 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.001 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1685234930}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3360884583}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3161402031}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=796806369}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.002 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3396847688}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=451632763}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.850+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4029093954}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2201076676}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984889952}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2095596320}) 2024-09-11T03:03:12.003 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4262050301}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2975024013}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2611266908}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=393345828}) 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.004 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1362537106}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2731770379}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349102555}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403856128}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007553455}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.005 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2621608639}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=405055978}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1651686778}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2855182196}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641069081}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=631892887}) 2024-09-11T03:03:12.006 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.854+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579040967}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3681809791}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523552007}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2354733343}) 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.007 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2810029362}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1129673691}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4130873961}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2596266780}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=513828072}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.008 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724704960}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557715442}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=350825241}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3865281018}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1130593287}) 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.009 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2846410721}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1041762037}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026296499}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4214758746}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.858+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279662788}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.010 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2893618380}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2941574186}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1218670002}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=967853990}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2709062625}) 2024-09-11T03:03:12.011 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1113708912}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1600973845}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3257850668}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281923374}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385360585}) 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.012 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971365872}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2532633251}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3631589297}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3728927524}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.013 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3808607518}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=73666843}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=815679632}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914479107}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1553625934}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3049142599}) 2024-09-11T03:03:12.014 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3029370968}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3073707274}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1163919409}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1805379072}) 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.015 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=551132574}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.862+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4104880672}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3818430361}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2238087141}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839018482}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.016 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1771374438}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1556514646}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=438803923}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=185267972}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2914766585}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1376640624}) 2024-09-11T03:03:12.017 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240800951}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=201931427}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406594567}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=19246780}) 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.018 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=641216893}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=593885509}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=60113034}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=624822126}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.866+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3574482721}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730901539}) 2024-09-11T03:03:12.019 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099607464}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=219346262}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=454506598}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1840842727}) 2024-09-11T03:03:12.020 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1838526056}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=741705646}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=164623343}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3846932719}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260796739}) 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.021 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988381929}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522851809}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2263036314}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=91217337}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1772316835}) 2024-09-11T03:03:12.022 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=704277048}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=956595206}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.870+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4269790221}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858297834}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=124360422}) 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.023 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111072292}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3011906373}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3855440210}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4040757432}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.024 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2854208562}) 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49578460}) 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1788530453}) 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259368149}) 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522282311}) 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.025 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=527274614}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1290352044}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4214400297}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=309323021}) 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.026 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=324883423}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.874+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3218381297}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2992412700}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188156900}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1887106738}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.027 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662883509}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3087940764}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=334213103}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=945269447}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.028 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=186292125}) 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137264255}) 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2098511502}) 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235407309}) 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.029 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1776728054}) 2024-09-11T03:03:12.031 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128953770}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4012589785}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3172072819}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1917929109}) 2024-09-11T03:03:12.032 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3681263138}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3066988105}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.878+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3186616993}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791195605}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=686714607}) 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.033 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015322595}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562930822}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2979679282}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=96317477}) 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.034 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=760024773}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1538668402}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=245570457}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946574063}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714420813}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3643445850}) 2024-09-11T03:03:12.035 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1708480002}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2703748098}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1696058858}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=401083225}) 2024-09-11T03:03:12.036 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=735918305}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.882+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=883556077}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352806755}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1820364300}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462884450}) 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.037 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2626038957}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2594445912}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3736609495}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2952160835}) 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.038 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2564314605}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=766066704}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613548162}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154629707}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=471665108}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.039 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3736491170}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3672365508}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3178833190}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3354333853}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4183138160}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.886+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1859175698}) 2024-09-11T03:03:12.040 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460951203}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157456981}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2155988188}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1809881835}) 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.041 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=205933440}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=770095346}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1203462229}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3062549939}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2337870829}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4139809795}) 2024-09-11T03:03:12.042 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4264413853}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955095962}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4122463865}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=780144360}) 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.043 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4090639423}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1666390447}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525905039}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4138251590}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041402536}) 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.044 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3998513833}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=551655320}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3484367705}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=149661855}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.045 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4271187645}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2417540195}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3197917088}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.890+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3721687784}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518805686}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3107909863}) 2024-09-11T03:03:12.046 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=541341956}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3045932230}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=453889729}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=364259366}) 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.047 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2738072656}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=768065904}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2607907652}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540095877}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879968694}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.048 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111491164}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3533200972}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2296519980}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3440969577}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.049 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2387869430}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=162138065}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.894+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1113389702}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764375624}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1396994658}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843605161}) 2024-09-11T03:03:12.050 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1748302689}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=57744797}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2182400591}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3739940756}) 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.051 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418227063}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1347604940}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057793259}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036092492}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691897550}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.052 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386322737}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=95752379}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=102246611}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1710637191}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2672728450}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.898+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=575229905}) 2024-09-11T03:03:12.053 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3864547769}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296537553}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470142149}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4008315703}) 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.054 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2337805484}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2332434270}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3765789637}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=981591794}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1149500231}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.055 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764477509}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1362383287}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2669314733}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=156208025}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1254236245}) 2024-09-11T03:03:12.056 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4279742373}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1051372717}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2248045369}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.902+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3743115529}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160005202}) 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.057 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3468197215}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619001628}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1368170800}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=229081371}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.058 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3473545224}) 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=647252795}) 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1707030550}) 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=780493172}) 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.059 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.060 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191466585}) 2024-09-11T03:03:12.060 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.060 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.060 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2642776919}) 2024-09-11T03:03:12.060 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.060 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.060 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=230532643}) 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=308576952}) 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4172827669}) 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3770584549}) 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.061 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2514636142}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=370883401}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.906+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2401785385}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281227526}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1036239681}) 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.062 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605389828}) 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210005910}) 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1165567160}) 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.063 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1113857698}) 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=228791016}) 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2539328780}) 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1170183180}) 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855644744}) 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.064 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812344248}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3841276147}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4167838763}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=336853712}) 2024-09-11T03:03:12.065 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=546014538}) 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3549828071}) 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197311899}) 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.910+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3552797938}) 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.066 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203844317}) 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1180327703}) 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158671258}) 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2437878978}) 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.067 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1934117257}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3994983719}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1801918881}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=471676400}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1687788770}) 2024-09-11T03:03:12.068 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237042791}) 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=92270146}) 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990014714}) 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.069 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1482721386}) 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=942235541}) 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2382999404}) 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=810258079}) 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.070 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781965640}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.914+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=737033089}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1144288174}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=417536719}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588240900}) 2024-09-11T03:03:12.071 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3329825755}) 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1119245434}) 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3518899723}) 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.072 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2958017926}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1704425680}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2958559384}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479689109}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=207828720}) 2024-09-11T03:03:12.073 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.918+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017152182}) 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2894643311}) 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3544939682}) 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.074 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=240297041}) 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=595356009}) 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=958909460}) 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4086833025}) 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398253151}) 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.075 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462760523}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212219156}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3197593312}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=965705554}) 2024-09-11T03:03:12.076 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=964996863}) 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=525923161}) 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1028357846}) 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.922+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1614418356}) 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.077 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937834624}) 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4278148436}) 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745481556}) 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1670401542}) 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.078 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=123597780}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=748649241}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3632832414}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3951784887}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2244314954}) 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.079 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534999578}) 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1763763639}) 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094268776}) 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.080 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=529412530}) 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2250975052}) 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=217071372}) 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.926+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3242470284}) 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.081 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506694544}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1533468052}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826101812}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399164527}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078823659}) 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.082 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763118591}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1815854300}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3195287412}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=111879000}) 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.083 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2976919249}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.930+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4017394326}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949063447}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285085697}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2223041662}) 2024-09-11T03:03:12.084 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3814665549}) 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2268359615}) 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2917777109}) 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.085 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2450888808}) 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1348435437}) 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3773829258}) 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386988062}) 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3828832442}) 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.086 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2865235511}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1790228354}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1287050530}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4042845723}) 2024-09-11T03:03:12.087 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=136146410}) 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3744496886}) 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.934+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1196114955}) 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695535081}) 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.088 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3407000466}) 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2111512304}) 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3649613734}) 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3365375993}) 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.089 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1602030873}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1718999604}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2210851010}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2419742181}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3854305867}) 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.090 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837655890}) 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630045258}) 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=943084287}) 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.091 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=57571318}) 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1195486660}) 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=47256422}) 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986888801}) 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1898463772}) 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.092 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2082288364}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3571353958}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=387683803}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562484785}) 2024-09-11T03:03:12.093 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.938+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1088623768}) 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=694504026}) 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3707540623}) 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1324007383}) 2024-09-11T03:03:12.094 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3930838162}) 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298891825}) 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=859215320}) 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074128074}) 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.095 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2886834646}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3762046282}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1770143158}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3457093992}) 2024-09-11T03:03:12.096 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4230098964}) 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1576365932}) 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1342719157}) 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3953091170}) 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.097 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=197486528}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3111988482}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4153296749}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2399569145}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.098 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3533701476}) 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1114407492}) 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.942+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3642296761}) 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1331662298}) 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2444571860}) 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.099 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596660050}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1470125891}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2404685818}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2142430901}) 2024-09-11T03:03:12.100 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1674443070}) 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=98218418}) 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1280568307}) 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429250453}) 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.101 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462121610}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163531539}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4204796728}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2510184870}) 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.102 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1210304789}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3423468300}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.946+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3703762903}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=334136535}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610299856}) 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.103 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3259585032}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2962069369}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3402031585}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3042337964}) 2024-09-11T03:03:12.104 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3680418718}) 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3544703959}) 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1827761984}) 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1812431184}) 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.105 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=2,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=1} crc {logm=3545272853,mdsmap=92030302,mgr=1656677489,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2882400814,mon_config_key=3200485260,monmap=2495409540,nvmeofgw=598337686,osd_metadata=2544375700}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=1} crc {logm=2751837241,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3696078984,mon_config_key=2106052021,monmap=1217224211}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=1} crc {logm=624115619,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1141212378,mon_config_key=2106052021,monmap=1217224211}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3} crc {logm=277054861,mdsmap=3491474967,mgr=492734240,mgr_metadata=3513241537,mgrstat=1141212378,mon_config_key=1130541923,monmap=2495409540}) 2024-09-11T03:03:12.106 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=1,nvmeofgw=2} crc {logm=2543249737,mdsmap=99049999,mgr=4089010074,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1141212378,mon_config_key=1738238056,monmap=1228700967,nvmeofgw=1596038039}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=59,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,nvmeofgw=2} crc {logm=3283055819,mdsmap=3491474967,mgr=1056758727,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=463870673,mon_config_key=2106052021,monmap=2495409540,nvmeofgw=1596038039}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,nvmeofgw=2,osd_metadata=4} crc {logm=2891136668,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=291794698,mon_config_key=2269478041,monmap=2495409540,nvmeofgw=1596038039,osd_metadata=1041320735}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=63,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=10,monmap=3} crc {logm=1761144217,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=3741007869,mon_config_key=4199536219,monmap=2495409540}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.950+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=2,nvmeofgw=2} crc {logm=2488927642,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=1141212378,mon_config_key=2106052021,monmap=416611756,nvmeofgw=1596038039}) 2024-09-11T03:03:12.107 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=55} crc {nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2845464069,osd_snap=1888845239,osdmap=4171824601}) 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=58} crc {nvmeofgw=1596038039,osd_metadata=1477139452,osd_pg_creating=2845464069,osd_snap=2150453966,osdmap=1881561196}) 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=56} crc {nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2845464069,osd_snap=195882041,osdmap=1710610982}) 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.108 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=37,osdmap=54} crc {nvmeofgw=598337686,osd_metadata=1863826368,osd_pg_creating=2845464069,osd_snap=227602720,osdmap=3522122716}) 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=53} crc {nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2845464069,osd_snap=2695005506,osdmap=4276687489}) 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=55} crc {nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2845464069,osd_snap=34632800,osdmap=3907926617}) 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=55} crc {nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2845464069,osd_snap=2695851324,osdmap=1565748178}) 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.109 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=57} crc {nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=2845464069,osd_snap=1720082332,osdmap=2822222557}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=54} crc {nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=2845464069,osd_snap=3025944384,osdmap=427396574}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4079963405}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.954+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2450288279}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2333722263}) 2024-09-11T03:03:12.110 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=381829230}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1167709617}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=566746020}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1559562698}) 2024-09-11T03:03:12.111 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2862213414}) 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3959891611}) 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1043222470}) 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2076416413}) 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.112 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=887390016}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1822598460}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3383574967}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=701480729}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.113 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3701375948}) 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=143171465}) 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.958+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3209597645}) 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3909889064}) 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3670361041}) 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.114 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=540006171}) 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2637589251}) 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1449009731}) 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.115 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3672485396}) 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2424239519}) 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1792033027}) 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2477192253}) 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4134182092}) 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.116 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1956835667}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2648155965}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3252571219}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1231692462}) 2024-09-11T03:03:12.117 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3150036476}) 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.962+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2089944182}) 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1580862517}) 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2898510500}) 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.118 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2954789226}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2120155601}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2215576147}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3957013147}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.119 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1562568458}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=22638091}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=95271776}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2949731949}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.966+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3447441921}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.120 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=306052738}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3421697396}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1095250311}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=689267597}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.121 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1194747688}) 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=610696482}) 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1715413058}) 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3432227103}) 2024-09-11T03:03:12.122 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.970+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=735820386}) 2024-09-11T03:03:12.125 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.125 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.125 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=92} crc {osdmap=1956316715}) 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=97} crc {osdmap=3827978347}) 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=91} crc {osdmap=292701126}) 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=97} crc {osdmap=1234287797}) 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.126 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=94} crc {osdmap=390456467}) 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=94} crc {osdmap=1632168319}) 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=94} crc {osdmap=3041968850}) 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=96} crc {osdmap=1687263934}) 2024-09-11T03:03:12.127 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:03:11.986+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=98} crc {osdmap=2309024125}) 2024-09-11T03:03:12.700 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:03:12.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:03:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient: get_monmap_and_config 2024-09-11T03:03:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient: build_initial_monmap 2024-09-11T03:03:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient: monmap: 2024-09-11T03:03:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:12.853 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:12.855353+0000 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:12.855353+0000 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:12.854 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:12.855 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:12.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 1 Processor -- start 2024-09-11T03:03:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 1 -- start start 2024-09-11T03:03:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient: init 2024-09-11T03:03:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient: _add_conns ranks=[10,1,8,3,9,5,4,0,2,7,6] 2024-09-11T03:03:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fda940a4820 0x7fda940a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:12.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient(hunting): picked mon.noname-f con 0x7fda940a4820 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fda9409fb10 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda940770c0 0x7fda940a5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient(hunting): picked mon.noname-e con 0x7fda940770c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda940a15e0 con 0x7fda9409fb10 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fda940a3380 con 0x7fda940a4820 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fda940a5680 con 0x7fda940770c0 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda98a91640 10 monclient(hunting): authenticate will time out at 1746.735297s 2024-09-11T03:03:12.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda940770c0 0x7fda940a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:12.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda940770c0 0x7fda940a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52408/0 (socket says 172.21.15.98:52408) 2024-09-11T03:03:12.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 -- 172.21.15.98:0/1438359947 learned_addr learned my addr 172.21.15.98:0/1438359947 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:12.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): get_auth_request con 0x7fda940770c0 auth_method 0 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 1 --2- 172.21.15.98:0/1438359947 >> v2:172.21.15.190:3300/0 conn(0x7fda940a4820 0x7fda940a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 10 monclient(hunting): get_auth_request con 0x7fda940a4820 auth_method 0 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:12.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient(hunting): handle_auth_done global_id 25010 payload 274 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 -- 172.21.15.98:0/1438359947 >> v2:172.21.15.190:3300/0 conn(0x7fda940a4820 msgr2=0x7fda940a4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 --2- 172.21.15.98:0/1438359947 >> v2:172.21.15.190:3300/0 conn(0x7fda940a4820 0x7fda940a4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 -- 172.21.15.98:0/1438359947 >> v1:172.21.15.190:6789/0 conn(0x7fda9409fb10 legacy=0x7fda941399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient: _finish_hunting 0 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 monclient: found mon.noname-e 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:12.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.851+0000 7fda92d9d640 1 -- 172.21.15.98:0/1438359947 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda94076e40 con 0x7fda940770c0 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda92d9d640 1 --2- 172.21.15.98:0/1438359947 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda940770c0 0x7fda940a5140 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009e50 tx=0x7fda8c01c890 comp rx=0 tx=0).ready entity=mon.8 client_cookie=19982714c1fb66fa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 1 -- 172.21.15.98:0/1438359947 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda8c0090d0 con 0x7fda940770c0 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: dump: 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:12.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: _finish_auth 0 2024-09-11T03:03:12.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: _check_auth_tickets 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 1 -- 172.21.15.98:0/1438359947 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8c0a59b0 con 0x7fda940770c0 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 1 -- 172.21.15.98:0/1438359947 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda8c0a5ba0 con 0x7fda940770c0 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 5 monclient: authenticate success, global_id 25010 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 4 set_mon_vals no callback set 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda9159a640 10 monclient: dump: 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:12.872 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:12.873 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 10 monclient: shutdown 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 -- 172.21.15.98:0/1438359947 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda940770c0 msgr2=0x7fda940a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 --2- 172.21.15.98:0/1438359947 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda940770c0 0x7fda940a5140 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009e50 tx=0x7fda8c01c890 comp rx=0 tx=0).stop 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 -- 172.21.15.98:0/1438359947 shutdown_connections 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 --2- 172.21.15.98:0/1438359947 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fda940770c0 0x7fda940a5140 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 --2- 172.21.15.98:0/1438359947 >> v2:172.21.15.190:3300/0 conn(0x7fda940a4820 0x7fda940a4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 -- 172.21.15.98:0/1438359947 >> 172.21.15.98:0/1438359947 conn(0x7fda9406ed00 msgr2=0x7fda9406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:12.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 -- 172.21.15.98:0/1438359947 shutdown_connections 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 -- 172.21.15.98:0/1438359947 wait complete. 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 10 monclient: build_initial_monmap 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 10 monclient: monmap: 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:12.875 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 librados: starting msgr at 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 librados: starting objecter 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 Processor -- start 2024-09-11T03:03:12.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 -- start start 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 librados: setting wanted keys 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 1 librados: calling monclient init 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.855+0000 7fda98a91640 10 monclient: init 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _add_conns ranks=[4,1,3,2,8,7,6,5,0] 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda940770c0 0x7fda9413f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): picked mon.noname-c con 0x7fda940770c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9409fb10 0x7fda941389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): picked mon.noname-f con 0x7fda9409fb10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda940a4820 0x7fda9413cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:12.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): picked mon.noname-g con 0x7fda940a4820 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fda94143fd0 con 0x7fda9409fb10 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fda94143e50 con 0x7fda940a4820 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fda94144150 con 0x7fda940770c0 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient(hunting): authenticate will time out at 1746.740207s 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda940770c0 0x7fda9413f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda940770c0 0x7fda9413f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42132/0 (socket says 172.21.15.98:42132) 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 -- 172.21.15.98:0/3547071730 learned_addr learned my addr 172.21.15.98:0/3547071730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:12.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): get_auth_request con 0x7fda940770c0 auth_method 0 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9409fb10 0x7fda941389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda940a4820 0x7fda9413cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient(hunting): get_auth_request con 0x7fda9409fb10 auth_method 0 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:12.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 10 monclient(hunting): get_auth_request con 0x7fda940a4820 auth_method 0 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:12.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient(hunting): handle_auth_done global_id 24502 payload 824 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 -- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda940a4820 msgr2=0x7fda9413cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda940a4820 0x7fda9413cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 -- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9409fb10 msgr2=0x7fda941389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9409fb10 0x7fda941389c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 monclient: found mon.noname-c 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 -- 172.21.15.98:0/3547071730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda94140e90 con 0x7fda940770c0 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda9259c640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda940770c0 0x7fda9413f190 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00e870 tx=0x7fda7c00ec70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:12.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 client.?.objecter ms_handle_connect 0x7fda940770c0 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 1 -- 172.21.15.98:0/3547071730 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda7c017070 con 0x7fda940770c0 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: dump: 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:12.882 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: _finish_auth 0 2024-09-11T03:03:12.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 1 -- 172.21.15.98:0/3547071730 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda7c00cdb0 con 0x7fda940770c0 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 1 -- 172.21.15.98:0/3547071730 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda7c0138e0 con 0x7fda940770c0 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 5 monclient: authenticate success, global_id 24502 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda90d99640 4 set_mon_vals no callback set 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 monclient: dump: 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:12.884 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:12.885 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _renew_subs 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda941406b0 con 0x7fda940770c0 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 client.24502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _renew_subs 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fda94139640 con 0x7fda940770c0 2024-09-11T03:03:12.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 librados: init done 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda941399d0 con 0x7fda940770c0 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 1 -- 172.21.15.98:0/3547071730 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fda7c013470 con 0x7fda940770c0 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fda7403df40 0x7fda74040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 1 -- 172.21.15.98:0/3547071730 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(527..527 src has 1..527) ==== 6662+0+0 (secure 0 0 0) 0x7fda7c00f040 con 0x7fda940770c0 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 client.24502.objecter ms_dispatch 0x7fda94000ba0 osd_map(527..527 src has 1..527) 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 3 client.24502.objecter handle_osd_map got epochs [527,527] > 0 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 3 client.24502.objecter handle_osd_map decoding full epoch 527 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 20 client.24502.objecter dump_active .. 0 homeless 2024-09-11T03:03:12.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fda7403df40 0x7fda74040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 10 monclient: get_auth_request con 0x7fda7403df40 auth_method 0 2024-09-11T03:03:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda8affd640 10 client.24502.objecter ms_handle_connect 0x7fda7403df40 2024-09-11T03:03:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.859+0000 7fda91d9b640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fda7403df40 0x7fda74040400 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7fda9413eca0 tx=0x7fda84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.863+0000 7fda8affd640 1 -- 172.21.15.98:0/3547071730 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fda7c051e20 con 0x7fda940770c0 2024-09-11T03:03:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.863+0000 7fda8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:12.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:12.863+0000 7fda8affd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:13.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.099+0000 7fda98a91640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:03:13.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.099+0000 7fda98a91640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.099+0000 7fda98a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.099+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda9413e630 con 0x7fda940770c0 2024-09-11T03:03:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.099+0000 7fda8affd640 1 -- 172.21.15.98:0/3547071730 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fda7c021070 con 0x7fda940770c0 2024-09-11T03:03:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.099+0000 7fda8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.099+0000 7fda8affd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:13.102 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":96,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:03:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 10 librados: watch_flush enter 2024-09-11T03:03:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 10 librados: watch_flush exit 2024-09-11T03:03:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 20 client.24502.objecter shutdown clearing up homeless session... 2024-09-11T03:03:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 10 client.24502.objecter successfully canceled tick 2024-09-11T03:03:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fda7403df40 msgr2=0x7fda74040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fda7403df40 0x7fda74040400 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7fda9413eca0 tx=0x7fda84009040 comp rx=0 tx=0).stop 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 10 monclient: shutdown 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda940770c0 msgr2=0x7fda9413f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda940770c0 0x7fda9413f190 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00e870 tx=0x7fda7c00ec70 comp rx=0 tx=0).stop 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 shutdown_connections 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fda7403df40 0x7fda74040400 unknown :-1 s=CLOSED pgs=3622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fda940a4820 0x7fda9413cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fda9409fb10 0x7fda941389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 --2- 172.21.15.98:0/3547071730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fda940770c0 0x7fda9413f190 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 >> 172.21.15.98:0/3547071730 conn(0x7fda9406ed00 msgr2=0x7fda940a1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 shutdown_connections 2024-09-11T03:03:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 -- 172.21.15.98:0/3547071730 wait complete. 2024-09-11T03:03:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.103+0000 7fda98a91640 1 librados: shutdown 2024-09-11T03:03:13.121 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:03:13.121 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T03:03:13.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T03:03:13.122 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:03:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient: get_monmap_and_config 2024-09-11T03:03:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient: monmap: 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:13.241619+0000 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:13.241619+0000 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:13.241 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:13.241 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 1 Processor -- start 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 1 -- start start 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient: init 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient: _add_conns ranks=[5,7,1,8,2,6,3,0,4,10,9] 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3dc4139b10 0x7f3dc4072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient(hunting): picked mon.noname-h con 0x7f3dc4139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139ef0 0x7f3dc40a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient(hunting): picked mon.noname-i con 0x7f3dc4139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3dc413a2d0 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc40702b0 con 0x7f3dc413a2d0 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f3dc4071bf0 con 0x7f3dc4139b10 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3dc4071350 con 0x7f3dc4139ef0 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dcc1cc640 10 monclient(hunting): authenticate will time out at 1747.121862s 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3dc4139b10 0x7f3dc4072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dca769640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7f3dc413a2d0 0x7f3dc40a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:33462/0 (socket says 172.21.15.98:33462) 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dca769640 1 -- 172.21.15.98:0/963232944 learned_addr learned my addr 172.21.15.98:0/963232944 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 1 --2- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139ef0 0x7f3dc40a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): get_auth_request con 0x7f3dc4139b10 auth_method 0 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 10 monclient(hunting): get_auth_request con 0x7f3dc4139ef0 auth_method 0 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc8f66640 1 -- 172.21.15.98:0/963232944 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076750194 0 0) 0x7f3dbc0014a0 con 0x7f3dc413a2d0 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc8f66640 10 monclient(hunting): _negotiate 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc8f66640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc8f66640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc8f66640 10 monclient(hunting): my global_id is 25840 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc8f66640 1 -- 172.21.15.98:0/963232944 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3db0003530 con 0x7f3dc413a2d0 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_done global_id 74345 payload 274 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 1 -- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139ef0 msgr2=0x7f3dc40a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 1 --2- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139ef0 0x7f3dc40a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 1 -- 172.21.15.98:0/963232944 >> v1:172.21.15.190:6789/0 conn(0x7f3dc413a2d0 legacy=0x7f3dc40a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 1 monclient: found mon.noname-h 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9f68640 1 -- 172.21.15.98:0/963232944 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc40a9620 con 0x7f3dc4139b10 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.239+0000 7f3dc9767640 1 --2- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139ef0 0x7f3dc40a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc9f68640 1 --2- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3dc4139b10 0x7f3dc4072600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00097f0 tx=0x7f3dc001c3a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=976afde684dcbefd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 1 -- 172.21.15.98:0/963232944 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dc00b0070 con 0x7f3dc4139b10 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: dump: 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 1 -- 172.21.15.98:0/963232944 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc00aa810 con 0x7f3dc4139b10 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 5 monclient: authenticate success, global_id 74345 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 1 -- 172.21.15.98:0/963232944 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dc00aae30 con 0x7f3dc4139b10 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 4 set_mon_vals no callback set 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc8f66640 10 monclient: dump: 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient: shutdown 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3dc4139b10 msgr2=0x7f3dc4072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3dc4139b10 0x7f3dc4072600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00097f0 tx=0x7f3dc001c3a0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/963232944 shutdown_connections 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139ef0 0x7f3dc40a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/963232944 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f3dc4139b10 0x7f3dc4072600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/963232944 >> 172.21.15.98:0/963232944 conn(0x7f3dc40ac790 msgr2=0x7f3dc40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/963232944 shutdown_connections 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/963232944 wait complete. 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient: monmap: 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 librados: starting msgr at 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 librados: starting objecter 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 Processor -- start 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- start start 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 librados: setting wanted keys 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 librados: calling monclient init 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient: init 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient: _add_conns ranks=[7,3,0,1,8,6,5,4,2] 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139b10 0x7f3dc413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): picked mon.noname-i con 0x7f3dc4139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dc4139ef0 0x7f3dc413ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): picked mon.noname-g con 0x7f3dc4139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc413a2d0 0x7f3dc413b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): picked mon.noname-a con 0x7f3dc413a2d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc4144470 con 0x7f3dc413a2d0 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc40a8870 con 0x7f3dc4139ef0 2024-09-11T03:03:13.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f3dc41445f0 con 0x7f3dc4139b10 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dcc1cc640 10 monclient(hunting): authenticate will time out at 1747.127625s 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc413a2d0 0x7f3dc413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc413a2d0 0x7f3dc413b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32926/0 (socket says 172.21.15.98:32926) 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 1 -- 172.21.15.98:0/3100831478 learned_addr learned my addr 172.21.15.98:0/3100831478 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 10 monclient(hunting): get_auth_request con 0x7f3dc413a2d0 auth_method 0 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dca769640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc9767640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dc4139ef0 0x7f3dc413ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc9f68640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139b10 0x7f3dc413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.243+0000 7f3dc9767640 10 monclient(hunting): get_auth_request con 0x7f3dc4139ef0 auth_method 0 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9767640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9767640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): get_auth_request con 0x7f3dc4139b10 auth_method 0 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 10 monclient(hunting): handle_auth_done global_id 45054 payload 824 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139b10 msgr2=0x7f3dc413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3dc4139b10 0x7f3dc413a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dc4139ef0 msgr2=0x7f3dc413ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f3dc4139ef0 0x7f3dc413ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 1 monclient: found mon.noname-a 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 1 -- 172.21.15.98:0/3100831478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc413cd80 con 0x7f3dc413a2d0 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dca769640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc413a2d0 0x7f3dc413b250 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00cb10 tx=0x7f3dbc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 client.?.objecter ms_handle_connect 0x7f3dc413a2d0 2024-09-11T03:03:13.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dbc012400 con 0x7f3dc413a2d0 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: dump: 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.269 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0125e0 con 0x7f3dc413a2d0 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 5 monclient: authenticate success, global_id 45054 2024-09-11T03:03:13.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dbc012c00 con 0x7f3dc413a2d0 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dbbfff640 4 set_mon_vals no callback set 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: dump: 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.271 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 10 monclient: _renew_subs 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc413d060 con 0x7f3dc413a2d0 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 10 client.45054.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 10 monclient: _renew_subs 2024-09-11T03:03:13.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc40aa1a0 con 0x7f3dc413a2d0 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 1 librados: init done 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4133760 0x7f3dc4133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc406ca10 con 0x7f3dc4133760 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4133760 0x7f3dc4133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient: get_auth_request con 0x7f3dc4133760 auth_method 0 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3dbc012790 con 0x7f3dc413a2d0 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_done global_id 45063 payload 824 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3da803df40 0x7f3da8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 1 -- 172.21.15.98:0/3100831478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dc413c800 con 0x7f3dc4133760 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(527..527 src has 1..527) ==== 6662+0+0 (secure 0 0 0) 0x7f3dbc04fe70 con 0x7f3dc413a2d0 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 client.45054.objecter ms_dispatch 0x7f3dc4000ba0 osd_map(527..527 src has 1..527) 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 3 client.45054.objecter handle_osd_map got epochs [527,527] > 0 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 3 client.45054.objecter handle_osd_map decoding full epoch 527 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 20 client.45054.objecter dump_active .. 0 homeless 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 client.45054.objecter ms_handle_connect 0x7f3dc4133760 2024-09-11T03:03:13.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9f68640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4133760 0x7f3dc4133b40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00ad000 tx=0x7f3dc0019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 client.45054.objecter resend_mon_ops 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dc00b9070 con 0x7f3dc4133760 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9767640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3da803df40 0x7f3da8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: dump: 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9767640 10 monclient: get_auth_request con 0x7f3da803df40 auth_method 0 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3dc0003620 con 0x7f3dc4133760 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:13.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4133760 msgr2=0x7f3dc4133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4133760 0x7f3dc4133b40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3dc00ad000 tx=0x7f3dc0019140 comp rx=0 tx=0).stop 2024-09-11T03:03:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4133760 msgr2=0x7f3dc4133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4133760 0x7f3dc4133b40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dba7fc640 10 client.45054.objecter ms_handle_connect 0x7f3da803df40 2024-09-11T03:03:13.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.247+0000 7f3dc9767640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3da803df40 0x7f3da8040400 secure :-1 s=READY pgs=3623 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009580 tx=0x7f3db4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:13.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4134080 0x7f3dc40a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc406e870 con 0x7f3dc4134080 2024-09-11T03:03:13.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4134080 0x7f3dc40a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient: get_auth_request con 0x7f3dc4134080 auth_method 0 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 10 monclient(hunting): handle_auth_done global_id 45072 payload 824 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 1 -- 172.21.15.98:0/3100831478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3dc406e680 con 0x7f3dc4134080 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 10 client.45054.objecter ms_handle_connect 0x7f3dc4134080 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 10 client.45054.objecter resend_mon_ops 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dc9f68640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4134080 0x7f3dc40a0e60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3dc001c3e0 tx=0x7f3dc0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dc00b9040 con 0x7f3dc4134080 2024-09-11T03:03:13.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 10 monclient: dump: 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f3dc00b5ca0 con 0x7f3dc4134080 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4134080 msgr2=0x7f3dc40a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4134080 0x7f3dc40a0e60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3dc001c3e0 tx=0x7f3dc0009190 comp rx=0 tx=0).stop 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4134080 msgr2=0x7f3dc40a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dba7fc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc4134080 0x7f3dc40a0e60 secure :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3dc001c3e0 tx=0x7f3dc0009190 comp rx=0 tx=0).stop 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 419694, 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:13.285 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.286 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.287 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:13.288 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.289 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:13.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:13.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.302 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.305 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.306 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.308 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:13.308 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 10 librados: watch_flush enter 2024-09-11T03:03:13.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 10 librados: watch_flush exit 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 20 client.45054.objecter shutdown clearing up homeless session... 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 10 client.45054.objecter successfully canceled tick 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3da803df40 msgr2=0x7f3da8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f3da803df40 0x7f3da8040400 secure :-1 s=READY pgs=3623 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009580 tx=0x7f3db4008040 comp rx=0 tx=0).stop 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 10 monclient: shutdown 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc413a2d0 msgr2=0x7f3dc413b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dca769640 1 -- 172.21.15.98:0/3100831478 reap_dead start 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc413a2d0 0x7f3dc413b250 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00cb10 tx=0x7f3dbc00a060 comp rx=0 tx=0).stop 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 shutdown_connections 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 --2- 172.21.15.98:0/3100831478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc413a2d0 0x7f3dc413b250 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 >> 172.21.15.98:0/3100831478 conn(0x7f3dc40ac790 msgr2=0x7f3dc40a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 shutdown_connections 2024-09-11T03:03:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 -- 172.21.15.98:0/3100831478 wait complete. 2024-09-11T03:03:13.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.255+0000 7f3dcc1cc640 1 librados: shutdown 2024-09-11T03:03:13.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:13.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T03:03:13.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:03:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient: get_monmap_and_config 2024-09-11T03:03:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient: monmap: 2024-09-11T03:03:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:13.426253+0000 2024-09-11T03:03:13.424 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:13.426253+0000 2024-09-11T03:03:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.425 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:13.426 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:13.426 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 1 Processor -- start 2024-09-11T03:03:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 1 -- start start 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient: init 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient: _add_conns ranks=[1,8,5,4,7,3,9,2,6,10,0] 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f481009e9b0 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4810068e30 0x7f481009bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient(hunting): picked mon.noname-e con 0x7f4810068e30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f481009ed90 0x7f48100a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient(hunting): picked mon.noname-h con 0x7f481009ed90 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f481009d450 con 0x7f481009e9b0 2024-09-11T03:03:13.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f481009d730 con 0x7f481009ed90 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f481009e1f0 con 0x7f4810068e30 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f4814f77640 10 monclient(hunting): authenticate will time out at 1747.306460s 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4810068e30 0x7f481009bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4810068e30 0x7f481009bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52416/0 (socket says 172.21.15.98:52416) 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 -- 172.21.15.98:0/434101406 learned_addr learned my addr 172.21.15.98:0/434101406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): get_auth_request con 0x7f4810068e30 auth_method 0 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480ed9d640 1 --2- 172.21.15.98:0/434101406 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f481009ed90 0x7f48100a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480ed9d640 10 monclient(hunting): get_auth_request con 0x7f481009ed90 auth_method 0 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient(hunting): handle_auth_done global_id 25019 payload 274 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 -- 172.21.15.98:0/434101406 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f481009ed90 msgr2=0x7f48100a44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 --2- 172.21.15.98:0/434101406 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f481009ed90 0x7f48100a44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 -- 172.21.15.98:0/434101406 >> v1:172.21.15.190:6789/0 conn(0x7f481009e9b0 legacy=0x7f48100a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 monclient: found mon.noname-e 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.423+0000 7f480dd9b640 1 -- 172.21.15.98:0/434101406 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48100a4a30 con 0x7f4810068e30 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480dd9b640 1 --2- 172.21.15.98:0/434101406 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4810068e30 0x7f481009bc80 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f47f80098a0 tx=0x7f47f801c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3f430d0d19107298 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 1 -- 172.21.15.98:0/434101406 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47f80b0070 con 0x7f4810068e30 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: dump: 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.439 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 1 -- 172.21.15.98:0/434101406 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47f80aa810 con 0x7f4810068e30 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 5 monclient: authenticate success, global_id 25019 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 4 set_mon_vals no callback set 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 1 -- 172.21.15.98:0/434101406 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47f80a9300 con 0x7f4810068e30 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480d59a640 10 monclient: dump: 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.442 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient: shutdown 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- 172.21.15.98:0/434101406 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4810068e30 msgr2=0x7f481009bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 --2- 172.21.15.98:0/434101406 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4810068e30 0x7f481009bc80 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f47f80098a0 tx=0x7f47f801c440 comp rx=0 tx=0).stop 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- 172.21.15.98:0/434101406 shutdown_connections 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 --2- 172.21.15.98:0/434101406 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4810068e30 0x7f481009bc80 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 --2- 172.21.15.98:0/434101406 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f481009ed90 0x7f48100a44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- 172.21.15.98:0/434101406 >> 172.21.15.98:0/434101406 conn(0x7f48100a4ed0 msgr2=0x7f48100a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- 172.21.15.98:0/434101406 shutdown_connections 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- 172.21.15.98:0/434101406 wait complete. 2024-09-11T03:03:13.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient: monmap: 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.444 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 librados: starting msgr at 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 librados: starting objecter 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 Processor -- start 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- start start 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 librados: setting wanted keys 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 librados: calling monclient init 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient: init 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient: _add_conns ranks=[7,1,0,4,6,2,5,8,3] 2024-09-11T03:03:13.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4810068e30 0x7f48101407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): picked mon.noname-i con 0x7f4810068e30 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f481009e9b0 0x7f4810140d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): picked mon.noname-f con 0x7f481009e9b0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481009ed90 0x7f4810094d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): picked mon.noname-a con 0x7f481009ed90 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4810144150 con 0x7f481009ed90 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4810143fd0 con 0x7f481009e9b0 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f48101442d0 con 0x7f4810068e30 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f4814f77640 10 monclient(hunting): authenticate will time out at 1747.311552s 2024-09-11T03:03:13.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481009ed90 0x7f4810094d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481009ed90 0x7f4810094d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32956/0 (socket says 172.21.15.98:32956) 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 1 -- 172.21.15.98:0/2947672861 learned_addr learned my addr 172.21.15.98:0/2947672861 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 10 monclient(hunting): get_auth_request con 0x7f481009ed90 auth_method 0 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480e59c640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4810068e30 0x7f48101407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480dd9b640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f481009e9b0 0x7f4810140d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480e59c640 10 monclient(hunting): get_auth_request con 0x7f4810068e30 auth_method 0 2024-09-11T03:03:13.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480dd9b640 10 monclient(hunting): get_auth_request con 0x7f481009e9b0 auth_method 0 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.427+0000 7f480dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 10 monclient(hunting): handle_auth_done global_id 45081 payload 824 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4810068e30 msgr2=0x7f48101407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4810068e30 0x7f48101407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f481009e9b0 msgr2=0x7f4810140d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f481009e9b0 0x7f4810140d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 1 monclient: found mon.noname-a 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 1 -- 172.21.15.98:0/2947672861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48100a2520 con 0x7f481009ed90 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480ed9d640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481009ed90 0x7f4810094d00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4808009fd0 tx=0x7f480800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 client.?.objecter ms_handle_connect 0x7f481009ed90 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f480800edd0 con 0x7f481009ed90 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: dump: 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.449 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4808019070 con 0x7f481009ed90 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 5 monclient: authenticate success, global_id 45081 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 10 monclient: _renew_subs 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4810098560 con 0x7f481009ed90 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4808014d00 con 0x7f481009ed90 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: dump: 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.451 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.452 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480cd99640 4 set_mon_vals no callback set 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 10 client.45081.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 10 monclient: _renew_subs 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48100a72e0 con 0x7f481009ed90 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 1 librados: init done 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810095240 0x7f4810095620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f48100967c0 con 0x7f4810095240 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810095240 0x7f4810095620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4808014420 con 0x7f481009ed90 2024-09-11T03:03:13.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient: get_auth_request con 0x7f4810095240 auth_method 0 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f47f003df70 0x7f47f0040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(527..527 src has 1..527) ==== 6662+0+0 (secure 0 0 0) 0x7f4808050120 con 0x7f481009ed90 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 client.45081.objecter ms_dispatch 0x7f4810000ba0 osd_map(527..527 src has 1..527) 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 3 client.45081.objecter handle_osd_map got epochs [527,527] > 0 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 3 client.45081.objecter handle_osd_map decoding full epoch 527 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 10 monclient(hunting): handle_auth_done global_id 64649 payload 824 2024-09-11T03:03:13.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 1 -- 172.21.15.98:0/2947672861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4810095b60 con 0x7f4810095240 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 20 client.45081.objecter dump_active .. 0 homeless 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480e59c640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810095240 0x7f4810095620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f480000e450 tx=0x7f480000e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 client.45081.objecter ms_handle_connect 0x7f4810095240 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 client.45081.objecter resend_mon_ops 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480dd9b640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f47f003df70 0x7f47f0040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480dd9b640 10 monclient: get_auth_request con 0x7f47f003df70 auth_method 0 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48000076b0 con 0x7f4810095240 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: dump: 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.455 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4800007240 con 0x7f4810095240 2024-09-11T03:03:13.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810095240 msgr2=0x7f4810095620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810095240 0x7f4810095620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f480000e450 tx=0x7f480000e910 comp rx=0 tx=0).stop 2024-09-11T03:03:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810095240 msgr2=0x7f4810095620 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810095240 0x7f4810095620 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f4806ffd640 10 client.45081.objecter ms_handle_connect 0x7f47f003df70 2024-09-11T03:03:13.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.431+0000 7f480dd9b640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f47f003df70 0x7f47f0040430 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7f47f80ad000 tx=0x7f47f80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810072780 0x7f4810072b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f481006ca10 con 0x7f4810072780 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810072780 0x7f4810072b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient: get_auth_request con 0x7f4810072780 auth_method 0 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 10 monclient(hunting): handle_auth_done global_id 64658 payload 824 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 1 -- 172.21.15.98:0/2947672861 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4810097cb0 con 0x7f4810072780 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 10 client.45081.objecter ms_handle_connect 0x7f4810072780 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 10 client.45081.objecter resend_mon_ops 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480e59c640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810072780 0x7f4810072b90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f480000efd0 tx=0x7f4800003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f480001ebd0 con 0x7f4810072780 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 10 monclient: dump: 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.460 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f480001e760 con 0x7f4810072780 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810072780 msgr2=0x7f4810072b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810072780 0x7f4810072b90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f480000efd0 tx=0x7f4800003770 comp rx=0 tx=0).stop 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810072780 msgr2=0x7f4810072b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4806ffd640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4810072780 0x7f4810072b90 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f480000efd0 tx=0x7f4800003770 comp rx=0 tx=0).stop 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:13.462 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 285632, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.463 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.464 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.465 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:13.466 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.467 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.468 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.469 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.471 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.472 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:13.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.475 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.476 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:13.477 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:13.478 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.479 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.480 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.481 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.482 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.483 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.484 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.485 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:13.486 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 10 librados: watch_flush enter 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 10 librados: watch_flush exit 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 20 client.45081.objecter shutdown clearing up homeless session... 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 10 client.45081.objecter successfully canceled tick 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f47f003df70 msgr2=0x7f47f0040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f47f003df70 0x7f47f0040430 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7f47f80ad000 tx=0x7f47f80a6000 comp rx=0 tx=0).stop 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 10 monclient: shutdown 2024-09-11T03:03:13.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481009ed90 msgr2=0x7f4810094d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 --2- 172.21.15.98:0/2947672861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f481009ed90 0x7f4810094d00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4808009fd0 tx=0x7f480800e4b0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480ed9d640 1 -- 172.21.15.98:0/2947672861 reap_dead start 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f480ed9d640 1 -- 172.21.15.98:0/2947672861 reap_dead start 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 shutdown_connections 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 >> 172.21.15.98:0/2947672861 conn(0x7f48100a4ed0 msgr2=0x7f48100a0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 shutdown_connections 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 -- 172.21.15.98:0/2947672861 wait complete. 2024-09-11T03:03:13.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.439+0000 7f4814f77640 1 librados: shutdown 2024-09-11T03:03:13.489 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:13.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T03:03:13.490 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:03:13.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient: get_monmap_and_config 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient: monmap: 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:13.605941+0000 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:13.605941+0000 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:13.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:13.605 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 1 Processor -- start 2024-09-11T03:03:13.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 1 -- start start 2024-09-11T03:03:13.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient: init 2024-09-11T03:03:13.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient: _add_conns ranks=[7,0,9,10,8,6,2,3,4,1,5] 2024-09-11T03:03:13.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff09c139b10 0x7ff09c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient(hunting): picked mon.noname-i con 0x7ff09c139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff09c139ef0 addr v1:172.21.15.98:6789/0 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7ff09c13a2d0 0x7ff09c0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff09c13a2d0 addr v2:172.21.15.98:3300/0 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff09c0702b0 con 0x7ff09c139ef0 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7ff09c071bf0 con 0x7ff09c13a2d0 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7ff09c071350 con 0x7ff09c139b10 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a45e0640 10 monclient(hunting): authenticate will time out at 1747.486209s 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a1b7b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff09c139ef0 0x7ff09c0a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46602/0 (socket says 172.21.15.98:46602) 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a1b7b640 1 -- 172.21.15.98:0/3991242622 learned_addr learned my addr 172.21.15.98:0/3991242622 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 1 --2- 172.21.15.98:0/3991242622 >> v2:172.21.15.98:3300/0 conn(0x7ff09c13a2d0 0x7ff09c0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): get_auth_request con 0x7ff09c13a2d0 auth_method 0 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a237c640 1 --2- 172.21.15.98:0/3991242622 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff09c139b10 0x7ff09c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_done global_id 45090 payload 274 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/3991242622 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff09c139b10 msgr2=0x7ff09c072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 1 --2- 172.21.15.98:0/3991242622 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff09c139b10 0x7ff09c072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/3991242622 >> v1:172.21.15.98:6789/0 conn(0x7ff09c139ef0 legacy=0x7ff09c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 1 monclient: found mon.noname-a 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/3991242622 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09c0a9620 con 0x7ff09c13a2d0 2024-09-11T03:03:13.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a237c640 10 monclient: get_auth_request con 0x7ff09c139b10 auth_method 0 2024-09-11T03:03:13.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.603+0000 7ff0a237c640 1 --2- 172.21.15.98:0/3991242622 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff09c139b10 0x7ff09c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 --2- 172.21.15.98:0/3991242622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c0a8120 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff094009d90 tx=0x7ff09401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545ee41ff22bae64 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 1 -- 172.21.15.98:0/3991242622 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0940b2070 con 0x7ff09c13a2d0 2024-09-11T03:03:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:13.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: dump: 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.618 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 1 -- 172.21.15.98:0/3991242622 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0940a59f0 con 0x7ff09c13a2d0 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 5 monclient: authenticate success, global_id 45090 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 4 set_mon_vals no callback set 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 1 -- 172.21.15.98:0/3991242622 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0940a5c20 con 0x7ff09c13a2d0 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a137a640 10 monclient: dump: 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.620 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient: shutdown 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- 172.21.15.98:0/3991242622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 msgr2=0x7ff09c0a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 --2- 172.21.15.98:0/3991242622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c0a8120 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff094009d90 tx=0x7ff09401c8c0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- 172.21.15.98:0/3991242622 shutdown_connections 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 --2- 172.21.15.98:0/3991242622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c0a8120 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 --2- 172.21.15.98:0/3991242622 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff09c139b10 0x7ff09c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- 172.21.15.98:0/3991242622 >> 172.21.15.98:0/3991242622 conn(0x7ff09c0ac790 msgr2=0x7ff09c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- 172.21.15.98:0/3991242622 shutdown_connections 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- 172.21.15.98:0/3991242622 wait complete. 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient: monmap: 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 librados: starting msgr at 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 librados: starting objecter 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 Processor -- start 2024-09-11T03:03:13.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- start start 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 librados: setting wanted keys 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 librados: calling monclient init 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient: init 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient: _add_conns ranks=[8,2,0,5,4,1,6,3,7] 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff09c139b10 0x7ff09c13a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): picked mon.noname-e con 0x7ff09c139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff09c139ef0 0x7ff09c13ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): picked mon.noname-b con 0x7ff09c139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): picked mon.noname-a con 0x7ff09c13a2d0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:13.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c144470 con 0x7ff09c13a2d0 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ff09c0a8870 con 0x7ff09c139ef0 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7ff09c1445f0 con 0x7ff09c139b10 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a45e0640 10 monclient(hunting): authenticate will time out at 1747.491367s 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c13b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff09c139ef0 0x7ff09c13ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff09c139b10 0x7ff09c13a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c13b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32976/0 (socket says 172.21.15.98:32976) 2024-09-11T03:03:13.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff09c139ef0 0x7ff09c13ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40318/0 (socket says 172.21.15.98:40318) 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/1512264783 learned_addr learned my addr 172.21.15.98:0/1512264783 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): get_auth_request con 0x7ff09c13a2d0 auth_method 0 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 10 monclient(hunting): get_auth_request con 0x7ff09c139ef0 auth_method 0 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 10 monclient(hunting): get_auth_request con 0x7ff09c139b10 auth_method 0 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a1b7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient(hunting): handle_auth_done global_id 45108 payload 824 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff09c139b10 msgr2=0x7ff09c13a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7ff09c139b10 0x7ff09c13a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff09c139ef0 msgr2=0x7ff09c13ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ff09c139ef0 0x7ff09c13ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 monclient: found mon.noname-a 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.607+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/1512264783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09c0a58c0 con 0x7ff09c13a2d0 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a2b7d640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c13b220 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff0940b0040 tx=0x7ff094019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 client.?.objecter ms_handle_connect 0x7ff09c13a2d0 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff0940b2050 con 0x7ff09c13a2d0 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:13.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: dump: 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0940be070 con 0x7ff09c13a2d0 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 5 monclient: authenticate success, global_id 45108 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 10 monclient: _renew_subs 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09c13cd50 con 0x7ff09c13a2d0 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff094003bb0 con 0x7ff09c13a2d0 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: dump: 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 10 client.45108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 10 monclient: _renew_subs 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a0b79640 4 set_mon_vals no callback set 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09c0a50a0 con 0x7ff09c13a2d0 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 1 librados: init done 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13bb10 0x7ff09c1340b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff09c06ca10 con 0x7ff09c13bb10 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ff094004040 con 0x7ff09c13a2d0 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13bb10 0x7ff09c1340b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient: get_auth_request con 0x7ff09c13bb10 auth_method 0 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff08003df40 0x7ff080040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 10 monclient(hunting): handle_auth_done global_id 24511 payload 824 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 1 -- 172.21.15.98:0/1512264783 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff09c06e680 con 0x7ff09c13bb10 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(527..527 src has 1..527) ==== 6662+0+0 (secure 0 0 0) 0x7ff0940eac20 con 0x7ff09c13a2d0 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 client.45108.objecter ms_dispatch 0x7ff09c000ba0 osd_map(527..527 src has 1..527) 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 3 client.45108.objecter handle_osd_map got epochs [527,527] > 0 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 3 client.45108.objecter handle_osd_map decoding full epoch 527 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 20 client.45108.objecter dump_active .. 0 homeless 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 client.45108.objecter ms_handle_connect 0x7ff09c13bb10 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 client.45108.objecter resend_mon_ops 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a237c640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13bb10 0x7ff09c1340b0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff09800f880 tx=0x7ff09800fd40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff09800de90 con 0x7ff09c13bb10 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:13.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: dump: 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.635 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a1b7b640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff08003df40 0x7ff080040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a1b7b640 10 monclient: get_auth_request con 0x7ff08003df40 auth_method 0 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff09800da20 con 0x7ff09c13bb10 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13bb10 msgr2=0x7ff09c1340b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13bb10 0x7ff09c1340b0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff09800f880 tx=0x7ff09800fd40 comp rx=0 tx=0).stop 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13bb10 msgr2=0x7ff09c1340b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13bb10 0x7ff09c1340b0 secure :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff09800f880 tx=0x7ff09800fd40 comp rx=0 tx=0).stop 2024-09-11T03:03:13.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff092ffd640 10 client.45108.objecter ms_handle_connect 0x7ff08003df40 2024-09-11T03:03:13.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.611+0000 7ff0a1b7b640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff08003df40 0x7ff080040400 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0a5f30 tx=0x7ff08c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13dcf0 0x7ff09c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff09c06e870 con 0x7ff09c13dcf0 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13dcf0 0x7ff09c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient: get_auth_request con 0x7ff09c13dcf0 auth_method 0 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 10 monclient(hunting): handle_auth_done global_id 24520 payload 824 2024-09-11T03:03:13.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 1 -- 172.21.15.98:0/1512264783 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff09c072600 con 0x7ff09c13dcf0 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 10 client.45108.objecter ms_handle_connect 0x7ff09c13dcf0 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a237c640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13dcf0 0x7ff09c02d450 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff09800f880 tx=0x7ff0980046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 10 client.45108.objecter resend_mon_ops 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff098004490 con 0x7ff09c13dcf0 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 10 monclient: dump: 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7ff098006710 con 0x7ff09c13dcf0 2024-09-11T03:03:13.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13dcf0 msgr2=0x7ff09c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13dcf0 0x7ff09c02d450 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff09800f880 tx=0x7ff0980046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13dcf0 msgr2=0x7ff09c02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff092ffd640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff09c13dcf0 0x7ff09c02d450 secure :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff09800f880 tx=0x7ff0980046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:13.642 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 809403, 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:13.643 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.645 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.646 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:13.647 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.648 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.649 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.650 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:13.653 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.655 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:13.656 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.657 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.659 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.660 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.662 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:13.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.665 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.666 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:13.667 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 10 librados: watch_flush enter 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 10 librados: watch_flush exit 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 20 client.45108.objecter shutdown clearing up homeless session... 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 10 client.45108.objecter successfully canceled tick 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff08003df40 msgr2=0x7ff080040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff08003df40 0x7ff080040400 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0a5f30 tx=0x7ff08c009040 comp rx=0 tx=0).stop 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 10 monclient: shutdown 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 msgr2=0x7ff09c13b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 --2- 172.21.15.98:0/1512264783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c13a2d0 0x7ff09c13b220 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff0940b0040 tx=0x7ff094019660 comp rx=0 tx=0).stop 2024-09-11T03:03:13.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/1512264783 reap_dead start 2024-09-11T03:03:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a2b7d640 1 -- 172.21.15.98:0/1512264783 reap_dead start 2024-09-11T03:03:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 shutdown_connections 2024-09-11T03:03:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 >> 172.21.15.98:0/1512264783 conn(0x7ff09c0ac790 msgr2=0x7ff09c0aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 shutdown_connections 2024-09-11T03:03:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 -- 172.21.15.98:0/1512264783 wait complete. 2024-09-11T03:03:13.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.619+0000 7ff0a45e0640 1 librados: shutdown 2024-09-11T03:03:13.670 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:13.670 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T03:03:13.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.791+0000 7f5a63b93640 10 monclient: get_monmap_and_config 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.791+0000 7f5a63b93640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.791+0000 7f5a63b93640 10 monclient: monmap: 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:13.797049+0000 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:13.797049+0000 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:13.795 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:13.796 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 Processor -- start 2024-09-11T03:03:13.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 -- start start 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient: init 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient: _add_conns ranks=[6,4,3,9,0,5,2,7,10,1,8] 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c139b80 0x7f5a5c139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): picked mon.noname-d con 0x7f5a5c139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5a5c06fe50 0x7f5a5c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): picked mon.noname-c con 0x7f5a5c06fe50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c072640 0x7f5a5c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): picked mon.noname-g con 0x7f5a5c072640 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5c0719c0 con 0x7f5a5c072640 2024-09-11T03:03:13.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5a5c071c10 con 0x7f5a5c06fe50 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5a5c0702d0 con 0x7f5a5c139b80 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a63b93640 10 monclient(hunting): authenticate will time out at 1747.677206s 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c139b80 0x7f5a5c139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5a5c06fe50 0x7f5a5c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c139b80 0x7f5a5c139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34384/0 (socket says 172.21.15.98:34384) 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5a5c06fe50 0x7f5a5c0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42154/0 (socket says 172.21.15.98:42154) 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 -- 172.21.15.98:0/1313415840 learned_addr learned my addr 172.21.15.98:0/1313415840 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): get_auth_request con 0x7f5a5c139b80 auth_method 0 2024-09-11T03:03:13.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 10 monclient(hunting): get_auth_request con 0x7f5a5c06fe50 auth_method 0 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a62130640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c072640 0x7f5a5c0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a62130640 10 monclient(hunting): get_auth_request con 0x7f5a5c072640 auth_method 0 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a62130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a62130640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a62130640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient(hunting): handle_auth_done global_id 84201 payload 274 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 -- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5a5c06fe50 msgr2=0x7f5a5c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5a5c06fe50 0x7f5a5c0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 -- 172.21.15.98:0/1313415840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c072640 msgr2=0x7f5a5c0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c072640 0x7f5a5c0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 monclient: found mon.noname-d 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 -- 172.21.15.98:0/1313415840 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5c071370 con 0x7f5a5c139b80 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6112e640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5a5c06fe50 0x7f5a5c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.795+0000 7f5a6192f640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c139b80 0x7f5a5c139f60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a580098d0 tx=0x7f5a5801c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4591f32f4ba2a68b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 1 -- 172.21.15.98:0/1313415840 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a580b0070 con 0x7f5a5c139b80 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: dump: 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.810 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 1 -- 172.21.15.98:0/1313415840 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a580aa810 con 0x7f5a5c139b80 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 5 monclient: authenticate success, global_id 84201 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 1 -- 172.21.15.98:0/1313415840 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a580a9300 con 0x7f5a5c139b80 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 4 set_mon_vals no callback set 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6092d640 10 monclient: dump: 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.812 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.813 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: shutdown 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c139b80 msgr2=0x7f5a5c139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c139b80 0x7f5a5c139f60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a580098d0 tx=0x7f5a5801c440 comp rx=0 tx=0).stop 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- 172.21.15.98:0/1313415840 shutdown_connections 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5a5c06fe50 0x7f5a5c0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c072640 0x7f5a5c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 --2- 172.21.15.98:0/1313415840 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c139b80 0x7f5a5c139f60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- 172.21.15.98:0/1313415840 >> 172.21.15.98:0/1313415840 conn(0x7f5a5c0ac730 msgr2=0x7f5a5c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- 172.21.15.98:0/1313415840 shutdown_connections 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- 172.21.15.98:0/1313415840 wait complete. 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: monmap: 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.815 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 librados: starting msgr at 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 librados: starting objecter 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 Processor -- start 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- start start 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 librados: setting wanted keys 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 librados: calling monclient init 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: init 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: _add_conns ranks=[2,6,3,4,1,7,8,0,5] 2024-09-11T03:03:13.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a5c06fe50 0x7f5a5c13a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): picked mon.noname-b con 0x7f5a5c06fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c072640 0x7f5a5c13abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): picked mon.noname-d con 0x7f5a5c072640 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c139b80 0x7f5a5c13b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): picked mon.noname-g con 0x7f5a5c139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5c1435c0 con 0x7f5a5c06fe50 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5c143440 con 0x7f5a5c139b80 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5a5c143740 con 0x7f5a5c072640 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient(hunting): authenticate will time out at 1747.682295s 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c072640 0x7f5a5c13abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a5c06fe50 0x7f5a5c13a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c072640 0x7f5a5c13abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34394/0 (socket says 172.21.15.98:34394) 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 1 -- 172.21.15.98:0/3121539798 learned_addr learned my addr 172.21.15.98:0/3121539798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 10 monclient(hunting): get_auth_request con 0x7f5a5c072640 auth_method 0 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): get_auth_request con 0x7f5a5c06fe50 auth_method 0 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient(hunting): handle_auth_done global_id 64676 payload 824 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a62130640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c139b80 0x7f5a5c13b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c072640 msgr2=0x7f5a5c13abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c072640 0x7f5a5c13abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c139b80 msgr2=0x7f5a5c13b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c139b80 0x7f5a5c13b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 monclient: found mon.noname-b 2024-09-11T03:03:13.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 -- 172.21.15.98:0/3121539798 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5c13cf40 con 0x7f5a5c06fe50 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a62130640 10 monclient: get_auth_request con 0x7f5a5c139b80 auth_method 0 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6112e640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c072640 0x7f5a5c13abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a62130640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5a5c139b80 0x7f5a5c13b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a6192f640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a5c06fe50 0x7f5a5c13a6b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a580ab000 tx=0x7f5a580191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 client.?.objecter ms_handle_connect 0x7f5a5c06fe50 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a580b0040 con 0x7f5a5c06fe50 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: dump: 2024-09-11T03:03:13.820 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.821 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 5 monclient: authenticate success, global_id 64676 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: _renew_subs 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a580044b0 con 0x7f5a5c06fe50 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5c13d220 con 0x7f5a5c06fe50 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a580aac80 con 0x7f5a5c06fe50 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a53fff640 4 set_mon_vals no callback set 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 client.64676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a527fc640 10 monclient: dump: 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: _renew_subs 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5c135910 con 0x7f5a5c06fe50 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.799+0000 7f5a63b93640 1 librados: init done 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5a580b3070 con 0x7f5a5c06fe50 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a63b93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a63b93640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c133760 0x7f5a5c133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5a5c06ca10 con 0x7f5a5c133760 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5a4003df40 0x7f5a40040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(528..528 src has 1..528) ==== 6662+0+0 (secure 0 0 0) 0x7f5a580e9750 con 0x7f5a5c06fe50 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 client.64676.objecter ms_dispatch 0x7f5a5c000ba0 osd_map(528..528 src has 1..528) 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 3 client.64676.objecter handle_osd_map got epochs [528,528] > 0 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 3 client.64676.objecter handle_osd_map decoding full epoch 528 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c133760 0x7f5a5c133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient: get_auth_request con 0x7f5a5c133760 auth_method 0 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 20 client.64676.objecter dump_active .. 0 homeless 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 10 monclient(hunting): handle_auth_done global_id 84219 payload 824 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 1 -- 172.21.15.98:0/3121539798 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a5c0aa4e0 con 0x7f5a5c133760 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a62130640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5a4003df40 0x7f5a40040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a62130640 10 monclient: get_auth_request con 0x7f5a4003df40 auth_method 0 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 client.64676.objecter ms_handle_connect 0x7f5a5c133760 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 client.64676.objecter resend_mon_ops 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a6112e640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c133760 0x7f5a5c133b40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5d40 tx=0x7f5a4c00f610 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a4c0148b0 con 0x7f5a5c133760 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 monclient: dump: 2024-09-11T03:03:13.826 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.827 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5a4c014440 con 0x7f5a5c133760 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c133760 msgr2=0x7f5a5c133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c133760 0x7f5a5c133b40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5d40 tx=0x7f5a4c00f610 comp rx=0 tx=0).stop 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c133760 msgr2=0x7f5a5c133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c133760 0x7f5a5c133b40 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5d40 tx=0x7f5a4c00f610 comp rx=0 tx=0).stop 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a527fc640 10 client.64676.objecter ms_handle_connect 0x7f5a4003df40 2024-09-11T03:03:13.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.803+0000 7f5a62130640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5a4003df40 0x7f5a40040400 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0704e0 tx=0x7f5a5400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a63b93640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:13.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a63b93640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c134080 0x7f5a5c137b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a63b93640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5a5c06e870 con 0x7f5a5c134080 2024-09-11T03:03:13.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c134080 0x7f5a5c137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient: get_auth_request con 0x7f5a5c134080 auth_method 0 2024-09-11T03:03:13.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 10 monclient(hunting): handle_auth_done global_id 84228 payload 824 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.807+0000 7f5a6112e640 1 -- 172.21.15.98:0/3121539798 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a5c140140 con 0x7f5a5c134080 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a6112e640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c134080 0x7f5a5c137b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5d40 tx=0x7f5a4c0046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 10 client.64676.objecter ms_handle_connect 0x7f5a5c134080 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 10 client.64676.objecter resend_mon_ops 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a4c0044a0 con 0x7f5a5c134080 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 10 monclient: dump: 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.832 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9266 (secure 0 0 0) 0x7f5a4c004030 con 0x7f5a5c134080 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c134080 msgr2=0x7f5a5c137b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c134080 0x7f5a5c137b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5d40 tx=0x7f5a4c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c134080 msgr2=0x7f5a5c137b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a527fc640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5a5c134080 0x7f5a5c137b30 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0a5d40 tx=0x7f5a4c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.833 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 19623, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:13.834 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.835 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.836 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:13.837 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:13.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:13.840 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:13.842 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.843 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.844 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:13.845 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.846 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:13.848 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.849 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:13.852 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:13.853 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.854 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:13.855 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:13.856 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 10 librados: watch_flush enter 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 10 librados: watch_flush exit 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 20 client.64676.objecter shutdown clearing up homeless session... 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 10 client.64676.objecter successfully canceled tick 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5a4003df40 msgr2=0x7f5a40040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5a4003df40 0x7f5a40040400 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c0704e0 tx=0x7f5a5400b040 comp rx=0 tx=0).stop 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 10 monclient: shutdown 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a5c06fe50 msgr2=0x7f5a5c13a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 --2- 172.21.15.98:0/3121539798 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5a5c06fe50 0x7f5a5c13a6b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a580ab000 tx=0x7f5a580191e0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a62130640 1 -- 172.21.15.98:0/3121539798 reap_dead start 2024-09-11T03:03:13.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a62130640 1 -- 172.21.15.98:0/3121539798 reap_dead start 2024-09-11T03:03:13.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 shutdown_connections 2024-09-11T03:03:13.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 >> 172.21.15.98:0/3121539798 conn(0x7f5a5c0ac730 msgr2=0x7f5a5c0a79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 shutdown_connections 2024-09-11T03:03:13.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 -- 172.21.15.98:0/3121539798 wait complete. 2024-09-11T03:03:13.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.811+0000 7f5a63b93640 1 librados: shutdown 2024-09-11T03:03:13.859 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:13.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T03:03:13.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:03:13.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient: get_monmap_and_config 2024-09-11T03:03:13.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient: monmap: 2024-09-11T03:03:13.975 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:13.977595+0000 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:13.977595+0000 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.976 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.977 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.977 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.977 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:13.977 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:13.977 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 Processor -- start 2024-09-11T03:03:13.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 -- start start 2024-09-11T03:03:13.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient: init 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient: _add_conns ranks=[8,10,3,6,0,5,9,1,7,2,4] 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc139b10 0x7f76bc072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): picked mon.noname-e con 0x7f76bc139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f76bc139ef0 0x7f76bc0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): picked mon.noname-f con 0x7f76bc139ef0 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc13a2d0 0x7f76bc0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): picked mon.noname-g con 0x7f76bc13a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f76bc0702b0 con 0x7f76bc139ef0 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc071bf0 con 0x7f76bc13a2d0 2024-09-11T03:03:13.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f76bc071350 con 0x7f76bc139b10 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c32ce640 10 monclient(hunting): authenticate will time out at 1747.857829s 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc139b10 0x7f76bc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc139b10 0x7f76bc072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:52426/0 (socket says 172.21.15.98:52426) 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 -- 172.21.15.98:0/1934134957 learned_addr learned my addr 172.21.15.98:0/1934134957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): get_auth_request con 0x7f76bc139b10 auth_method 0 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c0869640 1 --2- 172.21.15.98:0/1934134957 >> v2:172.21.15.190:3300/0 conn(0x7f76bc139ef0 0x7f76bc0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c186b640 1 --2- 172.21.15.98:0/1934134957 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc13a2d0 0x7f76bc0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c0869640 10 monclient(hunting): get_auth_request con 0x7f76bc139ef0 auth_method 0 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c0869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c0869640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c0869640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c186b640 10 monclient(hunting): get_auth_request con 0x7f76bc13a2d0 auth_method 0 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c186b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c186b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c186b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient(hunting): handle_auth_done global_id 25037 payload 274 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 -- 172.21.15.98:0/1934134957 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc13a2d0 msgr2=0x7f76bc0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 --2- 172.21.15.98:0/1934134957 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc13a2d0 0x7f76bc0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 -- 172.21.15.98:0/1934134957 >> v2:172.21.15.190:3300/0 conn(0x7f76bc139ef0 msgr2=0x7f76bc0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 --2- 172.21.15.98:0/1934134957 >> v2:172.21.15.190:3300/0 conn(0x7f76bc139ef0 0x7f76bc0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient: _finish_hunting 0 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 monclient: found mon.noname-e 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.975+0000 7f76c106a640 1 -- 172.21.15.98:0/1934134957 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc0a9620 con 0x7f76bc139b10 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 --2- 172.21.15.98:0/1934134957 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc139b10 0x7f76bc072600 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009870 tx=0x7f76b801c4b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=1773396fbdb34415 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 1 -- 172.21.15.98:0/1934134957 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b80b0070 con 0x7f76bc139b10 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: dump: 2024-09-11T03:03:13.984 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.985 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: _finish_auth 0 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 1 -- 172.21.15.98:0/1934134957 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b80aa810 con 0x7f76bc139b10 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 5 monclient: authenticate success, global_id 25037 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 1 -- 172.21.15.98:0/1934134957 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b80a9300 con 0x7f76bc139b10 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 4 set_mon_vals no callback set 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:13.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b3fff640 10 monclient: dump: 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:13.987 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient: shutdown 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- 172.21.15.98:0/1934134957 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc139b10 msgr2=0x7f76bc072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 --2- 172.21.15.98:0/1934134957 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc139b10 0x7f76bc072600 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009870 tx=0x7f76b801c4b0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- 172.21.15.98:0/1934134957 shutdown_connections 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 --2- 172.21.15.98:0/1934134957 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc13a2d0 0x7f76bc0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 --2- 172.21.15.98:0/1934134957 >> v2:172.21.15.190:3300/0 conn(0x7f76bc139ef0 0x7f76bc0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 --2- 172.21.15.98:0/1934134957 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc139b10 0x7f76bc072600 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:13.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- 172.21.15.98:0/1934134957 >> 172.21.15.98:0/1934134957 conn(0x7f76bc0ac790 msgr2=0x7f76bc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- 172.21.15.98:0/1934134957 shutdown_connections 2024-09-11T03:03:13.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- 172.21.15.98:0/1934134957 wait complete. 2024-09-11T03:03:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient: build_initial_monmap 2024-09-11T03:03:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient: monmap: 2024-09-11T03:03:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:13.995 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:13.996 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 librados: starting msgr at 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 librados: starting objecter 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 Processor -- start 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- start start 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 librados: setting wanted keys 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 librados: calling monclient init 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient: init 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient: _add_conns ranks=[6,3,1,0,4,8,5,7,2] 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f76bc139b10 0x7f76bc13a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): picked mon.noname-d con 0x7f76bc139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc139ef0 0x7f76bc13acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): picked mon.noname-g con 0x7f76bc139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76bc13a2d0 0x7f76bc13b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): picked mon.noname-f con 0x7f76bc13a2d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc143460 con 0x7f76bc13a2d0 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc0a88a0 con 0x7f76bc139ef0 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f76bc1435e0 con 0x7f76bc139b10 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c32ce640 10 monclient(hunting): authenticate will time out at 1747.863189s 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f76bc139b10 0x7f76bc13a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f76bc139b10 0x7f76bc13a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34428/0 (socket says 172.21.15.98:34428) 2024-09-11T03:03:13.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 -- 172.21.15.98:0/2283686690 learned_addr learned my addr 172.21.15.98:0/2283686690 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): get_auth_request con 0x7f76bc139b10 auth_method 0 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c186b640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76bc13a2d0 0x7f76bc13b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c0869640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc139ef0 0x7f76bc13acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c186b640 10 monclient(hunting): get_auth_request con 0x7f76bc13a2d0 auth_method 0 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c186b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c186b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c186b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:13.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient(hunting): handle_auth_done global_id 84237 payload 824 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc139ef0 msgr2=0x7f76bc13acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc139ef0 0x7f76bc13acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76bc13a2d0 msgr2=0x7f76bc13b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76bc13a2d0 0x7f76bc13b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 monclient: found mon.noname-d 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 -- 172.21.15.98:0/2283686690 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc13ce10 con 0x7f76bc139b10 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c0869640 10 monclient: get_auth_request con 0x7f76bc139ef0 auth_method 0 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c0869640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f76bc139ef0 0x7f76bc13acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:14.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76c106a640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f76bc139b10 0x7f76bc13a780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f76b80097c0 tx=0x7f76b8019250 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b1ffb640 10 client.?.objecter ms_handle_connect 0x7f76bc139b10 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.979+0000 7f76b1ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b80b9070 con 0x7f76bc139b10 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: dump: 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.001 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b8003620 con 0x7f76bc139b10 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 5 monclient: authenticate success, global_id 84237 2024-09-11T03:03:14.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 10 monclient: _renew_subs 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76bc13d0f0 con 0x7f76bc139b10 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76b80b0040 con 0x7f76bc139b10 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: dump: 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.003 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b37fe640 4 set_mon_vals no callback set 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 10 client.84237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:14.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 10 monclient: _renew_subs 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f76bc0aa470 con 0x7f76bc139b10 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 1 librados: init done 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc13bbd0 0x7f76bc1337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f76bc06ca10 con 0x7f76bc13bbd0 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc13bbd0 0x7f76bc1337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient: get_auth_request con 0x7f76bc13bbd0 auth_method 0 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f76b8004100 con 0x7f76bc139b10 2024-09-11T03:03:14.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f76a003df40 0x7f76a0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 10 monclient(hunting): handle_auth_done global_id 25046 payload 824 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 1 -- 172.21.15.98:0/2283686690 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76bc06e680 con 0x7f76bc13bbd0 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(528..528 src has 1..528) ==== 6662+0+0 (secure 0 0 0) 0x7f76b80e8eb0 con 0x7f76bc139b10 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 client.84237.objecter ms_dispatch 0x7f76bc000ba0 osd_map(528..528 src has 1..528) 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 3 client.84237.objecter handle_osd_map got epochs [528,528] > 0 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 3 client.84237.objecter handle_osd_map decoding full epoch 528 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 20 client.84237.objecter dump_active .. 0 homeless 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 client.84237.objecter ms_handle_connect 0x7f76bc13bbd0 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 client.84237.objecter resend_mon_ops 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c0869640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc13bbd0 0x7f76bc1337f0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0a6010 tx=0x7f76ac00eba0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76ac017070 con 0x7f76bc13bbd0 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: dump: 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.007 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c186b640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f76a003df40 0x7f76a0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c186b640 10 monclient: get_auth_request con 0x7f76a003df40 auth_method 0 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f76ac00ecd0 con 0x7f76bc13bbd0 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc13bbd0 msgr2=0x7f76bc1337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc13bbd0 0x7f76bc1337f0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0a6010 tx=0x7f76ac00eba0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc13bbd0 msgr2=0x7f76bc1337f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc13bbd0 0x7f76bc1337f0 secure :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0a6010 tx=0x7f76ac00eba0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76b1ffb640 10 client.84237.objecter ms_handle_connect 0x7f76a003df40 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.983+0000 7f76c186b640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f76a003df40 0x7f76a0040400 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f76bc139340 tx=0x7f76b400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.987+0000 7f76c32ce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.987+0000 7f76c32ce640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc133d30 0x7f76bc134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.987+0000 7f76c32ce640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.987+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f76bc06e870 con 0x7f76bc133d30 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc133d30 0x7f76bc134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient: get_auth_request con 0x7f76bc133d30 auth_method 0 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 10 monclient(hunting): handle_auth_done global_id 25055 payload 824 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 1 -- 172.21.15.98:0/2283686690 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f76bc134680 con 0x7f76bc133d30 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 10 client.84237.objecter ms_handle_connect 0x7f76bc133d30 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 10 client.84237.objecter resend_mon_ops 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c0869640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc133d30 0x7f76bc134140 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0a6010 tx=0x7f76ac0046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76ac00f040 con 0x7f76bc133d30 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 10 monclient: dump: 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.011 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f76ac00ce00 con 0x7f76bc133d30 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc133d30 msgr2=0x7f76bc134140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc133d30 0x7f76bc134140 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0a6010 tx=0x7f76ac0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc133d30 msgr2=0x7f76bc134140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76b1ffb640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f76bc133d30 0x7f76bc134140 secure :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f76bc0a6010 tx=0x7f76ac0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 809783, 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:14.013 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 4, 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.014 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:14.015 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.016 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:14.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.018 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.019 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.020 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.023 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:14.024 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.025 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.027 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:14.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:14.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:14.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.035 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.036 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:14.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.036 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:14.036 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:14.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 10 librados: watch_flush enter 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 10 librados: watch_flush exit 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 20 client.84237.objecter shutdown clearing up homeless session... 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 10 client.84237.objecter successfully canceled tick 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f76a003df40 msgr2=0x7f76a0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f76a003df40 0x7f76a0040400 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f76bc139340 tx=0x7f76b400b040 comp rx=0 tx=0).stop 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 10 monclient: shutdown 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f76bc139b10 msgr2=0x7f76bc13a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 --2- 172.21.15.98:0/2283686690 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f76bc139b10 0x7f76bc13a780 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f76b80097c0 tx=0x7f76b8019250 comp rx=0 tx=0).stop 2024-09-11T03:03:14.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c186b640 1 -- 172.21.15.98:0/2283686690 reap_dead start 2024-09-11T03:03:14.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c186b640 1 -- 172.21.15.98:0/2283686690 reap_dead start 2024-09-11T03:03:14.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 shutdown_connections 2024-09-11T03:03:14.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 >> 172.21.15.98:0/2283686690 conn(0x7f76bc0ac790 msgr2=0x7f76bc0aa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 shutdown_connections 2024-09-11T03:03:14.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 -- 172.21.15.98:0/2283686690 wait complete. 2024-09-11T03:03:14.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:13.991+0000 7f76c32ce640 1 librados: shutdown 2024-09-11T03:03:14.039 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:14.039 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T03:03:14.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:03:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 10 monclient: get_monmap_and_config 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 10 monclient: monmap: 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:14.159791+0000 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:14.159791+0000 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.158 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:14.159 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:14.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 1 Processor -- start 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 1 -- start start 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 10 monclient: init 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 10 monclient: _add_conns ranks=[6,8,5,1,0,3,7,2,10,4,9] 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.155+0000 7f2f6cea8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f68072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): picked mon.noname-d con 0x7f2f68139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2f68139ef0 0x7f2f680a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): picked mon.noname-e con 0x7f2f68139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2f6813a2d0 0x7f2f680a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): picked mon.noname-h con 0x7f2f6813a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f2f680702b0 con 0x7f2f6813a2d0 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2f68071bf0 con 0x7f2f68139b10 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2f68071350 con 0x7f2f68139ef0 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient(hunting): authenticate will time out at 1748.040092s 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f68072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f68072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34434/0 (socket says 172.21.15.98:34434) 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 -- 172.21.15.98:0/278454187 learned_addr learned my addr 172.21.15.98:0/278454187 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): get_auth_request con 0x7f2f68139b10 auth_method 0 2024-09-11T03:03:14.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2f68139ef0 0x7f2f680a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 10 monclient(hunting): get_auth_request con 0x7f2f68139ef0 auth_method 0 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f66d9d640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2f6813a2d0 0x7f2f680a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f66d9d640 10 monclient(hunting): get_auth_request con 0x7f2f6813a2d0 auth_method 0 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f66d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f66d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f66d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient(hunting): handle_auth_done global_id 84246 payload 274 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 -- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2f68139ef0 msgr2=0x7f2f680a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2f68139ef0 0x7f2f680a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 -- 172.21.15.98:0/278454187 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2f6813a2d0 msgr2=0x7f2f680a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2f6813a2d0 0x7f2f680a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 monclient: found mon.noname-d 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 -- 172.21.15.98:0/278454187 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f680a9620 con 0x7f2f68139b10 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6659c640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f68072600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009e80 tx=0x7f2f6001c910 comp rx=0 tx=0).ready entity=mon.6 client_cookie=217ba832f4d11bbd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 1 -- 172.21.15.98:0/278454187 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2f600090d0 con 0x7f2f68139b10 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: dump: 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.168 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 1 -- 172.21.15.98:0/278454187 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f600a5a70 con 0x7f2f68139b10 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 5 monclient: authenticate success, global_id 84246 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 1 -- 172.21.15.98:0/278454187 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2f600a5c60 con 0x7f2f68139b10 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 4 set_mon_vals no callback set 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6559a640 10 monclient: dump: 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.170 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.171 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 10 monclient: shutdown 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 1 -- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 msgr2=0x7f2f68072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.159+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f68072600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009e80 tx=0x7f2f6001c910 comp rx=0 tx=0).stop 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- 172.21.15.98:0/278454187 shutdown_connections 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2f6813a2d0 0x7f2f680a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2f68139ef0 0x7f2f680a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/278454187 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f68072600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- 172.21.15.98:0/278454187 >> 172.21.15.98:0/278454187 conn(0x7f2f680ac790 msgr2=0x7f2f680a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- 172.21.15.98:0/278454187 shutdown_connections 2024-09-11T03:03:14.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- 172.21.15.98:0/278454187 wait complete. 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: monmap: 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.173 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 librados: starting msgr at 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 librados: starting objecter 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 Processor -- start 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- start start 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 librados: setting wanted keys 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 librados: calling monclient init 2024-09-11T03:03:14.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: init 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: _add_conns ranks=[6,4,1,5,2,7,3,0,8] 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f681349d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): picked mon.noname-d con 0x7f2f68139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2f68139ef0 0x7f2f68134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): picked mon.noname-c con 0x7f2f68139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813a2d0 0x7f2f6813a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): picked mon.noname-f con 0x7f2f6813a2d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68143460 con 0x7f2f6813a2d0 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f2f680a88a0 con 0x7f2f68139ef0 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2f681435e0 con 0x7f2f68139b10 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): authenticate will time out at 1748.045675s 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f681349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f681349d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34450/0 (socket says 172.21.15.98:34450) 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 -- 172.21.15.98:0/4046381322 learned_addr learned my addr 172.21.15.98:0/4046381322 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): get_auth_request con 0x7f2f68139b10 auth_method 0 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2f68139ef0 0x7f2f68134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): get_auth_request con 0x7f2f68139ef0 auth_method 0 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient(hunting): handle_auth_done global_id 84255 payload 824 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f66d9d640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813a2d0 0x7f2f6813a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2f68139ef0 msgr2=0x7f2f68134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f2f68139ef0 0x7f2f68134f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813a2d0 msgr2=0x7f2f6813a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813a2d0 0x7f2f6813a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 monclient: found mon.noname-d 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 -- 172.21.15.98:0/4046381322 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f680aa1e0 con 0x7f2f68139b10 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6659c640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f681349d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009e20 tx=0x7f2f600196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 client.?.objecter ms_handle_connect 0x7f2f68139b10 2024-09-11T03:03:14.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2f600bc020 con 0x7f2f68139b10 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: dump: 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.185 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 5 monclient: authenticate success, global_id 84255 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: _renew_subs 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f60003e10 con 0x7f2f68139b10 2024-09-11T03:03:14.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6813dfa0 con 0x7f2f68139b10 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2f600044b0 con 0x7f2f68139b10 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f64d99640 4 set_mon_vals no callback set 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 client.84255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 monclient: dump: 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.187 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: _renew_subs 2024-09-11T03:03:14.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f6813ca80 con 0x7f2f68139b10 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 librados: init done 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813ad10 0x7f2f6813b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2f68074a50 con 0x7f2f6813ad10 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2f60004650 con 0x7f2f68139b10 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2f4803df40 0x7f2f48040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813ad10 0x7f2f6813b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(528..528 src has 1..528) ==== 6662+0+0 (secure 0 0 0) 0x7f2f600eaa90 con 0x7f2f68139b10 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 10 client.84255.objecter ms_dispatch 0x7f2f68000ba0 osd_map(528..528 src has 1..528) 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 3 client.84255.objecter handle_osd_map got epochs [528,528] > 0 2024-09-11T03:03:14.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 3 client.84255.objecter handle_osd_map decoding full epoch 528 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient: get_auth_request con 0x7f2f6813ad10 auth_method 0 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f56ffd640 20 client.84255.objecter dump_active .. 0 homeless 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.163+0000 7f2f66d9d640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2f4803df40 0x7f2f48040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f66d9d640 10 monclient: get_auth_request con 0x7f2f4803df40 auth_method 0 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_done global_id 25876 payload 824 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f65d9b640 1 -- 172.21.15.98:0/4046381322 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f6806e680 con 0x7f2f6813ad10 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 client.84255.objecter ms_handle_connect 0x7f2f6813ad10 2024-09-11T03:03:14.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 client.84255.objecter resend_mon_ops 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f65d9b640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813ad10 0x7f2f6813b0f0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f2f680a5d10 tx=0x7f2f5c011e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 client.84255.objecter ms_handle_connect 0x7f2f4803df40 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f66d9d640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2f4803df40 0x7f2f48040400 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f2f6813e770 tx=0x7f2f58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2f5c019070 con 0x7f2f6813ad10 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 monclient: dump: 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2f5c004240 con 0x7f2f6813ad10 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813ad10 msgr2=0x7f2f6813b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813ad10 0x7f2f6813b0f0 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f2f680a5d10 tx=0x7f2f5c011e30 comp rx=0 tx=0).stop 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813ad10 msgr2=0x7f2f6813b0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.167+0000 7f2f56ffd640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813ad10 0x7f2f6813b0f0 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.171+0000 7f2f6cea8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.171+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813b630 0x7f2f680a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.171+0000 7f2f6cea8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.171+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2f6806ca10 con 0x7f2f6813b630 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813b630 0x7f2f680a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient: get_auth_request con 0x7f2f6813b630 auth_method 0 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 10 monclient(hunting): handle_auth_done global_id 25885 payload 824 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 1 -- 172.21.15.98:0/4046381322 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f680aa410 con 0x7f2f6813b630 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f65d9b640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813b630 0x7f2f680a0e60 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c01b000 tx=0x7f2f5c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 10 client.84255.objecter ms_handle_connect 0x7f2f6813b630 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 10 client.84255.objecter resend_mon_ops 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2f5c019040 con 0x7f2f6813b630 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 10 monclient: dump: 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.194 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.195 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f2f5c01e450 con 0x7f2f6813b630 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813b630 msgr2=0x7f2f680a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813b630 0x7f2f680a0e60 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c01b000 tx=0x7f2f5c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813b630 msgr2=0x7f2f680a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f56ffd640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2f6813b630 0x7f2f680a0e60 secure :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c01b000 tx=0x7f2f5c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.196 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 809949, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:14.197 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.198 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.199 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:14.200 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.201 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:14.202 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:14.203 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.204 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.205 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.207 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.212 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:14.213 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:14.216 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.218 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.219 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.220 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.220 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.220 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:14.220 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:14.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 10 librados: watch_flush enter 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 10 librados: watch_flush exit 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 20 client.84255.objecter shutdown clearing up homeless session... 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 10 client.84255.objecter successfully canceled tick 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2f4803df40 msgr2=0x7f2f48040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2f4803df40 0x7f2f48040400 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f2f6813e770 tx=0x7f2f58008040 comp rx=0 tx=0).stop 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 10 monclient: shutdown 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 msgr2=0x7f2f681349d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 --2- 172.21.15.98:0/4046381322 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2f68139b10 0x7f2f681349d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2f60009e20 tx=0x7f2f600196b0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f66d9d640 1 -- 172.21.15.98:0/4046381322 reap_dead start 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f66d9d640 1 -- 172.21.15.98:0/4046381322 reap_dead start 2024-09-11T03:03:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 shutdown_connections 2024-09-11T03:03:14.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 >> 172.21.15.98:0/4046381322 conn(0x7f2f680ac790 msgr2=0x7f2f680aa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 shutdown_connections 2024-09-11T03:03:14.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 -- 172.21.15.98:0/4046381322 wait complete. 2024-09-11T03:03:14.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.175+0000 7f2f6cea8640 1 librados: shutdown 2024-09-11T03:03:14.223 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:14.223 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T03:03:14.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.335+0000 7fa2c301e640 10 monclient: get_monmap_and_config 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.335+0000 7fa2c301e640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.335+0000 7fa2c301e640 10 monclient: monmap: 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:14.341123+0000 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:14.341123+0000 2024-09-11T03:03:14.339 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.340 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:14.341 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:14.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 Processor -- start 2024-09-11T03:03:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 -- start start 2024-09-11T03:03:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient: init 2024-09-11T03:03:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient: _add_conns ranks=[3,10,5,6,0,4,1,7,8,2,9] 2024-09-11T03:03:14.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139b10 0x7fa2bc072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): picked mon.noname-g con 0x7fa2bc139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa2bc139ef0 0x7fa2bc0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): picked mon.noname-f con 0x7fa2bc139ef0 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa2bc13a2d0 0x7fa2bc0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): picked mon.noname-h con 0x7fa2bc13a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa2bc0702b0 con 0x7fa2bc139ef0 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc071bf0 con 0x7fa2bc139b10 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa2bc071350 con 0x7fa2bc13a2d0 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c301e640 10 monclient(hunting): authenticate will time out at 1748.221744s 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa2bc13a2d0 0x7fa2bc0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa2bc139ef0 0x7fa2bc0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139b10 0x7fa2bc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa2bc139ef0 0x7fa2bc0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:55000/0 (socket says 172.21.15.98:55000) 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa2bc13a2d0 0x7fa2bc0a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:55428/0 (socket says 172.21.15.98:55428) 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 1 -- 172.21.15.98:0/816132908 learned_addr learned my addr 172.21.15.98:0/816132908 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 10 monclient(hunting): get_auth_request con 0x7fa2bc13a2d0 auth_method 0 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): get_auth_request con 0x7fa2bc139ef0 auth_method 0 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 10 monclient(hunting): get_auth_request con 0x7fa2bc139b10 auth_method 0 2024-09-11T03:03:14.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c15bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2c0dba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_done global_id 25894 payload 274 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 -- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa2bc13a2d0 msgr2=0x7fa2bc0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa2bc13a2d0 0x7fa2bc0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 -- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139b10 msgr2=0x7fa2bc072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139b10 0x7fa2bc072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 monclient: found mon.noname-f 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:03:14.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.339+0000 7fa2bbfff640 1 -- 172.21.15.98:0/816132908 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc0a9620 con 0x7fa2bc139ef0 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2bc139ef0 0x7fa2bc0a7a10 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097f0 tx=0x7fa2b001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c64d3270d32bf85 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 1 -- 172.21.15.98:0/816132908 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b00b0070 con 0x7fa2bc139ef0 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: dump: 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.357 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 5 monclient: authenticate success, global_id 25894 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 1 -- 172.21.15.98:0/816132908 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b00aa810 con 0x7fa2bc139ef0 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 1 -- 172.21.15.98:0/816132908 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b00a9300 con 0x7fa2bc139ef0 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 4 set_mon_vals no callback set 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2bb7fe640 10 monclient: dump: 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.359 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.360 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 10 monclient: shutdown 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 -- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2bc139ef0 msgr2=0x7fa2bc0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 --2- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2bc139ef0 0x7fa2bc0a7a10 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097f0 tx=0x7fa2b001c3f0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 -- 172.21.15.98:0/816132908 shutdown_connections 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 --2- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa2bc13a2d0 0x7fa2bc0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 --2- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2bc139ef0 0x7fa2bc0a7a10 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 --2- 172.21.15.98:0/816132908 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139b10 0x7fa2bc072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 -- 172.21.15.98:0/816132908 >> 172.21.15.98:0/816132908 conn(0x7fa2bc0ac790 msgr2=0x7fa2bc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 -- 172.21.15.98:0/816132908 shutdown_connections 2024-09-11T03:03:14.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 -- 172.21.15.98:0/816132908 wait complete. 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 10 monclient: monmap: 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 librados: starting msgr at 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 librados: starting objecter 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.343+0000 7fa2c301e640 1 Processor -- start 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 -- start start 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 librados: setting wanted keys 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 librados: calling monclient init 2024-09-11T03:03:14.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient: init 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient: _add_conns ranks=[2,3,8,6,7,4,1,0,5] 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2bc139b10 0x7fa2bc13a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): picked mon.noname-b con 0x7fa2bc139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139ef0 0x7fa2bc13acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): picked mon.noname-g con 0x7fa2bc139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2bc13a2d0 0x7fa2bc13b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): picked mon.noname-e con 0x7fa2bc13a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc143460 con 0x7fa2bc139b10 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc0a88a0 con 0x7fa2bc139ef0 2024-09-11T03:03:14.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fa2bc1435e0 con 0x7fa2bc13a2d0 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): authenticate will time out at 1748.228685s 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2bc139b10 0x7fa2bc13a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2bc13a2d0 0x7fa2bc13b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2bc139b10 0x7fa2bc13a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40334/0 (socket says 172.21.15.98:40334) 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 -- 172.21.15.98:0/2486825481 learned_addr learned my addr 172.21.15.98:0/2486825481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): get_auth_request con 0x7fa2bc139b10 auth_method 0 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient(hunting): get_auth_request con 0x7fa2bc13a2d0 auth_method 0 2024-09-11T03:03:14.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139ef0 0x7fa2bc13acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): get_auth_request con 0x7fa2bc139ef0 auth_method 0 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient(hunting): handle_auth_done global_id 64685 payload 824 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2bc13a2d0 msgr2=0x7fa2bc13b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2bc13a2d0 0x7fa2bc13b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139ef0 msgr2=0x7fa2bc13acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc139ef0 0x7fa2bc13acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 monclient: found mon.noname-b 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 -- 172.21.15.98:0/2486825481 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2bc0aa3c0 con 0x7fa2bc139b10 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fa2bc13a2d0 0x7fa2bc13b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c0dba640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2bc139b10 0x7fa2bc13a780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00e930 tx=0x7fa2ac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 client.?.objecter ms_handle_connect 0x7fa2bc139b10 2024-09-11T03:03:14.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2ac017070 con 0x7fa2bc139b10 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: dump: 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.368 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ac013470 con 0x7fa2bc139b10 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 5 monclient: authenticate success, global_id 64685 2024-09-11T03:03:14.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2ac013a90 con 0x7fa2bc139b10 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 monclient: dump: 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.370 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2baffd640 4 set_mon_vals no callback set 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient: _renew_subs 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2bc0a9e90 con 0x7fa2bc139b10 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 client.64685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:14.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient: _renew_subs 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2bc13cda0 con 0x7fa2bc139b10 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 librados: init done 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc133760 0x7fa2bc133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc06ca10 con 0x7fa2bc133760 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa2ac013c30 con 0x7fa2bc139b10 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc133760 0x7fa2bc133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient: get_auth_request con 0x7fa2bc133760 auth_method 0 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa2a803df70 0x7fa2a8040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(528..528 src has 1..528) ==== 6662+0+0 (secure 0 0 0) 0x7fa2ac00f040 con 0x7fa2bc139b10 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 10 client.64685.objecter ms_dispatch 0x7fa2bc000ba0 osd_map(528..528 src has 1..528) 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 3 client.64685.objecter handle_osd_map got epochs [528,528] > 0 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 3 client.64685.objecter handle_osd_map decoding full epoch 528 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2b97fa640 20 client.64685.objecter dump_active .. 0 homeless 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa2a803df70 0x7fa2a8040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2c15bb640 10 monclient: get_auth_request con 0x7fa2a803df70 auth_method 0 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_done global_id 74388 payload 824 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.347+0000 7fa2bbfff640 1 -- 172.21.15.98:0/2486825481 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2bc06e680 con 0x7fa2bc133760 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 client.64685.objecter ms_handle_connect 0x7fa2bc133760 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 client.64685.objecter resend_mon_ops 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc133760 0x7fa2bc133b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0a9590 tx=0x7fa2b0019190 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 client.64685.objecter ms_handle_connect 0x7fa2a803df70 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2c15bb640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa2a803df70 0x7fa2a8040430 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009fd0 tx=0x7fa2b400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b00b9070 con 0x7fa2bc133760 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 monclient: dump: 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.374 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.375 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa2b0003830 con 0x7fa2bc133760 2024-09-11T03:03:14.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:14.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc133760 msgr2=0x7fa2bc133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc133760 0x7fa2bc133b40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc0a9590 tx=0x7fa2b0019190 comp rx=0 tx=0).stop 2024-09-11T03:03:14.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc133760 msgr2=0x7fa2bc133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.351+0000 7fa2b97fa640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc133760 0x7fa2bc133b40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2c301e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2c301e640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc134080 0x7fa2bc0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2c301e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa2bc06e870 con 0x7fa2bc134080 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc134080 0x7fa2bc0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 10 monclient: get_auth_request con 0x7fa2bc134080 auth_method 0 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.355+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2bbfff640 10 monclient(hunting): handle_auth_done global_id 74397 payload 824 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2bbfff640 1 -- 172.21.15.98:0/2486825481 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa2bc072600 con 0x7fa2bc134080 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2bbfff640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc134080 0x7fa2bc0a0e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097f0 tx=0x7fa2b00a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 10 client.64685.objecter ms_handle_connect 0x7fa2bc134080 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 10 client.64685.objecter resend_mon_ops 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa2b00b9040 con 0x7fa2bc134080 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 10 monclient: dump: 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.379 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.380 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7fa2b00bd400 con 0x7fa2bc134080 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc134080 msgr2=0x7fa2bc0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc134080 0x7fa2bc0a0e60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097f0 tx=0x7fa2b00a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc134080 msgr2=0x7fa2bc0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2b97fa640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa2bc134080 0x7fa2bc0a0e60 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097f0 tx=0x7fa2b00a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 151291, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.386 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:14.387 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.388 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:14.389 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.390 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.393 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:14.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.399 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.401 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.402 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.403 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:14.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:14.405 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 10 librados: watch_flush enter 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 10 librados: watch_flush exit 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 20 client.64685.objecter shutdown clearing up homeless session... 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 10 client.64685.objecter successfully canceled tick 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa2a803df70 msgr2=0x7fa2a8040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa2a803df70 0x7fa2a8040430 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fa2b4009fd0 tx=0x7fa2b400b040 comp rx=0 tx=0).stop 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 10 monclient: shutdown 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2bc139b10 msgr2=0x7fa2bc13a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 --2- 172.21.15.98:0/2486825481 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa2bc139b10 0x7fa2bc13a780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac00e930 tx=0x7fa2ac00edf0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c15bb640 1 -- 172.21.15.98:0/2486825481 reap_dead start 2024-09-11T03:03:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c15bb640 1 -- 172.21.15.98:0/2486825481 reap_dead start 2024-09-11T03:03:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 shutdown_connections 2024-09-11T03:03:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 >> 172.21.15.98:0/2486825481 conn(0x7fa2bc0ac790 msgr2=0x7fa2bc0a6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 shutdown_connections 2024-09-11T03:03:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 -- 172.21.15.98:0/2486825481 wait complete. 2024-09-11T03:03:14.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.359+0000 7fa2c301e640 1 librados: shutdown 2024-09-11T03:03:14.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:14.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T03:03:14.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:03:14.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient: get_monmap_and_config 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient: monmap: 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:14.555863+0000 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:14.555863+0000 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.554 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:14.555 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 Processor -- start 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 -- start start 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient: init 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient: _add_conns ranks=[5,9,2,1,0,7,10,3,8,6,4] 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5104139b10 0x7f5104072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): picked mon.noname-h con 0x7f5104139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5104139ef0 0x7f51040a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): picked mon.noname-a con 0x7f5104139ef0 addr v2:172.21.15.98:3300/0 2024-09-11T03:03:14.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510413a2d0 0x7f51040a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): picked mon.noname-b con 0x7f510413a2d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f51040702b0 con 0x7f5104139ef0 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5104071bf0 con 0x7f510413a2d0 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5104071350 con 0x7f5104139b10 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.551+0000 7f510b580640 10 monclient(hunting): authenticate will time out at 1748.435981s 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510413a2d0 0x7f51040a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5104139ef0 0x7f51040a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5104139b10 0x7f5104072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5104139ef0 0x7f51040a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32978/0 (socket says 172.21.15.98:32978) 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510413a2d0 0x7f51040a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40348/0 (socket says 172.21.15.98:40348) 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5104139b10 0x7f5104072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:55440/0 (socket says 172.21.15.98:55440) 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 -- 172.21.15.98:0/3478145638 learned_addr learned my addr 172.21.15.98:0/3478145638 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): get_auth_request con 0x7f5104139ef0 auth_method 0 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 10 monclient(hunting): get_auth_request con 0x7f510413a2d0 auth_method 0 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 10 monclient(hunting): get_auth_request con 0x7f5104139b10 auth_method 0 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510931c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient(hunting): handle_auth_done global_id 45117 payload 274 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 -- 172.21.15.98:0/3478145638 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5104139b10 msgr2=0x7f5104072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5104139b10 0x7f5104072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 -- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510413a2d0 msgr2=0x7f51040a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510413a2d0 0x7f51040a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 monclient: found mon.noname-a 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 -- 172.21.15.98:0/3478145638 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51040a9620 con 0x7f5104139ef0 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5109b1d640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510413a2d0 0x7f51040a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5108b1b640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5104139ef0 0x7f51040a7a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f50f80098a0 tx=0x7f50f801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fada1cb09fbeb19d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 1 -- 172.21.15.98:0/3478145638 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f80b0070 con 0x7f5104139ef0 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: dump: 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.565 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 5 monclient: authenticate success, global_id 45117 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 1 -- 172.21.15.98:0/3478145638 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f80aa810 con 0x7f5104139ef0 2024-09-11T03:03:14.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 4 set_mon_vals no callback set 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 1 -- 172.21.15.98:0/3478145638 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f80a9300 con 0x7f5104139ef0 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f5103fff640 10 monclient: dump: 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.567 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 10 monclient: shutdown 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 1 -- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5104139ef0 msgr2=0x7f51040a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5104139ef0 0x7f51040a7a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f50f80098a0 tx=0x7f50f801c490 comp rx=0 tx=0).stop 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 1 -- 172.21.15.98:0/3478145638 shutdown_connections 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.555+0000 7f510b580640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f510413a2d0 0x7f51040a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5104139ef0 0x7f51040a7a10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 --2- 172.21.15.98:0/3478145638 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5104139b10 0x7f5104072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- 172.21.15.98:0/3478145638 >> 172.21.15.98:0/3478145638 conn(0x7f51040ac790 msgr2=0x7f51040a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- 172.21.15.98:0/3478145638 shutdown_connections 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- 172.21.15.98:0/3478145638 wait complete. 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: monmap: 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:14.569 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 librados: starting msgr at 2024-09-11T03:03:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 librados: starting objecter 2024-09-11T03:03:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 Processor -- start 2024-09-11T03:03:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- start start 2024-09-11T03:03:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 librados: setting wanted keys 2024-09-11T03:03:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 librados: calling monclient init 2024-09-11T03:03:14.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: init 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: _add_conns ranks=[2,6,8,1,4,5,0,7,3] 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5104139b10 0x7f51040a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): picked mon.noname-b con 0x7f5104139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5104139ef0 0x7f510413f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): picked mon.noname-d con 0x7f5104139ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f510413a2d0 0x7f510413fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): picked mon.noname-e con 0x7f510413a2d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5104143460 con 0x7f5104139b10 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f51040a88a0 con 0x7f5104139ef0 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f51041435e0 con 0x7f510413a2d0 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): authenticate will time out at 1748.441747s 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5104139b10 0x7f51040a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5104139b10 0x7f51040a5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40356/0 (socket says 172.21.15.98:40356) 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f510413a2d0 0x7f510413fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 -- 172.21.15.98:0/311548599 learned_addr learned my addr 172.21.15.98:0/311548599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): get_auth_request con 0x7f5104139b10 auth_method 0 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5104139ef0 0x7f510413f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 10 monclient(hunting): get_auth_request con 0x7f510413a2d0 auth_method 0 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 10 monclient(hunting): get_auth_request con 0x7f5104139ef0 auth_method 0 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5109b1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient(hunting): handle_auth_done global_id 64703 payload 824 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f510413a2d0 msgr2=0x7f510413fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f510413a2d0 0x7f510413fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5104139ef0 msgr2=0x7f510413f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5104139ef0 0x7f510413f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 monclient: found mon.noname-b 2024-09-11T03:03:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 -- 172.21.15.98:0/311548599 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51040a9f70 con 0x7f5104139b10 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510931c640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5104139b10 0x7f51040a5010 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f50f400efc0 tx=0x7f50f400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 client.?.objecter ms_handle_connect 0x7f5104139b10 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f4018020 con 0x7f5104139b10 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: dump: 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f400cdb0 con 0x7f5104139b10 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f4014d60 con 0x7f5104139b10 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 monclient: dump: 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.584 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f51037fe640 4 set_mon_vals no callback set 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 5 monclient: authenticate success, global_id 64703 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: _renew_subs 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51040aa4a0 con 0x7f5104139b10 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 client.64703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: _renew_subs 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f510413d410 con 0x7f5104139b10 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 librados: init done 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f50f4014480 con 0x7f5104139b10 2024-09-11T03:03:14.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1570 0x7f51041d1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f510406ca10 con 0x7f51041d1570 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50f003df40 0x7f50f0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(529..529 src has 1..529) ==== 6657+0+0 (secure 0 0 0) 0x7f50f40504d0 con 0x7f5104139b10 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 10 client.64703.objecter ms_dispatch 0x7f5104000ba0 osd_map(529..529 src has 1..529) 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.559+0000 7f5101ffb640 3 client.64703.objecter handle_osd_map got epochs [529,529] > 0 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 3 client.64703.objecter handle_osd_map decoding full epoch 529 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1570 0x7f51041d1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient: get_auth_request con 0x7f51041d1570 auth_method 0 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 20 client.64703.objecter dump_active .. 0 homeless 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5109b1d640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50f003df40 0x7f50f0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5109b1d640 10 monclient: get_auth_request con 0x7f50f003df40 auth_method 0 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 10 monclient(hunting): handle_auth_done global_id 74390 payload 824 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 1 -- 172.21.15.98:0/311548599 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51041403b0 con 0x7f51041d1570 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 client.64703.objecter ms_handle_connect 0x7f51041d1570 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 client.64703.objecter resend_mon_ops 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5108b1b640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1570 0x7f51041d1950 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f51040a9590 tx=0x7f50f8019230 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 client.64703.objecter ms_handle_connect 0x7f50f003df40 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5109b1d640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50f003df40 0x7f50f0040400 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7f50fc0095c0 tx=0x7f50fc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f80ba030 con 0x7f51041d1570 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 monclient: dump: 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.589 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f50f80038d0 con 0x7f51041d1570 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1570 msgr2=0x7f51041d1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1570 0x7f51041d1950 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f51040a9590 tx=0x7f50f8019230 comp rx=0 tx=0).stop 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1570 msgr2=0x7f51041d1950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.563+0000 7f5101ffb640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1570 0x7f51041d1950 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.567+0000 7f510b580640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.567+0000 7f510b580640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1e90 0x7f5104137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.567+0000 7f510b580640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.567+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f510406e870 con 0x7f51041d1e90 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1e90 0x7f5104137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient: get_auth_request con 0x7f51041d1e90 auth_method 0 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 10 monclient(hunting): handle_auth_done global_id 74399 payload 824 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 1 -- 172.21.15.98:0/311548599 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f510406e680 con 0x7f51041d1e90 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5108b1b640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1e90 0x7f5104137ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50f801c4d0 tx=0x7f50f80a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 10 client.64703.objecter ms_handle_connect 0x7f51041d1e90 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 10 client.64703.objecter resend_mon_ops 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f80ba040 con 0x7f51041d1e90 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 10 monclient: dump: 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.592 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.593 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f50f80b6470 con 0x7f51041d1e90 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1e90 msgr2=0x7f5104137ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1e90 0x7f5104137ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50f801c4d0 tx=0x7f50f80a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1e90 msgr2=0x7f5104137ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5101ffb640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f51041d1e90 0x7f5104137ba0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50f801c4d0 tx=0x7f50f80a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 151503, 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:14.595 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.596 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:14.597 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:14.599 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.600 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:14.601 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.604 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.605 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.606 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.608 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.610 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.612 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.614 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.618 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:14.619 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:14.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 10 librados: watch_flush enter 2024-09-11T03:03:14.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 10 librados: watch_flush exit 2024-09-11T03:03:14.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 20 client.64703.objecter shutdown clearing up homeless session... 2024-09-11T03:03:14.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 10 client.64703.objecter successfully canceled tick 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50f003df40 msgr2=0x7f50f0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f50f003df40 0x7f50f0040400 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7f50fc0095c0 tx=0x7f50fc009190 comp rx=0 tx=0).stop 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 10 monclient: shutdown 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5104139b10 msgr2=0x7f51040a5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 --2- 172.21.15.98:0/311548599 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5104139b10 0x7f51040a5010 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f50f400efc0 tx=0x7f50f400c490 comp rx=0 tx=0).stop 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5109b1d640 1 -- 172.21.15.98:0/311548599 reap_dead start 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f5109b1d640 1 -- 172.21.15.98:0/311548599 reap_dead start 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 shutdown_connections 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 >> 172.21.15.98:0/311548599 conn(0x7f51040ac790 msgr2=0x7f51040a7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 shutdown_connections 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 -- 172.21.15.98:0/311548599 wait complete. 2024-09-11T03:03:14.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.571+0000 7f510b580640 1 librados: shutdown 2024-09-11T03:03:14.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:14.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T03:03:14.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient: get_monmap_and_config 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient: monmap: 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:14.741672+0000 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:14.741672+0000 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.740 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 1 Processor -- start 2024-09-11T03:03:14.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 1 -- start start 2024-09-11T03:03:14.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient: init 2024-09-11T03:03:14.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient: _add_conns ranks=[7,6,0,5,9,4,2,3,10,1,8] 2024-09-11T03:03:14.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient(hunting): picked mon.noname-i con 0x7f4cec139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec06fe50 0x7f4cec0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient(hunting): picked mon.noname-d con 0x7f4cec06fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4cec072640 addr v1:172.21.15.98:6789/0 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cec071370 con 0x7f4cec072640 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4cec0719c0 con 0x7f4cec06fe50 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4cec071c10 con 0x7f4cec139b80 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cf1677640 10 monclient(hunting): authenticate will time out at 1748.621884s 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec06fe50 0x7f4cec0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4ceb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4cec072640 0x7f4cec0a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:46612/0 (socket says 172.21.15.98:46612) 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec06fe50 0x7f4cec0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:34462/0 (socket says 172.21.15.98:34462) 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4ceb7fe640 1 -- 172.21.15.98:0/49236440 learned_addr learned my addr 172.21.15.98:0/49236440 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): get_auth_request con 0x7f4cec06fe50 auth_method 0 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4ceaffd640 1 --2- 172.21.15.98:0/49236440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_done global_id 84273 payload 274 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 1 -- 172.21.15.98:0/49236440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 msgr2=0x7f4cec139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 1 --2- 172.21.15.98:0/49236440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 1 -- 172.21.15.98:0/49236440 >> v1:172.21.15.98:6789/0 conn(0x7f4cec072640 legacy=0x7f4cec0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 1 monclient: found mon.noname-d 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.739+0000 7f4cea7fc640 1 -- 172.21.15.98:0/49236440 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cec0a9600 con 0x7f4cec06fe50 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ceaffd640 10 monclient: get_auth_request con 0x7f4cec139b80 auth_method 0 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ceaffd640 1 --2- 172.21.15.98:0/49236440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cea7fc640 1 --2- 172.21.15.98:0/49236440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec06fe50 0x7f4cec0a7a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009820 tx=0x7f4cd401c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4bf0dba5b40682d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 1 -- 172.21.15.98:0/49236440 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cd40b0070 con 0x7f4cec06fe50 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: dump: 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.753 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 1 -- 172.21.15.98:0/49236440 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd40aa810 con 0x7f4cec06fe50 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 5 monclient: authenticate success, global_id 84273 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:14.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 4 set_mon_vals no callback set 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 1 -- 172.21.15.98:0/49236440 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cd40a9300 con 0x7f4cec06fe50 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4ce9ffb640 10 monclient: dump: 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient: shutdown 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- 172.21.15.98:0/49236440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec06fe50 msgr2=0x7f4cec0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 --2- 172.21.15.98:0/49236440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec06fe50 0x7f4cec0a7a10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009820 tx=0x7f4cd401c3a0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- 172.21.15.98:0/49236440 shutdown_connections 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 --2- 172.21.15.98:0/49236440 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec06fe50 0x7f4cec0a7a10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 --2- 172.21.15.98:0/49236440 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- 172.21.15.98:0/49236440 >> 172.21.15.98:0/49236440 conn(0x7f4cec0ac730 msgr2=0x7f4cec0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- 172.21.15.98:0/49236440 shutdown_connections 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- 172.21.15.98:0/49236440 wait complete. 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient: monmap: 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.757 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 librados: starting msgr at 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 librados: starting objecter 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 Processor -- start 2024-09-11T03:03:14.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- start start 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 librados: setting wanted keys 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 librados: calling monclient init 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient: init 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient: _add_conns ranks=[2,6,7,0,8,3,4,1,5] 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cec06fe50 0x7f4cec13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): picked mon.noname-b con 0x7f4cec06fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec072640 0x7f4cec13ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): picked mon.noname-d con 0x7f4cec072640 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec13b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): picked mon.noname-i con 0x7f4cec139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4cec144450 con 0x7f4cec06fe50 2024-09-11T03:03:14.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4cec0a8870 con 0x7f4cec072640 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4cec1445d0 con 0x7f4cec139b80 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.743+0000 7f4cf1677640 10 monclient(hunting): authenticate will time out at 1748.627978s 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cec06fe50 0x7f4cec13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec072640 0x7f4cec13ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cec06fe50 0x7f4cec13a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:40360/0 (socket says 172.21.15.98:40360) 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 -- 172.21.15.98:0/3385088115 learned_addr learned my addr 172.21.15.98:0/3385088115 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): get_auth_request con 0x7f4cec072640 auth_method 0 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): get_auth_request con 0x7f4cec06fe50 auth_method 0 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceb7fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec13b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient(hunting): handle_auth_done global_id 64712 payload 824 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 msgr2=0x7f4cec13b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec13b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec072640 msgr2=0x7f4cec13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4cec072640 0x7f4cec13ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 monclient: found mon.noname-b 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 -- 172.21.15.98:0/3385088115 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cec141030 con 0x7f4cec06fe50 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceb7fe640 10 monclient: get_auth_request con 0x7f4cec139b80 auth_method 0 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceb7fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec139b80 0x7f4cec13b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceaffd640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cec06fe50 0x7f4cec13a7d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00c7f0 tx=0x7f4cdc00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 client.?.objecter ms_handle_connect 0x7f4cec06fe50 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:14.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cdc015070 con 0x7f4cec06fe50 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: dump: 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.763 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cdc007a90 con 0x7f4cec06fe50 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 5 monclient: authenticate success, global_id 64712 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 10 monclient: _renew_subs 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cec13cf00 con 0x7f4cec06fe50 2024-09-11T03:03:14.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cdc007c80 con 0x7f4cec06fe50 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce97fa640 4 set_mon_vals no callback set 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 10 client.64712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 monclient: dump: 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 10 monclient: _renew_subs 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cec0aa190 con 0x7f4cec06fe50 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 1 librados: init done 2024-09-11T03:03:14.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec133f20 0x7f4cec02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4cdc0113f0 con 0x7f4cec06fe50 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4cec0a5bc0 con 0x7f4cec133f20 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4ccc03df40 0x7f4ccc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(529..529 src has 1..529) ==== 6657+0+0 (secure 0 0 0) 0x7f4cdc04fdd0 con 0x7f4cec06fe50 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 client.64712.objecter ms_dispatch 0x7f4cec000ba0 osd_map(529..529 src has 1..529) 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 3 client.64712.objecter handle_osd_map got epochs [529,529] > 0 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 3 client.64712.objecter handle_osd_map decoding full epoch 529 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec133f20 0x7f4cec02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 20 client.64712.objecter dump_active .. 0 homeless 2024-09-11T03:03:14.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient: get_auth_request con 0x7f4cec133f20 auth_method 0 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceb7fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4ccc03df40 0x7f4ccc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceb7fe640 10 monclient: get_auth_request con 0x7f4ccc03df40 auth_method 0 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_done global_id 20140 payload 824 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 1 -- 172.21.15.98:0/3385088115 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cec134890 con 0x7f4cec133f20 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ceb7fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4ccc03df40 0x7f4ccc040400 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7f4cec140fe0 tx=0x7f4ce4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 client.64712.objecter ms_handle_connect 0x7f4ccc03df40 2024-09-11T03:03:14.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 client.64712.objecter ms_handle_connect 0x7f4cec133f20 2024-09-11T03:03:14.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4cea7fc640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec133f20 0x7f4cec02d450 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0704e0 tx=0x7f4cd4019140 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.747+0000 7f4ce37fe640 10 client.64712.objecter resend_mon_ops 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cd40b0040 con 0x7f4cec133f20 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 10 monclient: dump: 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4cd40036e0 con 0x7f4cec133f20 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec133f20 msgr2=0x7f4cec02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec133f20 0x7f4cec02d450 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0704e0 tx=0x7f4cd4019140 comp rx=0 tx=0).stop 2024-09-11T03:03:14.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec133f20 msgr2=0x7f4cec02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.759+0000 7f4ce37fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec133f20 0x7f4cec02d450 secure :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0704e0 tx=0x7f4cd4019140 comp rx=0 tx=0).stop 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec134300 0x7f4cec02d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4cec06ca10 con 0x7f4cec134300 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec134300 0x7f4cec02d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient: get_auth_request con 0x7f4cec134300 auth_method 0 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 10 monclient(hunting): handle_auth_done global_id 20149 payload 824 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 1 -- 172.21.15.98:0/3385088115 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4cec0a8300 con 0x7f4cec134300 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cea7fc640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec134300 0x7f4cec02d990 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4cec02eb10 tx=0x7f4cd40a8000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 10 client.64712.objecter ms_handle_connect 0x7f4cec134300 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 10 client.64712.objecter resend_mon_ops 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cd40b0040 con 0x7f4cec134300 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 10 monclient: dump: 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7f4cd40b5b90 con 0x7f4cec134300 2024-09-11T03:03:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec134300 msgr2=0x7f4cec02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec134300 0x7f4cec02d990 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f4cec02eb10 tx=0x7f4cd40a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec134300 msgr2=0x7f4cec02d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:14.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4ce37fe640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4cec134300 0x7f4cec02d990 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 96, 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 810541, 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:14.777 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.778 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:14.780 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:14.781 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:14.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:14.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:14.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:14.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:14.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:14.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.798 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:14.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:14.800 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:14.800 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 10 librados: watch_flush enter 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 10 librados: watch_flush exit 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 20 client.64712.objecter shutdown clearing up homeless session... 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.767+0000 7f4cf1677640 10 client.64712.objecter successfully canceled tick 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4ccc03df40 msgr2=0x7f4ccc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4ccc03df40 0x7f4ccc040400 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7f4cec140fe0 tx=0x7f4ce4009040 comp rx=0 tx=0).stop 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 10 monclient: shutdown 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cec06fe50 msgr2=0x7f4cec13a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 --2- 172.21.15.98:0/3385088115 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4cec06fe50 0x7f4cec13a7d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00c7f0 tx=0x7f4cdc00ccb0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4ceb7fe640 1 -- 172.21.15.98:0/3385088115 reap_dead start 2024-09-11T03:03:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4ceb7fe640 1 -- 172.21.15.98:0/3385088115 reap_dead start 2024-09-11T03:03:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 shutdown_connections 2024-09-11T03:03:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 >> 172.21.15.98:0/3385088115 conn(0x7f4cec0ac730 msgr2=0x7f4cec0aa5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 shutdown_connections 2024-09-11T03:03:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 -- 172.21.15.98:0/3385088115 wait complete. 2024-09-11T03:03:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.771+0000 7f4cf1677640 1 librados: shutdown 2024-09-11T03:03:14.803 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:03:14.803 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-11T03:03:14.804 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-11T03:03:14.804 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-11T03:03:14.804 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-11T03:03:14.804 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T03:03:14.804 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:03:14.818 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T03:03:14.818 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-09-11T03:03:14.818 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:03:14.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient: get_monmap_and_config 2024-09-11T03:03:14.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient: monmap: 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:14.978636+0000 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:14.978636+0000 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.977 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:14.978 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:14.978 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 Processor -- start 2024-09-11T03:03:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 -- start start 2024-09-11T03:03:14.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient: init 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient: _add_conns ranks=[10,6,4,0,5,2,7,3,8,9,1] 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f895406f650 0x7f8954072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): picked mon.noname-f con 0x7f895406f650 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89540a8380 0x7f895406fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): picked mon.noname-d con 0x7f89540a8380 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f895406ff70 0x7f89541397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): picked mon.noname-c con 0x7f895406ff70 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f89540a8800 con 0x7f895406f650 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8954071a80 con 0x7f895406ff70 2024-09-11T03:03:14.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8954071d60 con 0x7f89540a8380 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895bcc0640 10 monclient(hunting): authenticate will time out at 1748.858901s 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f895406ff70 0x7f89541397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89540a8380 0x7f895406fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f895406ff70 0x7f89541397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:42172/0 (socket says 172.21.15.98:42172) 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): get_auth_request con 0x7f89540a8380 auth_method 0 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 1 -- 172.21.15.98:0/4189099697 learned_addr learned my addr 172.21.15.98:0/4189099697 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:14.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 10 monclient(hunting): get_auth_request con 0x7f895406ff70 auth_method 0 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 1 --2- 172.21.15.98:0/4189099697 >> v2:172.21.15.190:3300/0 conn(0x7f895406f650 0x7f8954072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 10 monclient(hunting): get_auth_request con 0x7f895406f650 auth_method 0 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:14.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f8959a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient(hunting): handle_auth_done global_id 84291 payload 274 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 1 -- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f895406ff70 msgr2=0x7f89541397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 1 --2- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f895406ff70 0x7f89541397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 1 -- 172.21.15.98:0/4189099697 >> v2:172.21.15.190:3300/0 conn(0x7f895406f650 msgr2=0x7f8954072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 1 --2- 172.21.15.98:0/4189099697 >> v2:172.21.15.190:3300/0 conn(0x7f895406f650 0x7f8954072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient: _finish_hunting 0 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 1 monclient: found mon.noname-d 2024-09-11T03:03:14.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895925b640 1 -- 172.21.15.98:0/4189099697 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89540714c0 con 0x7f89540a8380 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.975+0000 7f895a25d640 1 --2- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f895406ff70 0x7f89541397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895925b640 1 --2- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89540a8380 0x7f895406fa30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f89500097c0 tx=0x7f895001cad0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7d7ae05aa6553443 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 1 -- 172.21.15.98:0/4189099697 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89500b0070 con 0x7f89540a8380 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: dump: 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.986 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: _finish_auth 0 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: _check_auth_tickets 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 1 -- 172.21.15.98:0/4189099697 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89500aa810 con 0x7f89540a8380 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 5 monclient: authenticate success, global_id 84291 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 1 -- 172.21.15.98:0/4189099697 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89500a9300 con 0x7f89540a8380 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 4 set_mon_vals no callback set 2024-09-11T03:03:14.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f8958a5a640 10 monclient: dump: 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:14.989 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 10 monclient: shutdown 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 -- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89540a8380 msgr2=0x7f895406fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 --2- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89540a8380 0x7f895406fa30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f89500097c0 tx=0x7f895001cad0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 -- 172.21.15.98:0/4189099697 shutdown_connections 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 --2- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f895406ff70 0x7f89541397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 --2- 172.21.15.98:0/4189099697 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89540a8380 0x7f895406fa30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 --2- 172.21.15.98:0/4189099697 >> v2:172.21.15.190:3300/0 conn(0x7f895406f650 0x7f8954072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:14.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 -- 172.21.15.98:0/4189099697 >> 172.21.15.98:0/4189099697 conn(0x7f89540ac8e0 msgr2=0x7f89540a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:14.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 -- 172.21.15.98:0/4189099697 shutdown_connections 2024-09-11T03:03:14.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 -- 172.21.15.98:0/4189099697 wait complete. 2024-09-11T03:03:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 10 monclient: build_initial_monmap 2024-09-11T03:03:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 10 monclient: monmap: 2024-09-11T03:03:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:14.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:14.997 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 librados: starting msgr at 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 librados: starting objecter 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 Processor -- start 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 -- start start 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 librados: setting wanted keys 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 1 librados: calling monclient init 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.979+0000 7f895bcc0640 10 monclient: init 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _add_conns ranks=[0,3,5,2,8,7,6,1,4] 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f895406f650 0x7f895413e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): picked mon.noname-a con 0x7f895406f650 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f895406ff70 0x7f8954122b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): picked mon.noname-g con 0x7f895406ff70 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:14.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f89540a8380 0x7f8954138950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): picked mon.noname-h con 0x7f89540a8380 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8954142fc0 con 0x7f895406f650 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f8954142e40 con 0x7f895406ff70 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8954143140 con 0x7f89540a8380 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient(hunting): authenticate will time out at 1748.864183s 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f895406f650 0x7f895413e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f895406f650 0x7f895413e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32984/0 (socket says 172.21.15.98:32984) 2024-09-11T03:03:14.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 -- 172.21.15.98:0/2558806655 learned_addr learned my addr 172.21.15.98:0/2558806655 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): get_auth_request con 0x7f895406f650 auth_method 0 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f895406ff70 0x7f8954122b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 10 monclient(hunting): get_auth_request con 0x7f895406ff70 auth_method 0 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:15.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895a25d640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f89540a8380 0x7f8954138950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient(hunting): handle_auth_done global_id 45126 payload 824 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 -- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f89540a8380 msgr2=0x7f8954138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f89540a8380 0x7f8954138950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 -- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f895406ff70 msgr2=0x7f8954122b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f895406ff70 0x7f8954122b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 monclient: found mon.noname-a 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 -- 172.21.15.98:0/2558806655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8954123a70 con 0x7f895406f650 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f8959a5c640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f895406f650 0x7f895413e020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f894400e960 tx=0x7f894400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 client.?.objecter ms_handle_connect 0x7f895406f650 2024-09-11T03:03:15.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 1 -- 172.21.15.98:0/2558806655 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8944017070 con 0x7f895406f650 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: dump: 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:15.002 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: _finish_auth 0 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:03:15.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 5 monclient: authenticate success, global_id 45126 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 1 -- 172.21.15.98:0/2558806655 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f894400ce70 con 0x7f895406f650 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _renew_subs 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 1 -- 172.21.15.98:0/2558806655 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89440138e0 con 0x7f895406f650 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8954072860 con 0x7f895406f650 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894bfff640 4 set_mon_vals no callback set 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 monclient: dump: 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:15.004 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:15.005 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 client.45126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _renew_subs 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8954124310 con 0x7f895406f650 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 librados: init done 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 1 -- 172.21.15.98:0/2558806655 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8944013470 con 0x7f895406f650 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89541397c0 con 0x7f895406f650 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f893803df40 0x7f8938040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:15.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 1 -- 172.21.15.98:0/2558806655 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 6657+0+0 (secure 0 0 0) 0x7f894400f040 con 0x7f895406f650 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 client.45126.objecter ms_dispatch 0x7f8954000ba0 osd_map(529..529 src has 1..529) 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 3 client.45126.objecter handle_osd_map got epochs [529,529] > 0 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 3 client.45126.objecter handle_osd_map decoding full epoch 529 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 20 client.45126.objecter dump_active .. 0 homeless 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f893803df40 0x7f8938040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 10 monclient: get_auth_request con 0x7f893803df40 auth_method 0 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f895925b640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f893803df40 0x7f8938040400 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7f89540ad070 tx=0x7f8950004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.983+0000 7f894a7fc640 10 client.45126.objecter ms_handle_connect 0x7f893803df40 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.987+0000 7f894a7fc640 1 -- 172.21.15.98:0/2558806655 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8944021070 con 0x7f895406f650 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.987+0000 7f894a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:15.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:14.987+0000 7f894a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:15.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.223+0000 7f895bcc0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:03:15.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.223+0000 7f895bcc0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:15.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.223+0000 7f895bcc0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:15.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.223+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8954068f80 con 0x7f895406f650 2024-09-11T03:03:15.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f894a7fc640 1 -- 172.21.15.98:0/2558806655 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f8954123a70 con 0x7f895406f650 2024-09-11T03:03:15.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f894a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:15.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f894a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:15.228 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":96,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:03:15.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 10 librados: watch_flush enter 2024-09-11T03:03:15.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 10 librados: watch_flush exit 2024-09-11T03:03:15.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 20 client.45126.objecter shutdown clearing up homeless session... 2024-09-11T03:03:15.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 10 client.45126.objecter successfully canceled tick 2024-09-11T03:03:15.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f893803df40 msgr2=0x7f8938040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:15.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f893803df40 0x7f8938040400 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7f89540ad070 tx=0x7f8950004740 comp rx=0 tx=0).stop 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 10 monclient: shutdown 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f895406f650 msgr2=0x7f895413e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f895406f650 0x7f895413e020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f894400e960 tx=0x7f894400ee20 comp rx=0 tx=0).stop 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 shutdown_connections 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f893803df40 0x7f8938040400 unknown :-1 s=CLOSED pgs=3632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f89540a8380 0x7f8954138950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f895406ff70 0x7f8954122b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 --2- 172.21.15.98:0/2558806655 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f895406f650 0x7f895413e020 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 >> 172.21.15.98:0/2558806655 conn(0x7f89540ac8e0 msgr2=0x7f8954072c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 shutdown_connections 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 -- 172.21.15.98:0/2558806655 wait complete. 2024-09-11T03:03:15.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:15.227+0000 7f895bcc0640 1 librados: shutdown 2024-09-11T03:03:18.246 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient: get_monmap_and_config 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient: build_initial_monmap 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient: monmap: 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:18.378950+0000 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:18.378950+0000 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:18.377 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:18.378 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:18.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 1 Processor -- start 2024-09-11T03:03:18.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 1 -- start start 2024-09-11T03:03:18.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient: init 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient: _add_conns ranks=[3,1,10,2,5,0,7,6,9,4,8] 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b806f650 0x7f29b8072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient(hunting): picked mon.noname-g con 0x7f29b806f650 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f29b80a8380 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f29b8070140 0x7f29b8140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient(hunting): picked mon.noname-f con 0x7f29b8070140 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29b8071a80 con 0x7f29b80a8380 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f29b80a8800 con 0x7f29b8070140 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:18.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f29b8071d60 con 0x7f29b806f650 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bf66d640 10 monclient(hunting): authenticate will time out at 1752.259242s 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b806f650 0x7f29b8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bdc0a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f29b8070140 0x7f29b8140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b806f650 0x7f29b8072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:47502/0 (socket says 172.21.15.98:47502) 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 1 -- 172.21.15.98:0/1703898282 learned_addr learned my addr 172.21.15.98:0/1703898282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 10 monclient(hunting): get_auth_request con 0x7f29b806f650 auth_method 0 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:18.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bdc0a640 10 monclient(hunting): get_auth_request con 0x7f29b8070140 auth_method 0 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bdc0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bdc0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bdc0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.375+0000 7f29bd409640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 10 monclient(hunting): handle_auth_done global_id 25912 payload 274 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 1 -- 172.21.15.98:0/1703898282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b806f650 msgr2=0x7f29b8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 1 --2- 172.21.15.98:0/1703898282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b806f650 0x7f29b8072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 1 -- 172.21.15.98:0/1703898282 >> v1:172.21.15.190:6789/0 conn(0x7f29b80a8380 legacy=0x7f29b806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:18.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 10 monclient: _finish_hunting 0 2024-09-11T03:03:18.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 1 monclient: found mon.noname-f 2024-09-11T03:03:18.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:18.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:03:18.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 1 -- 172.21.15.98:0/1703898282 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b80714c0 con 0x7f29b8070140 2024-09-11T03:03:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bdc0a640 1 --2- 172.21.15.98:0/1703898282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29b8070140 0x7f29b8140c20 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009d90 tx=0x7f29b001c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97cd0b1bcdf6f159 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 1 -- 172.21.15.98:0/1703898282 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b00b2070 con 0x7f29b8070140 2024-09-11T03:03:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:03:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: dump: 2024-09-11T03:03:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: _finish_auth 0 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 1 -- 172.21.15.98:0/1703898282 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b00a5710 con 0x7f29b8070140 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 1 -- 172.21.15.98:0/1703898282 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b00a5db0 con 0x7f29b8070140 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 5 monclient: authenticate success, global_id 25912 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 4 set_mon_vals no callback set 2024-09-11T03:03:18.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29affff640 10 monclient: dump: 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:18.393 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 10 monclient: shutdown 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:18.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 -- 172.21.15.98:0/1703898282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29b8070140 msgr2=0x7f29b8140c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 --2- 172.21.15.98:0/1703898282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29b8070140 0x7f29b8140c20 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009d90 tx=0x7f29b001c830 comp rx=0 tx=0).stop 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 -- 172.21.15.98:0/1703898282 shutdown_connections 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 --2- 172.21.15.98:0/1703898282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29b8070140 0x7f29b8140c20 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 --2- 172.21.15.98:0/1703898282 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b806f650 0x7f29b8072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 -- 172.21.15.98:0/1703898282 >> 172.21.15.98:0/1703898282 conn(0x7f29b80ac8e0 msgr2=0x7f29b80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 -- 172.21.15.98:0/1703898282 shutdown_connections 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.379+0000 7f29bf66d640 1 -- 172.21.15.98:0/1703898282 wait complete. 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: build_initial_monmap 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: monmap: 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:18.397 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:18.398 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 librados: starting msgr at 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 librados: starting objecter 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 Processor -- start 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 -- start start 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 librados: setting wanted keys 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 librados: calling monclient init 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: init 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _add_conns ranks=[6,3,2,4,8,5,0,1,7] 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f29b806f650 0x7f29b8128de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): picked mon.noname-d con 0x7f29b806f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:18.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b8070140 0x7f29b813fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): picked mon.noname-g con 0x7f29b8070140 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29b80a8380 0x7f29b8126810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): picked mon.noname-b con 0x7f29b80a8380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f29b8143d10 con 0x7f29b80a8380 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f29b8143b90 con 0x7f29b8070140 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f29b8143e90 con 0x7f29b806f650 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient(hunting): authenticate will time out at 1752.265291s 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bdc0a640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29b80a8380 msgr2=0x7f29b8126810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T03:03:18.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bdc0a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29b80a8380 0x7f29b8126810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f29b806f650 0x7f29b8128de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f29b806f650 0x7f29b8128de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49066/0 (socket says 172.21.15.98:49066) 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 -- 172.21.15.98:0/994165870 learned_addr learned my addr 172.21.15.98:0/994165870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): get_auth_request con 0x7f29b806f650 auth_method 0 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bcc08640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b8070140 0x7f29b813fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:18.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient(hunting): handle_auth_done global_id 84300 payload 824 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 -- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b8070140 msgr2=0x7f29b813fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b8070140 0x7f29b813fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 -- 172.21.15.98:0/994165870 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29b80a8380 msgr2=0x7f29b8126810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29b80a8380 0x7f29b8126810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient: _finish_hunting 0 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 monclient: found mon.noname-d 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 -- 172.21.15.98:0/994165870 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b812a5e0 con 0x7f29b806f650 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bcc08640 10 monclient: get_auth_request con 0x7f29b8070140 auth_method 0 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bcc08640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b8070140 0x7f29b813fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bd409640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f29b806f650 0x7f29b8128de0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f29b400e890 tx=0x7f29b400ed50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 client.?.objecter ms_handle_connect 0x7f29b806f650 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 1 -- 172.21.15.98:0/994165870 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b4017070 con 0x7f29b806f650 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: dump: 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:18.403 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 1 -- 172.21.15.98:0/994165870 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b400cdb0 con 0x7f29b806f650 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 5 monclient: authenticate success, global_id 84300 2024-09-11T03:03:18.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _renew_subs 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b80733a0 con 0x7f29b806f650 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 1 -- 172.21.15.98:0/994165870 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b40138e0 con 0x7f29b806f650 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 monclient: dump: 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:18.405 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29aeffd640 4 set_mon_vals no callback set 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 client.84300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:18.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _renew_subs 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b81408f0 con 0x7f29b806f650 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 librados: init done 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b8073a00 con 0x7f29b806f650 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 1 -- 172.21.15.98:0/994165870 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f29b4024060 con 0x7f29b806f650 2024-09-11T03:03:18.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299c03df40 0x7f299c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 1 -- 172.21.15.98:0/994165870 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(529..529 src has 1..529) ==== 6657+0+0 (secure 0 0 0) 0x7f29b400f040 con 0x7f29b806f650 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 10 client.84300.objecter ms_dispatch 0x7f29b8000ba0 osd_map(529..529 src has 1..529) 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 3 client.84300.objecter handle_osd_map got epochs [529,529] > 0 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 3 client.84300.objecter handle_osd_map decoding full epoch 529 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29adffb640 20 client.84300.objecter dump_active .. 0 homeless 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bcc08640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299c03df40 0x7f299c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.383+0000 7f29bcc08640 10 monclient: get_auth_request con 0x7f299c03df40 auth_method 0 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.387+0000 7f29adffb640 10 client.84300.objecter ms_handle_connect 0x7f299c03df40 2024-09-11T03:03:18.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.387+0000 7f29bcc08640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299c03df40 0x7f299c040400 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f29b8127f90 tx=0x7f29a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.387+0000 7f29adffb640 1 -- 172.21.15.98:0/994165870 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f29b8073a00 con 0x7f29b806f650 2024-09-11T03:03:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.387+0000 7f29adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:18.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.387+0000 7f29adffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:18.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.615+0000 7f29bf66d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:03:18.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.615+0000 7f29bf66d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:18.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.615+0000 7f29bf66d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:18.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.615+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f29b8128280 con 0x7f29b806f650 2024-09-11T03:03:18.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.615+0000 7f29adffb640 1 -- 172.21.15.98:0/994165870 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f29b401a360 con 0x7f29b806f650 2024-09-11T03:03:18.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.615+0000 7f29adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:18.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.615+0000 7f29adffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:18.618 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":96,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:03:18.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 10 librados: watch_flush enter 2024-09-11T03:03:18.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 10 librados: watch_flush exit 2024-09-11T03:03:18.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 20 client.84300.objecter shutdown clearing up homeless session... 2024-09-11T03:03:18.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 10 client.84300.objecter successfully canceled tick 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299c03df40 msgr2=0x7f299c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299c03df40 0x7f299c040400 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f29b8127f90 tx=0x7f29a8009040 comp rx=0 tx=0).stop 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 10 monclient: shutdown 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f29b806f650 msgr2=0x7f29b8128de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f29b806f650 0x7f29b8128de0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f29b400e890 tx=0x7f29b400ed50 comp rx=0 tx=0).stop 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 shutdown_connections 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f299c03df40 0x7f299c040400 unknown :-1 s=CLOSED pgs=3633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f29b80a8380 0x7f29b8126810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f29b8070140 0x7f29b813fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 --2- 172.21.15.98:0/994165870 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f29b806f650 0x7f29b8128de0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:18.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 >> 172.21.15.98:0/994165870 conn(0x7f29b80ac8e0 msgr2=0x7f29b80a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:18.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 shutdown_connections 2024-09-11T03:03:18.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 -- 172.21.15.98:0/994165870 wait complete. 2024-09-11T03:03:18.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:18.619+0000 7f29bf66d640 1 librados: shutdown 2024-09-11T03:03:21.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:03:21.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient: get_monmap_and_config 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient: build_initial_monmap 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient: monmap: 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:21.766864+0000 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:21.766864+0000 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:21.765 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:21.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 Processor -- start 2024-09-11T03:03:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 -- start start 2024-09-11T03:03:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient: init 2024-09-11T03:03:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:21.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient: _add_conns ranks=[4,2,10,1,0,5,6,3,7,9,8] 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd32806f650 0x7fd328072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): picked mon.noname-c con 0x7fd32806f650 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd32806fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): picked mon.noname-b con 0x7fd3280a8380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd32806ff70 0x7fd3281397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): picked mon.noname-f con 0x7fd32806ff70 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd3280a8800 con 0x7fd32806ff70 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd328071a80 con 0x7fd3280a8380 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd328071d60 con 0x7fd32806f650 2024-09-11T03:03:21.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd3300c5640 10 monclient(hunting): authenticate will time out at 1755.647159s 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32d660640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 msgr2=0x7fd32806fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32d660640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd32806fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd32806f650 0x7fd328072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd32806f650 0x7fd328072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:56310/0 (socket says 172.21.15.98:56310) 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 -- 172.21.15.98:0/1995726685 learned_addr learned my addr 172.21.15.98:0/1995726685 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): get_auth_request con 0x7fd32806f650 auth_method 0 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:21.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32e662640 1 --2- 172.21.15.98:0/1995726685 >> v2:172.21.15.190:3300/0 conn(0x7fd32806ff70 0x7fd3281397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32e662640 10 monclient(hunting): get_auth_request con 0x7fd32806ff70 auth_method 0 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32e662640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32e662640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32e662640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient(hunting): handle_auth_done global_id 24556 payload 274 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 -- 172.21.15.98:0/1995726685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 msgr2=0x7fd32806fa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 --2- 172.21.15.98:0/1995726685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd32806fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 -- 172.21.15.98:0/1995726685 >> v2:172.21.15.190:3300/0 conn(0x7fd32806ff70 msgr2=0x7fd3281397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 --2- 172.21.15.98:0/1995726685 >> v2:172.21.15.190:3300/0 conn(0x7fd32806ff70 0x7fd3281397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient: _finish_hunting 0 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 monclient: found mon.noname-c 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:21.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32de61640 1 -- 172.21.15.98:0/1995726685 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3280714c0 con 0x7fd32806f650 2024-09-11T03:03:21.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.763+0000 7fd32e662640 1 --2- 172.21.15.98:0/1995726685 >> v2:172.21.15.190:3300/0 conn(0x7fd32806ff70 0x7fd3281397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:03:21.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32de61640 1 --2- 172.21.15.98:0/1995726685 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd32806f650 0x7fd328072640 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fd318009870 tx=0x7fd31801cc70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ab6cc531b65c11d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:21.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 1 -- 172.21.15.98:0/1995726685 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3180b0070 con 0x7fd32806f650 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: dump: 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: _finish_auth 0 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: _check_auth_tickets 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 1 -- 172.21.15.98:0/1995726685 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3180aa810 con 0x7fd32806f650 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 1 -- 172.21.15.98:0/1995726685 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3180a9300 con 0x7fd32806f650 2024-09-11T03:03:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 5 monclient: authenticate success, global_id 24556 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 4 set_mon_vals no callback set 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd32ce5f640 10 monclient: dump: 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 10 monclient: shutdown 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 -- 172.21.15.98:0/1995726685 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd32806f650 msgr2=0x7fd328072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 --2- 172.21.15.98:0/1995726685 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd32806f650 0x7fd328072640 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fd318009870 tx=0x7fd31801cc70 comp rx=0 tx=0).stop 2024-09-11T03:03:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 -- 172.21.15.98:0/1995726685 shutdown_connections 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 --2- 172.21.15.98:0/1995726685 >> v2:172.21.15.190:3300/0 conn(0x7fd32806ff70 0x7fd3281397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 --2- 172.21.15.98:0/1995726685 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd32806fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 --2- 172.21.15.98:0/1995726685 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd32806f650 0x7fd328072640 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 -- 172.21.15.98:0/1995726685 >> 172.21.15.98:0/1995726685 conn(0x7fd3280ac8e0 msgr2=0x7fd3280a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 -- 172.21.15.98:0/1995726685 shutdown_connections 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.767+0000 7fd3300c5640 1 -- 172.21.15.98:0/1995726685 wait complete. 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: build_initial_monmap 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: monmap: 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:21.784 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:21.784 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 librados: starting msgr at 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 librados: starting objecter 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 Processor -- start 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 -- start start 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 librados: setting wanted keys 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 librados: calling monclient init 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: init 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _add_conns ranks=[6,7,2,4,8,3,1,0,5] 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd32806f650 0x7fd3281d1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): picked mon.noname-d con 0x7fd32806f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd32806ff70 0x7fd3281386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:21.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): picked mon.noname-i con 0x7fd32806ff70 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd328138c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): picked mon.noname-b con 0x7fd3280a8380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd328142fc0 con 0x7fd3280a8380 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd328142e40 con 0x7fd32806f650 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fd328143140 con 0x7fd32806ff70 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient(hunting): authenticate will time out at 1755.653115s 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32e662640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 msgr2=0x7fd328138c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32e662640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd328138c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:21.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd32806f650 0x7fd3281d1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd32806f650 0x7fd3281d1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49076/0 (socket says 172.21.15.98:49076) 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 -- 172.21.15.98:0/2741471979 learned_addr learned my addr 172.21.15.98:0/2741471979 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): get_auth_request con 0x7fd32806f650 auth_method 0 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32d660640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd32806ff70 0x7fd3281386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient(hunting): handle_auth_done global_id 84309 payload 824 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 -- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd32806ff70 msgr2=0x7fd3281386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:21.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd32806ff70 0x7fd3281386f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 -- 172.21.15.98:0/2741471979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 msgr2=0x7fd328138c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd328138c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient: _finish_hunting 0 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 monclient: found mon.noname-d 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 -- 172.21.15.98:0/2741471979 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd328123bd0 con 0x7fd32806f650 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32de61640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd32806f650 0x7fd3281d1160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd31801c040 tx=0x7fd318018fb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 client.?.objecter ms_handle_connect 0x7fd32806f650 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 1 -- 172.21.15.98:0/2741471979 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3180b0040 con 0x7fd32806f650 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:21.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: dump: 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:21.789 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: _finish_auth 0 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 5 monclient: authenticate success, global_id 84309 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 1 -- 172.21.15.98:0/2741471979 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3180044b0 con 0x7fd32806f650 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 1 -- 172.21.15.98:0/2741471979 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3180aade0 con 0x7fd32806f650 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _renew_subs 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:21.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd328123eb0 con 0x7fd32806f650 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31ffff640 4 set_mon_vals no callback set 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 monclient: dump: 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:21.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 client.84309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _renew_subs 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd32813f520 con 0x7fd32806f650 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 librados: init done 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:21.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 1 -- 172.21.15.98:0/2741471979 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd3180b3070 con 0x7fd32806f650 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3281397c0 con 0x7fd32806f650 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd30c03df40 0x7fd30c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 1 -- 172.21.15.98:0/2741471979 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(529..529 src has 1..529) ==== 6657+0+0 (secure 0 0 0) 0x7fd3180e9180 con 0x7fd32806f650 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 10 client.84309.objecter ms_dispatch 0x7fd328000ba0 osd_map(529..529 src has 1..529) 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 3 client.84309.objecter handle_osd_map got epochs [529,529] > 0 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 3 client.84309.objecter handle_osd_map decoding full epoch 529 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd31e7fc640 20 client.84309.objecter dump_active .. 0 homeless 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32d660640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd30c03df40 0x7fd30c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.771+0000 7fd32d660640 10 monclient: get_auth_request con 0x7fd30c03df40 auth_method 0 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.775+0000 7fd31e7fc640 10 client.84309.objecter ms_handle_connect 0x7fd30c03df40 2024-09-11T03:03:21.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.775+0000 7fd32d660640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd30c03df40 0x7fd30c040400 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7fd32813f230 tx=0x7fd324008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:21.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.775+0000 7fd31e7fc640 1 -- 172.21.15.98:0/2741471979 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fd318016150 con 0x7fd32806f650 2024-09-11T03:03:21.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.775+0000 7fd31e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:21.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:21.775+0000 7fd31e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.011+0000 7fd3300c5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:03:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.011+0000 7fd3300c5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.011+0000 7fd3300c5640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:22.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.011+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd328122990 con 0x7fd32806f650 2024-09-11T03:03:22.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.011+0000 7fd31e7fc640 1 -- 172.21.15.98:0/2741471979 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fd3180aa7e0 con 0x7fd32806f650 2024-09-11T03:03:22.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.011+0000 7fd31e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:22.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.011+0000 7fd31e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:22.014 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":96,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 10 librados: watch_flush enter 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 10 librados: watch_flush exit 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 20 client.84309.objecter shutdown clearing up homeless session... 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 10 client.84309.objecter successfully canceled tick 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd30c03df40 msgr2=0x7fd30c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd30c03df40 0x7fd30c040400 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7fd32813f230 tx=0x7fd324008040 comp rx=0 tx=0).stop 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 10 monclient: shutdown 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd32806f650 msgr2=0x7fd3281d1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:22.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd32806f650 0x7fd3281d1160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd31801c040 tx=0x7fd318018fb0 comp rx=0 tx=0).stop 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 shutdown_connections 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd30c03df40 0x7fd30c040400 unknown :-1 s=CLOSED pgs=3634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd3280a8380 0x7fd328138c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fd32806ff70 0x7fd3281386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 --2- 172.21.15.98:0/2741471979 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd32806f650 0x7fd3281d1160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 >> 172.21.15.98:0/2741471979 conn(0x7fd3280ac8e0 msgr2=0x7fd3280a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 shutdown_connections 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 -- 172.21.15.98:0/2741471979 wait complete. 2024-09-11T03:03:22.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:22.015+0000 7fd3300c5640 1 librados: shutdown 2024-09-11T03:03:25.035 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:03:25.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient: get_monmap_and_config 2024-09-11T03:03:25.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient: build_initial_monmap 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient: monmap: 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:25.166762+0000 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:25.166762+0000 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:25.165 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:25.166 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:25.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 1 Processor -- start 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 1 -- start start 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient: init 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient: _add_conns ranks=[4,6,0,5,2,1,8,3,9,7,10] 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient(hunting): picked mon.noname-c con 0x7fd56c139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient(hunting): picked mon.noname-d con 0x7fd56c06fe30 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd56c0a7a10 addr v1:172.21.15.98:6789/0 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd56c0a9650 con 0x7fd56c0a7a10 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:25.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fd56c071350 con 0x7fd56c139b10 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd56c071630 con 0x7fd56c06fe30 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd571203640 10 monclient(hunting): authenticate will time out at 1759.046995s 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56b59e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd56c0a7a10 0x7fd56c0a7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52582/0 (socket says 172.21.15.98:52582) 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 learned_addr learned my addr 172.21.15.98:0/1968873072 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request con 0x7fd56c139b10 auth_method 0 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:25.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd56c06fe30 auth_method 0 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:25.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.163+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request con 0x7fd56c139b10 auth_method 0 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:25.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd56c06fe30 auth_method 0 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:25.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.367+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:03:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request con 0x7fd56c139b10 auth_method 0 2024-09-11T03:03:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:25.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd56c06fe30 auth_method 0 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:25.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:25.771+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd56c06fe30 auth_method 0 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request con 0x7fd56c139b10 auth_method 0 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:03:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:26.575+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:03:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): tick 2024-09-11T03:03:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T03:03:28.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 monclient(hunting): continuing hunt 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 msgr2=0x7fd56c139ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd56c06fe30 0x7fd56c139ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 msgr2=0x7fd56c0725d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fd56c139b10 0x7fd56c0725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 -- 172.21.15.98:0/1968873072 >> v1:172.21.15.98:6789/0 conn(0x7fd56c0a7a10 legacy=0x7fd56c0a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient: _add_conns ranks=[1,8,3,10,2,5,9,7] 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd54c003850 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): picked mon.noname-e con 0x7fd54c007460 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:28.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): picked mon.noname-g con 0x7fd54c009f80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 -- 172.21.15.98:0/1968873072 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd56c0702b0 con 0x7fd54c003850 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 -- 172.21.15.98:0/1968873072 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd56c071350 con 0x7fd54c009f80 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 1 -- 172.21.15.98:0/1968873072 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd56c071630 con 0x7fd54c007460 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd568d99640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd54c007460 auth_method 0 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:28.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56b59e640 10 monclient(hunting): get_auth_request con 0x7fd54c009f80 auth_method 0 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.162+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.166+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.166+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.166+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.166+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.166+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.166+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:28.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.166+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd54c007460 auth_method 0 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:03:28.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 10 monclient(hunting): get_auth_request con 0x7fd54c009f80 auth_method 0 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:28.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.370+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:03:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd54c007460 auth_method 0 2024-09-11T03:03:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:28.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 10 monclient(hunting): get_auth_request con 0x7fd54c009f80 auth_method 0 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:28.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:03:28.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:28.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:28.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:28.774+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:03:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd54c007460 auth_method 0 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:03:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 10 monclient(hunting): get_auth_request con 0x7fd54c009f80 auth_method 0 2024-09-11T03:03:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.578+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:03:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3328074905 0 0) 0x7fd56c0702b0 con 0x7fd54c003850 2024-09-11T03:03:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): _negotiate 2024-09-11T03:03:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): my global_id is 25930 2024-09-11T03:03:29.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd558003530 con 0x7fd54c003850 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3502119824 0 0) 0x7fd558003530 con 0x7fd54c003850 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 msgr2=0x7fd54c0098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd54c007460 0x7fd54c0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 msgr2=0x7fd54c00c440 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd54c009f80 0x7fd54c00c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 monclient: found mon.noname-f-legacy 2024-09-11T03:03:29.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 reap_dead start 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.190:6789/0 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54c00dcd0 con 0x7fd54c003850 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: _finish_auth 0 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: _check_auth_tickets 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fd56400f040 con 0x7fd54c003850 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd571203640 5 monclient: authenticate success, global_id 25930 2024-09-11T03:03:29.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd571203640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-09-11T03:03:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: dump: 2024-09-11T03:03:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:29.796 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:29.797 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] but i'm connected to v1:172.21.15.190:6789/0, reconnecting 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 >> v1:172.21.15.190:6789/0 conn(0x7fd54c003850 legacy=0x7fd54c005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient: _add_conns ranks=[0,8,1,7,6,2,3,5,4] 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd558004170 0x7fd558004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): picked mon.a con 0x7fd558004170 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd558005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): picked mon.e con 0x7fd558004da0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558009b60 0x7fd55800bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): picked mon.f con 0x7fd558009b60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:29.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd56c071350 con 0x7fd558004170 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd56c071630 con 0x7fd558009b60 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd55800c900 con 0x7fd558004da0 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd569d9b640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.786+0000 7fd571203640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd558005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 10 monclient(hunting): get_auth_request con 0x7fd558004da0 auth_method 0 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd558004170 0x7fd558004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558009b60 0x7fd55800bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): get_auth_request con 0x7fd558009b60 auth_method 0 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 msgr2=0x7fd558005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 msgr2=0x7fd558005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd558005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd558005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:29.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request con 0x7fd558004170 auth_method 0 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient(hunting): handle_auth_done global_id 25930 payload 293 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 msgr2=0x7fd558005200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd558005200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd558004170 msgr2=0x7fd558004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:29.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd558004170 0x7fd558004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient: _finish_hunting 0 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 1 monclient: found mon.f 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 1 -- 172.21.15.98:0/1968873072 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54c00dcd0 con 0x7fd558009b60 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient: _finish_auth 0 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 10 monclient: _check_auth_tickets 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56b59e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:29.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.790+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd558004170 0x7fd558004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558009b60 0x7fd55800bff0 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0a7df0 tx=0x7fd55c01cb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee8a97475f7d17fd server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd55c0b2070 con 0x7fd558009b60 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 10 monclient: dump: 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.808 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:29.809 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55c0a59f0 con 0x7fd558009b60 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 1 -- 172.21.15.98:0/1968873072 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd55c0049d0 con 0x7fd558009b60 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd569d9b640 10 monclient: dump: 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:29.810 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd571203640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.794+0000 7fd571203640 4 set_mon_vals no callback set 2024-09-11T03:03:29.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient: shutdown 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558009b60 msgr2=0x7fd55800bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558009b60 0x7fd55800bff0 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fd56c0a7df0 tx=0x7fd55c01cb40 comp rx=0 tx=0).stop 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- 172.21.15.98:0/1968873072 shutdown_connections 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558009b60 0x7fd55800bff0 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd558005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 --2- 172.21.15.98:0/1968873072 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd558004170 0x7fd558004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- 172.21.15.98:0/1968873072 >> 172.21.15.98:0/1968873072 conn(0x7fd56c0ac740 msgr2=0x7fd56c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- 172.21.15.98:0/1968873072 shutdown_connections 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- 172.21.15.98:0/1968873072 wait complete. 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient: build_initial_monmap 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient: monmap: 2024-09-11T03:03:29.812 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:29.813 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 librados: starting msgr at 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 librados: starting objecter 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 Processor -- start 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- start start 2024-09-11T03:03:29.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 librados: setting wanted keys 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 librados: calling monclient init 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient: init 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient: _add_conns ranks=[5,1,8,3,4,7,2,6,0] 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd54c003850 0x7fd56c138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): picked mon.noname-h con 0x7fd54c003850 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558004170 0x7fd56c139150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): picked mon.noname-f con 0x7fd558004170 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd56c1228a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): picked mon.noname-e con 0x7fd558004da0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:29.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd56c0a81b0 con 0x7fd558004170 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd56c0a8030 con 0x7fd54c003850 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fd56c0a8330 con 0x7fd558004da0 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd571203640 10 monclient(hunting): authenticate will time out at 1763.683496s 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd56c1228a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd56c1228a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44898/0 (socket says 172.21.15.98:44898) 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 1 -- 172.21.15.98:0/1423500107 learned_addr learned my addr 172.21.15.98:0/1423500107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:29.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 10 monclient(hunting): get_auth_request con 0x7fd558004da0 auth_method 0 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd54c003850 0x7fd56c138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request con 0x7fd54c003850 auth_method 0 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:29.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.798+0000 7fd56ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 10 monclient(hunting): handle_auth_done global_id 25091 payload 824 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56a59c640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558004170 0x7fd56c139150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 1 -- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd54c003850 msgr2=0x7fd56c138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd54c003850 0x7fd56c138c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 1 -- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558004170 msgr2=0x7fd56c139150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558004170 0x7fd56c139150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 10 monclient: _finish_hunting 0 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 1 monclient: found mon.noname-e 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:29.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 1 -- 172.21.15.98:0/1423500107 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56c125490 con 0x7fd558004da0 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56b59e640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd56c1228a0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fd55c01d020 tx=0x7fd55c019980 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 client.?.objecter ms_handle_connect 0x7fd558004da0 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 1 -- 172.21.15.98:0/1423500107 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd55c0b2040 con 0x7fd558004da0 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: dump: 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:29.819 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:29.820 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 1 -- 172.21.15.98:0/1423500107 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55c0bd070 con 0x7fd558004da0 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 1 -- 172.21.15.98:0/1423500107 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd55c0b88c0 con 0x7fd558004da0 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 5 monclient: authenticate success, global_id 25091 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 monclient: _renew_subs 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56959a640 4 set_mon_vals no callback set 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56c0738e0 con 0x7fd558004da0 2024-09-11T03:03:29.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 monclient: dump: 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:29.822 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 client.25091.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 monclient: _renew_subs 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd56c1265d0 con 0x7fd558004da0 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 1 librados: init done 2024-09-11T03:03:29.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 1 -- 172.21.15.98:0/1423500107 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd55c0b8450 con 0x7fd558004da0 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd56c0a6200 con 0x7fd558004da0 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd54c03f1c0 0x7fd54c041680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 1 -- 172.21.15.98:0/1423500107 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(530..530 src has 1..530) ==== 6657+0+0 (secure 0 0 0) 0x7fd55c0e9ea0 con 0x7fd558004da0 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 client.25091.objecter ms_dispatch 0x7fd56c000ba0 osd_map(530..530 src has 1..530) 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 3 client.25091.objecter handle_osd_map got epochs [530,530] > 0 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 3 client.25091.objecter handle_osd_map decoding full epoch 530 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 20 client.25091.objecter dump_active .. 0 homeless 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd54c03f1c0 0x7fd54c041680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:29.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56ad9d640 10 monclient: get_auth_request con 0x7fd54c03f1c0 auth_method 0 2024-09-11T03:03:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd5637fe640 10 client.25091.objecter ms_handle_connect 0x7fd54c03f1c0 2024-09-11T03:03:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.802+0000 7fd56ad9d640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd54c03f1c0 0x7fd54c041680 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7fd564000eb0 tx=0x7fd564010040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:29.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.806+0000 7fd5637fe640 1 -- 172.21.15.98:0/1423500107 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fd56c0a6200 con 0x7fd558004da0 2024-09-11T03:03:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.806+0000 7fd5637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:29.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:29.806+0000 7fd5637fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:30.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:03:30.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd56c0a86f0 con 0x7fd558004da0 2024-09-11T03:03:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd5637fe640 1 -- 172.21.15.98:0/1423500107 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3187 (secure 0 0 0) 0x7fd55c016960 con 0x7fd558004da0 2024-09-11T03:03:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd5637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:03:30.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd5637fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:30.045 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 10 librados: watch_flush enter 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 10 librados: watch_flush exit 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 20 client.25091.objecter shutdown clearing up homeless session... 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 10 client.25091.objecter successfully canceled tick 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd54c03f1c0 msgr2=0x7fd54c041680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd54c03f1c0 0x7fd54c041680 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7fd564000eb0 tx=0x7fd564010040 comp rx=0 tx=0).stop 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 10 monclient: shutdown 2024-09-11T03:03:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 msgr2=0x7fd56c1228a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd56c1228a0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fd55c01d020 tx=0x7fd55c019980 comp rx=0 tx=0).stop 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 shutdown_connections 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd54c03f1c0 0x7fd54c041680 unknown :-1 s=CLOSED pgs=3635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fd558004da0 0x7fd56c1228a0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd558004170 0x7fd56c139150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 --2- 172.21.15.98:0/1423500107 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd54c003850 0x7fd56c138c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 >> 172.21.15.98:0/1423500107 conn(0x7fd56c0ac740 msgr2=0x7fd56c1c1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 shutdown_connections 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.042+0000 7fd571203640 1 -- 172.21.15.98:0/1423500107 wait complete. 2024-09-11T03:03:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.046+0000 7fd571203640 1 librados: shutdown 2024-09-11T03:03:30.063 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-09-11T03:03:30.063 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-09-11T03:03:30.063 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:03:30.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient: get_monmap_and_config 2024-09-11T03:03:30.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient: monmap: 2024-09-11T03:03:30.172 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.172 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:30.174441+0000 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:30.174441+0000 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:30.174 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:30.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 Processor -- start 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 -- start start 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient: init 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient: _add_conns ranks=[7,6,3,0,4,2,5,9,10,1,8] 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b80a3de0 0x7f60b80a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): picked mon.noname-i con 0x7f60b80a3de0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b809b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): picked mon.noname-d con 0x7f60b80a4700 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b809b5f0 0x7f60b809ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): picked mon.noname-g con 0x7f60b809b5f0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:30.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f60b80a11a0 con 0x7f60b809b5f0 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f60b80a1d20 con 0x7f60b80a4700 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f60b80a2000 con 0x7f60b80a3de0 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60bc847640 10 monclient(hunting): authenticate will time out at 1764.055434s 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b809b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b809b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46558/0 (socket says 172.21.15.98:46558) 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 1 -- 172.21.15.98:0/1033663592 learned_addr learned my addr 172.21.15.98:0/1033663592 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): get_auth_request con 0x7f60b80a4700 auth_method 0 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b6d9d640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b809b5f0 0x7f60b809ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b659c640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b80a3de0 0x7f60b80a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b659c640 10 monclient(hunting): get_auth_request con 0x7f60b80a3de0 auth_method 0 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b6d9d640 10 monclient(hunting): get_auth_request con 0x7f60b809b5f0 auth_method 0 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 10 monclient(hunting): handle_auth_done global_id 84318 payload 274 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 1 -- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b80a3de0 msgr2=0x7f60b80a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.170+0000 7f60affff640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b80a3de0 0x7f60b80a41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 1 -- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b809b5f0 msgr2=0x7f60b809ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b809b5f0 0x7f60b809ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 1 monclient: found mon.noname-d 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:30.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 1 -- 172.21.15.98:0/1033663592 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b80a22e0 con 0x7f60b80a4700 2024-09-11T03:03:30.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60affff640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b809b0b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f60a00098a0 tx=0x7f60a001c4d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e13f162d64992930 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 1 -- 172.21.15.98:0/1033663592 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60a00b0070 con 0x7f60b80a4700 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: dump: 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.190 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 1 -- 172.21.15.98:0/1033663592 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a00aa810 con 0x7f60b80a4700 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 1 -- 172.21.15.98:0/1033663592 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60a00a9300 con 0x7f60b80a4700 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 5 monclient: authenticate success, global_id 84318 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 4 set_mon_vals no callback set 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60b5d9b640 10 monclient: dump: 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.192 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.193 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 10 monclient: shutdown 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 -- 172.21.15.98:0/1033663592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 msgr2=0x7f60b809b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b809b0b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f60a00098a0 tx=0x7f60a001c4d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 -- 172.21.15.98:0/1033663592 shutdown_connections 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60b809b5f0 0x7f60b809ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b809b0b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 --2- 172.21.15.98:0/1033663592 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b80a3de0 0x7f60b80a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 -- 172.21.15.98:0/1033663592 >> 172.21.15.98:0/1033663592 conn(0x7f60b80a4ed0 msgr2=0x7f60b80a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 -- 172.21.15.98:0/1033663592 shutdown_connections 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.174+0000 7f60bc847640 1 -- 172.21.15.98:0/1033663592 wait complete. 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: monmap: 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.195 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 librados: starting msgr at 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 librados: starting objecter 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 Processor -- start 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 -- start start 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 librados: setting wanted keys 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 librados: calling monclient init 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: init 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: _add_conns ranks=[7,8,6,1,4,3,0,2,5] 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b809b5f0 0x7f60b812d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): picked mon.noname-i con 0x7f60b809b5f0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60b80a3de0 0x7f60b812d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): picked mon.noname-e con 0x7f60b80a3de0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b812df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): picked mon.noname-d con 0x7f60b80a4700 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f60b80a5d40 con 0x7f60b80a4700 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f60b80a5bc0 con 0x7f60b809b5f0 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f60b80a2700 con 0x7f60b80a3de0 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): authenticate will time out at 1764.061040s 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b812df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60b80a3de0 0x7f60b812d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b812df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46572/0 (socket says 172.21.15.98:46572) 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 1 -- 172.21.15.98:0/2020360002 learned_addr learned my addr 172.21.15.98:0/2020360002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): get_auth_request con 0x7f60b80a3de0 auth_method 0 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient(hunting): get_auth_request con 0x7f60b80a4700 auth_method 0 2024-09-11T03:03:30.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b809b5f0 0x7f60b812d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): get_auth_request con 0x7f60b809b5f0 auth_method 0 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient(hunting): handle_auth_done global_id 25100 payload 824 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b809b5f0 msgr2=0x7f60b812d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f60b809b5f0 0x7f60b812d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 msgr2=0x7f60b812df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b812df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 monclient: found mon.noname-e 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 -- 172.21.15.98:0/2020360002 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b812f7d0 con 0x7f60b80a3de0 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f60b80a4700 0x7f60b812df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:30.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60affff640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60b80a3de0 0x7f60b812d9d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009870 tx=0x7f60a0019270 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 client.?.objecter ms_handle_connect 0x7f60b80a3de0 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60a00ba020 con 0x7f60b80a3de0 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: dump: 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.201 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 5 monclient: authenticate success, global_id 25100 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a00044b0 con 0x7f60b80a3de0 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: _renew_subs 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b80a3640 con 0x7f60b80a3de0 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60a00b0040 con 0x7f60b80a3de0 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 client.25100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: dump: 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.204 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b559a640 4 set_mon_vals no callback set 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: _renew_subs 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b813c2a0 con 0x7f60b80a3de0 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 librados: init done 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b812e870 0x7f60b809a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60b806ca10 con 0x7f60b812e870 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b812e870 0x7f60b809a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient: get_auth_request con 0x7f60b812e870 auth_method 0 2024-09-11T03:03:30.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f60a00b3070 con 0x7f60b80a3de0 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 10 monclient(hunting): handle_auth_done global_id 45135 payload 824 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 1 -- 172.21.15.98:0/2020360002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60b80a41c0 con 0x7f60b812e870 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b659c640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b812e870 0x7f60b809a0a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f60a800f4b0 tx=0x7f60a800f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f609803df30 0x7f60980403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 client.25100.objecter ms_handle_connect 0x7f60b812e870 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 client.25100.objecter resend_mon_ops 2024-09-11T03:03:30.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(530..530 src has 1..530) ==== 6657+0+0 (secure 0 0 0) 0x7f60a00e95e0 con 0x7f60b80a3de0 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 client.25100.objecter ms_dispatch 0x7f60b8000ba0 osd_map(530..530 src has 1..530) 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 3 client.25100.objecter handle_osd_map got epochs [530,530] > 0 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 3 client.25100.objecter handle_osd_map decoding full epoch 530 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 20 client.25100.objecter dump_active .. 0 homeless 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60a8013a80 con 0x7f60b812e870 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: dump: 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.207 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60a801a070 con 0x7f60b812e870 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b812e870 msgr2=0x7f60b809a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b812e870 0x7f60b809a0a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f60a800f4b0 tx=0x7f60a800f970 comp rx=0 tx=0).stop 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b812e870 msgr2=0x7f60b809a0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60aeffd640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b812e870 0x7f60b809a0a0 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f60a800f4b0 tx=0x7f60a800f970 comp rx=0 tx=0).stop 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f609803df30 0x7f60980403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.178+0000 7f60b6d9d640 10 monclient: get_auth_request con 0x7f609803df30 auth_method 0 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.182+0000 7f60aeffd640 10 client.25100.objecter ms_handle_connect 0x7f609803df30 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.182+0000 7f60b6d9d640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f609803df30 0x7f60980403f0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7f60b00094c0 tx=0x7f60b00098b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b809a5e0 0x7f60b809a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f60b809af30 con 0x7f60b809a5e0 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b809a5e0 0x7f60b809a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient: get_auth_request con 0x7f60b809a5e0 auth_method 0 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 10 monclient(hunting): handle_auth_done global_id 45144 payload 824 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 1 -- 172.21.15.98:0/2020360002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f60b806ce80 con 0x7f60b809a5e0 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 10 client.25100.objecter ms_handle_connect 0x7f60b809a5e0 2024-09-11T03:03:30.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b659c640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b809a5e0 0x7f60b809a9f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f60a800ff10 tx=0x7f60a80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 10 client.25100.objecter resend_mon_ops 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60a8013c70 con 0x7f60b809a5e0 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 10 monclient: dump: 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.211 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9424 (secure 0 0 0) 0x7f60a8013800 con 0x7f60b809a5e0 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b809a5e0 msgr2=0x7f60b809a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b809a5e0 0x7f60b809a9f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f60a800ff10 tx=0x7f60a80046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b809a5e0 msgr2=0x7f60b809a9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60aeffd640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60b809a5e0 0x7f60b809a9f0 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f60a800ff10 tx=0x7f60a80046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.213 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:30.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 436626, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:30.214 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.215 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.216 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:30.217 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.218 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.219 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:30.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:30.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.224 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.225 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.226 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.227 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.228 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.229 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.230 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:30.231 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.232 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.233 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:30.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.236 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.237 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:30.238 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 10 librados: watch_flush enter 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 10 librados: watch_flush exit 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 20 client.25100.objecter shutdown clearing up homeless session... 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 10 client.25100.objecter successfully canceled tick 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f609803df30 msgr2=0x7f60980403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f609803df30 0x7f60980403f0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7f60b00094c0 tx=0x7f60b00098b0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 10 monclient: shutdown 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60b80a3de0 msgr2=0x7f60b812d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60b6d9d640 1 -- 172.21.15.98:0/2020360002 reap_dead start 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60b80a3de0 0x7f60b812d9d0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009870 tx=0x7f60a0019270 comp rx=0 tx=0).stop 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 shutdown_connections 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 --2- 172.21.15.98:0/2020360002 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60b80a3de0 0x7f60b812d9d0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 >> 172.21.15.98:0/2020360002 conn(0x7f60b80a4ed0 msgr2=0x7f60b813caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 shutdown_connections 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 -- 172.21.15.98:0/2020360002 wait complete. 2024-09-11T03:03:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.186+0000 7f60bc847640 1 librados: shutdown 2024-09-11T03:03:30.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:30.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-09-11T03:03:30.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:03:30.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.358+0000 7f5d93a2f640 10 monclient: get_monmap_and_config 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.358+0000 7f5d93a2f640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.358+0000 7f5d93a2f640 10 monclient: monmap: 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:30.363870+0000 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:30.363870+0000 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.362 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.363 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.363 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.363 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.363 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.363 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:30.363 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:30.363 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.358+0000 7f5d93a2f640 1 Processor -- start 2024-09-11T03:03:30.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.358+0000 7f5d93a2f640 1 -- start start 2024-09-11T03:03:30.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.358+0000 7f5d93a2f640 10 monclient: init 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient: _add_conns ranks=[9,6,10,5,8,1,7,4,2,3,0] 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d8c139b10 0x7f5d8c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): picked mon.noname-a con 0x7f5d8c139b10 addr v2:172.21.15.98:3300/0 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d8c139ef0 0x7f5d8c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): picked mon.noname-d con 0x7f5d8c139ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5d8c13a2d0 0x7f5d8c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): picked mon.noname-f con 0x7f5d8c13a2d0 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5d8c0702b0 con 0x7f5d8c139b10 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5d8c071bf0 con 0x7f5d8c13a2d0 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5d8c071350 con 0x7f5d8c139ef0 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient(hunting): authenticate will time out at 1764.244236s 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5d8c139b10 0x7f5d8c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d8c139ef0 0x7f5d8c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d8c139ef0 0x7f5d8c0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46574/0 (socket says 172.21.15.98:46574) 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 -- 172.21.15.98:0/4271843595 learned_addr learned my addr 172.21.15.98:0/4271843595 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): get_auth_request con 0x7f5d8c139ef0 auth_method 0 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 10 monclient(hunting): get_auth_request con 0x7f5d8c139b10 auth_method 0 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d91fcc640 1 --2- 172.21.15.98:0/4271843595 >> v2:172.21.15.190:3300/0 conn(0x7f5d8c13a2d0 0x7f5d8c0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d91fcc640 10 monclient(hunting): get_auth_request con 0x7f5d8c13a2d0 auth_method 0 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d91fcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d91fcc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d91fcc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient(hunting): handle_auth_done global_id 84336 payload 274 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 -- 172.21.15.98:0/4271843595 >> v2:172.21.15.190:3300/0 conn(0x7f5d8c13a2d0 msgr2=0x7f5d8c0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 --2- 172.21.15.98:0/4271843595 >> v2:172.21.15.190:3300/0 conn(0x7f5d8c13a2d0 0x7f5d8c0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 -- 172.21.15.98:0/4271843595 >> v2:172.21.15.98:3300/0 conn(0x7f5d8c139b10 msgr2=0x7f5d8c072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 --2- 172.21.15.98:0/4271843595 >> v2:172.21.15.98:3300/0 conn(0x7f5d8c139b10 0x7f5d8c072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 monclient: found mon.noname-d 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 -- 172.21.15.98:0/4271843595 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8c0a9620 con 0x7f5d8c139ef0 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d90fca640 1 --2- 172.21.15.98:0/4271843595 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d8c139ef0 0x7f5d8c0a7a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c0a9e90 tx=0x7f5d7c01c470 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a0610af204a9b1a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 1 -- 172.21.15.98:0/4271843595 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d7c0b0070 con 0x7f5d8c139ef0 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: dump: 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.372 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 1 -- 172.21.15.98:0/4271843595 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7c0aa810 con 0x7f5d8c139ef0 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 5 monclient: authenticate success, global_id 84336 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 1 -- 172.21.15.98:0/4271843595 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d7c0a9300 con 0x7f5d8c139ef0 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 4 set_mon_vals no callback set 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d83fff640 10 monclient: dump: 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.374 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 10 monclient: shutdown 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- 172.21.15.98:0/4271843595 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d8c139ef0 msgr2=0x7f5d8c0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/4271843595 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d8c139ef0 0x7f5d8c0a7a10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c0a9e90 tx=0x7f5d7c01c470 comp rx=0 tx=0).stop 2024-09-11T03:03:30.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- 172.21.15.98:0/4271843595 shutdown_connections 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/4271843595 >> v2:172.21.15.190:3300/0 conn(0x7f5d8c13a2d0 0x7f5d8c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/4271843595 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5d8c139ef0 0x7f5d8c0a7a10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/4271843595 >> v2:172.21.15.98:3300/0 conn(0x7f5d8c139b10 0x7f5d8c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- 172.21.15.98:0/4271843595 >> 172.21.15.98:0/4271843595 conn(0x7f5d8c0ac790 msgr2=0x7f5d8c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- 172.21.15.98:0/4271843595 shutdown_connections 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.362+0000 7f5d93a2f640 1 -- 172.21.15.98:0/4271843595 wait complete. 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: monmap: 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:30.376 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 librados: starting msgr at 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 librados: starting objecter 2024-09-11T03:03:30.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 Processor -- start 2024-09-11T03:03:30.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 -- start start 2024-09-11T03:03:30.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 librados: setting wanted keys 2024-09-11T03:03:30.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 librados: calling monclient init 2024-09-11T03:03:30.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: init 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: _add_conns ranks=[1,8,5,4,2,3,0,6,7] 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d8c139b10 0x7f5d8c134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): picked mon.noname-f con 0x7f5d8c139b10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d8c139ef0 0x7f5d8c134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): picked mon.noname-e con 0x7f5d8c139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5d8c13a2d0 0x7f5d8c13a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): picked mon.noname-h con 0x7f5d8c13a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8c143460 con 0x7f5d8c139b10 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5d8c0a88a0 con 0x7f5d8c13a2d0 2024-09-11T03:03:30.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5d8c1435e0 con 0x7f5d8c139ef0 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): authenticate will time out at 1764.249227s 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d8c139ef0 0x7f5d8c134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d8c139ef0 0x7f5d8c134f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44924/0 (socket says 172.21.15.98:44924) 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 -- 172.21.15.98:0/1931661907 learned_addr learned my addr 172.21.15.98:0/1931661907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): get_auth_request con 0x7f5d8c139ef0 auth_method 0 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d8c139b10 0x7f5d8c134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d91fcc640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5d8c13a2d0 0x7f5d8c13a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient(hunting): handle_auth_done global_id 25109 payload 824 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5d8c13a2d0 msgr2=0x7f5d8c13a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5d8c13a2d0 0x7f5d8c13a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d8c139b10 msgr2=0x7f5d8c134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d8c139b10 0x7f5d8c134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 monclient: found mon.noname-e 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 -- 172.21.15.98:0/1931661907 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8c0aa2c0 con 0x7f5d8c139ef0 2024-09-11T03:03:30.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient: get_auth_request con 0x7f5d8c139b10 auth_method 0 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d8c139b10 0x7f5d8c134a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d90fca640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d8c139ef0 0x7f5d8c134f40 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c01ab20 tx=0x7f5d7c019210 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 client.?.objecter ms_handle_connect 0x7f5d8c139ef0 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d7c0b0040 con 0x7f5d8c139ef0 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: dump: 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.388 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7c0044a0 con 0x7f5d8c139ef0 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d7c003a90 con 0x7f5d8c139ef0 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 5 monclient: authenticate success, global_id 25109 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: _renew_subs 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d8c13dee0 con 0x7f5d8c139ef0 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d837fe640 4 set_mon_vals no callback set 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 client.25109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:30.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: dump: 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.391 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: _renew_subs 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d8c135880 con 0x7f5d8c139ef0 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 librados: init done 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13acc0 0x7f5d8c13b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d8c074a50 con 0x7f5d8c13acc0 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13acc0 0x7f5d8c13b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient: get_auth_request con 0x7f5d8c13acc0 auth_method 0 2024-09-11T03:03:30.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5d7c003c30 con 0x7f5d8c139ef0 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 10 monclient(hunting): handle_auth_done global_id 24565 payload 824 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 1 -- 172.21.15.98:0/1931661907 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d8c06e680 con 0x7f5d8c13acc0 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d7003df70 0x7f5d70040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(530..530 src has 1..530) ==== 6657+0+0 (secure 0 0 0) 0x7f5d7c0e8ef0 con 0x7f5d8c139ef0 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 client.25109.objecter ms_dispatch 0x7f5d8c000ba0 osd_map(530..530 src has 1..530) 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 3 client.25109.objecter handle_osd_map got epochs [530,530] > 0 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 3 client.25109.objecter handle_osd_map decoding full epoch 530 2024-09-11T03:03:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d917cb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13acc0 0x7f5d8c13b0a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5d88010820 tx=0x7f5d88010ce0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 20 client.25109.objecter dump_active .. 0 homeless 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 client.25109.objecter ms_handle_connect 0x7f5d8c13acc0 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 client.25109.objecter resend_mon_ops 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d8800f040 con 0x7f5d8c13acc0 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d81ffb640 10 monclient: dump: 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d91fcc640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d7003df70 0x7f5d70040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.366+0000 7f5d91fcc640 10 monclient: get_auth_request con 0x7f5d7003df70 auth_method 0 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d8801c400 con 0x7f5d8c13acc0 2024-09-11T03:03:30.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13acc0 msgr2=0x7f5d8c13b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13acc0 0x7f5d8c13b0a0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5d88010820 tx=0x7f5d88010ce0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13acc0 msgr2=0x7f5d8c13b0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13acc0 0x7f5d8c13b0a0 secure :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f5d88010820 tx=0x7f5d88010ce0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d81ffb640 10 client.25109.objecter ms_handle_connect 0x7f5d7003df70 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.370+0000 7f5d91fcc640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d7003df70 0x7f5d70040430 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009d90 tx=0x7f5d8400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13b5e0 0x7f5d8c0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f5d8c13c9c0 con 0x7f5d8c13b5e0 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13b5e0 0x7f5d8c0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient: get_auth_request con 0x7f5d8c13b5e0 auth_method 0 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 10 monclient(hunting): handle_auth_done global_id 24574 payload 824 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 1 -- 172.21.15.98:0/1931661907 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d8c0a9e00 con 0x7f5d8c13b5e0 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 10 client.25109.objecter ms_handle_connect 0x7f5d8c13b5e0 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d917cb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13b5e0 0x7f5d8c0a0e60 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5d88010820 tx=0x7f5d880046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 10 client.25109.objecter resend_mon_ops 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d88005e10 con 0x7f5d8c13b5e0 2024-09-11T03:03:30.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 10 monclient: dump: 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.398 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9424 (secure 0 0 0) 0x7f5d880059a0 con 0x7f5d8c13b5e0 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13b5e0 msgr2=0x7f5d8c0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13b5e0 0x7f5d8c0a0e60 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5d88010820 tx=0x7f5d880046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13b5e0 msgr2=0x7f5d8c0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d81ffb640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f5d8c13b5e0 0x7f5d8c0a0e60 secure :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f5d88010820 tx=0x7f5d880046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 826159, 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:30.400 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.401 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.402 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:30.403 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.404 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:30.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.409 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:30.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.414 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:30.416 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.419 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.420 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:30.421 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:30.423 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 10 librados: watch_flush enter 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 10 librados: watch_flush exit 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 20 client.25109.objecter shutdown clearing up homeless session... 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 10 client.25109.objecter successfully canceled tick 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d7003df70 msgr2=0x7f5d70040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5d7003df70 0x7f5d70040430 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7f5d84009d90 tx=0x7f5d8400b040 comp rx=0 tx=0).stop 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 10 monclient: shutdown 2024-09-11T03:03:30.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d8c139ef0 msgr2=0x7f5d8c134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d91fcc640 1 -- 172.21.15.98:0/1931661907 reap_dead start 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d8c139ef0 0x7f5d8c134f40 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c01ab20 tx=0x7f5d7c019210 comp rx=0 tx=0).stop 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 shutdown_connections 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 --2- 172.21.15.98:0/1931661907 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5d8c139ef0 0x7f5d8c134f40 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 >> 172.21.15.98:0/1931661907 conn(0x7f5d8c0ac790 msgr2=0x7f5d8c0aa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 shutdown_connections 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 -- 172.21.15.98:0/1931661907 wait complete. 2024-09-11T03:03:30.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.374+0000 7f5d93a2f640 1 librados: shutdown 2024-09-11T03:03:30.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:30.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-09-11T03:03:30.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient: get_monmap_and_config 2024-09-11T03:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient: monmap: 2024-09-11T03:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:30.546216+0000 2024-09-11T03:03:30.544 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:30.546216+0000 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:30.545 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:30.546 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 1 Processor -- start 2024-09-11T03:03:30.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 1 -- start start 2024-09-11T03:03:30.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient: init 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient: _add_conns ranks=[1,6,5,10,7,4,2,9,8,0,3] 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa9a80a52a0 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8073c20 0x7fa9a809d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient(hunting): picked mon.noname-d con 0x7fa9a8073c20 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a813d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient(hunting): picked mon.noname-h con 0x7fa9a813b480 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9a80a5680 con 0x7fa9a80a52a0 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa9a809dbf0 con 0x7fa9a813b480 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa9a809f600 con 0x7fa9a8073c20 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9af169640 10 monclient(hunting): authenticate will time out at 1764.427043s 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8073c20 0x7fa9a809d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8073c20 0x7fa9a809d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46590/0 (socket says 172.21.15.98:46590) 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 -- 172.21.15.98:0/1010154792 learned_addr learned my addr 172.21.15.98:0/1010154792 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): get_auth_request con 0x7fa9a8073c20 auth_method 0 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9ad706640 1 --2- 172.21.15.98:0/1010154792 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a813d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9ad706640 10 monclient(hunting): get_auth_request con 0x7fa9a813b480 auth_method 0 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9ad706640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9ad706640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa9ad706640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient(hunting): handle_auth_done global_id 84345 payload 274 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 -- 172.21.15.98:0/1010154792 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 msgr2=0x7fa9a813d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 --2- 172.21.15.98:0/1010154792 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a813d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 -- 172.21.15.98:0/1010154792 >> v1:172.21.15.190:6789/0 conn(0x7fa9a80a52a0 legacy=0x7fa9a809ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 monclient: found mon.noname-d 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:30.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.542+0000 7fa99ffff640 1 -- 172.21.15.98:0/1010154792 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a80a41a0 con 0x7fa9a8073c20 2024-09-11T03:03:30.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99ffff640 1 --2- 172.21.15.98:0/1010154792 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8073c20 0x7fa9a809d160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa99800aa40 tx=0x7fa99801d7e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=20f8d093daa86a41 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 1 -- 172.21.15.98:0/1010154792 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9980b3070 con 0x7fa9a8073c20 2024-09-11T03:03:30.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: dump: 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.559 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 1 -- 172.21.15.98:0/1010154792 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9980a6a40 con 0x7fa9a8073c20 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 5 monclient: authenticate success, global_id 84345 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:30.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 4 set_mon_vals no callback set 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 1 -- 172.21.15.98:0/1010154792 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa99800b040 con 0x7fa9a8073c20 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa99f7fe640 10 monclient: dump: 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.561 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 10 monclient: shutdown 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 -- 172.21.15.98:0/1010154792 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8073c20 msgr2=0x7fa9a809d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 --2- 172.21.15.98:0/1010154792 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8073c20 0x7fa9a809d160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa99800aa40 tx=0x7fa99801d7e0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 -- 172.21.15.98:0/1010154792 shutdown_connections 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 --2- 172.21.15.98:0/1010154792 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8073c20 0x7fa9a809d160 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 --2- 172.21.15.98:0/1010154792 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a813d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 -- 172.21.15.98:0/1010154792 >> 172.21.15.98:0/1010154792 conn(0x7fa9a80a4780 msgr2=0x7fa9a80a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 -- 172.21.15.98:0/1010154792 shutdown_connections 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 -- 172.21.15.98:0/1010154792 wait complete. 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 10 monclient: monmap: 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.563 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 librados: starting msgr at 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 librados: starting objecter 2024-09-11T03:03:30.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 Processor -- start 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 -- start start 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 librados: setting wanted keys 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 1 librados: calling monclient init 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.546+0000 7fa9af169640 10 monclient: init 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient: _add_conns ranks=[4,0,5,8,6,1,3,2,7] 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9a8073c20 0x7fa9a80725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): picked mon.noname-c con 0x7fa9a8073c20 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a80a52a0 0x7fa9a81368f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): picked mon.noname-a con 0x7fa9a80a52a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a81d1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): picked mon.noname-h con 0x7fa9a813b480 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:30.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a809d820 con 0x7fa9a80a52a0 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa9a809d6a0 con 0x7fa9a8073c20 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa9a813fdc0 con 0x7fa9a813b480 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): authenticate will time out at 1764.432348s 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9a8073c20 0x7fa9a80725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a80a52a0 0x7fa9a81368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9a8073c20 0x7fa9a80725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45712/0 (socket says 172.21.15.98:45712) 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 -- 172.21.15.98:0/3360395583 learned_addr learned my addr 172.21.15.98:0/3360395583 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): get_auth_request con 0x7fa9a8073c20 auth_method 0 2024-09-11T03:03:30.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): get_auth_request con 0x7fa9a80a52a0 auth_method 0 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9ad706640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a81d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient(hunting): handle_auth_done global_id 24583 payload 824 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 msgr2=0x7fa9a81d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a81d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a80a52a0 msgr2=0x7fa9a81368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a80a52a0 0x7fa9a81368f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 monclient: found mon.noname-c 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 -- 172.21.15.98:0/3360395583 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a8124fe0 con 0x7fa9a8073c20 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa9a80a52a0 0x7fa9a81368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:30.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9ad706640 10 monclient: get_auth_request con 0x7fa9a813b480 auth_method 0 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9ad706640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa9a813b480 0x7fa9a81d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9acf05640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9a8073c20 0x7fa9a80725e0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400c500 tx=0x7fa9a400c9c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 client.?.objecter ms_handle_connect 0x7fa9a8073c20 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9a4017070 con 0x7fa9a8073c20 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: dump: 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.569 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a4003600 con 0x7fa9a8073c20 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 5 monclient: authenticate success, global_id 24583 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient: _renew_subs 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a8138690 con 0x7fa9a8073c20 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9a4003ca0 con 0x7fa9a8073c20 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 client.24583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: dump: 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.571 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.572 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99effd640 4 set_mon_vals no callback set 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient: _renew_subs 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a8124d00 con 0x7fa9a8073c20 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 librados: init done 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8136e30 0x7fa9a802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa9a4003830 con 0x7fa9a8073c20 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa9a806ca10 con 0x7fa9a8136e30 2024-09-11T03:03:30.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8136e30 0x7fa9a802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient: get_auth_request con 0x7fa9a8136e30 auth_method 0 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa99403df40 0x7fa994040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(530..530 src has 1..530) ==== 6657+0+0 (secure 0 0 0) 0x7fa9a404eb60 con 0x7fa9a8073c20 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 client.24583.objecter ms_dispatch 0x7fa9a8000ba0 osd_map(530..530 src has 1..530) 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 3 client.24583.objecter handle_osd_map got epochs [530,530] > 0 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 3 client.24583.objecter handle_osd_map decoding full epoch 530 2024-09-11T03:03:30.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 10 monclient(hunting): handle_auth_done global_id 84354 payload 824 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 1 -- 172.21.15.98:0/3360395583 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9a8124780 con 0x7fa9a8136e30 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 20 client.24583.objecter dump_active .. 0 homeless 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 client.24583.objecter ms_handle_connect 0x7fa9a8136e30 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8136e30 0x7fa9a802d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b0000 tx=0x7fa99801a580 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 client.24583.objecter resend_mon_ops 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9980bb020 con 0x7fa9a8136e30 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: dump: 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.575 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9ad706640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa99403df40 0x7fa994040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9ad706640 10 monclient: get_auth_request con 0x7fa99403df40 auth_method 0 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa9980a6980 con 0x7fa9a8136e30 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8136e30 msgr2=0x7fa9a802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8136e30 0x7fa9a802d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b0000 tx=0x7fa99801a580 comp rx=0 tx=0).stop 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8136e30 msgr2=0x7fa9a802d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a8136e30 0x7fa9a802d450 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa99d7fa640 10 client.24583.objecter ms_handle_connect 0x7fa99403df40 2024-09-11T03:03:30.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.550+0000 7fa9ad706640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa99403df40 0x7fa994040400 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80a5df0 tx=0x7fa9a000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a802d990 0x7fa9a802dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa9a806e870 con 0x7fa9a802d990 2024-09-11T03:03:30.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a802d990 0x7fa9a802dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient: get_auth_request con 0x7fa9a802d990 auth_method 0 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 10 monclient(hunting): handle_auth_done global_id 84363 payload 824 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 1 -- 172.21.15.98:0/3360395583 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9a8068b40 con 0x7fa9a802d990 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 10 client.24583.objecter ms_handle_connect 0x7fa9a802d990 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 10 client.24583.objecter resend_mon_ops 2024-09-11T03:03:30.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99ffff640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a802d990 0x7fa9a802dda0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b1040 tx=0x7fa9980a8000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa9980bb040 con 0x7fa9a802d990 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 10 monclient: dump: 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.581 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9256 (secure 0 0 0) 0x7fa9980c0420 con 0x7fa9a802d990 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a802d990 msgr2=0x7fa9a802dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a802d990 0x7fa9a802dda0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa9980b1040 tx=0x7fa9980a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:30.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a802d990 msgr2=0x7fa9a802dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa99d7fa640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa9a802d990 0x7fa9a802dda0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 36371, 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:30.583 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:30.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.588 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.589 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.590 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.592 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.593 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.594 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:30.595 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.596 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.597 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:30.598 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.599 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.601 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.602 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.604 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:30.606 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 10 librados: watch_flush enter 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 10 librados: watch_flush exit 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 20 client.24583.objecter shutdown clearing up homeless session... 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 10 client.24583.objecter successfully canceled tick 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa99403df40 msgr2=0x7fa994040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa99403df40 0x7fa994040400 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80a5df0 tx=0x7fa9a000b040 comp rx=0 tx=0).stop 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 10 monclient: shutdown 2024-09-11T03:03:30.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9a8073c20 msgr2=0x7fa9a80725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 --2- 172.21.15.98:0/3360395583 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa9a8073c20 0x7fa9a80725e0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400c500 tx=0x7fa9a400c9c0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9ad706640 1 -- 172.21.15.98:0/3360395583 reap_dead start 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9ad706640 1 -- 172.21.15.98:0/3360395583 reap_dead start 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 shutdown_connections 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 >> 172.21.15.98:0/3360395583 conn(0x7fa9a80a4780 msgr2=0x7fa9a809fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 shutdown_connections 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 -- 172.21.15.98:0/3360395583 wait complete. 2024-09-11T03:03:30.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.558+0000 7fa9af169640 1 librados: shutdown 2024-09-11T03:03:30.609 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:30.610 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-09-11T03:03:30.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:03:30.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.722+0000 7f4e0c531640 10 monclient: get_monmap_and_config 2024-09-11T03:03:30.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.722+0000 7f4e0c531640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.722+0000 7f4e0c531640 10 monclient: monmap: 2024-09-11T03:03:30.727 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.727 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.727 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:30.729072+0000 2024-09-11T03:03:30.727 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:30.729072+0000 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:30.728 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:30.729 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 1 Processor -- start 2024-09-11T03:03:30.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 1 -- start start 2024-09-11T03:03:30.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient: init 2024-09-11T03:03:30.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient: _add_conns ranks=[6,9,0,5,2,8,10,3,4,1,7] 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e04139b10 0x7f4e04072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient(hunting): picked mon.noname-d con 0x7f4e04139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4e04139ef0 0x7f4e040a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient(hunting): picked mon.noname-a con 0x7f4e04139ef0 addr v2:172.21.15.98:3300/0 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e0413a2d0 addr v1:172.21.15.98:6789/0 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e040702b0 con 0x7f4e0413a2d0 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f4e04071bf0 con 0x7f4e04139ef0 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f4e04071350 con 0x7f4e04139b10 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0c531640 10 monclient(hunting): authenticate will time out at 1764.609346s 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e04139b10 0x7f4e04072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4e04139ef0 0x7f4e040a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f4e04139ef0 0x7f4e040a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46384/0 (socket says 172.21.15.98:46384) 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 -- 172.21.15.98:0/1740210473 learned_addr learned my addr 172.21.15.98:0/1740210473 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): get_auth_request con 0x7f4e04139ef0 auth_method 0 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 1 --2- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e04139b10 0x7f4e04072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46614/0 (socket says 172.21.15.98:46614) 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 10 monclient(hunting): get_auth_request con 0x7f4e04139b10 auth_method 0 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient(hunting): handle_auth_done global_id 45171 payload 274 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 -- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e04139b10 msgr2=0x7f4e04072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 --2- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e04139b10 0x7f4e04072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 -- 172.21.15.98:0/1740210473 >> v1:172.21.15.98:6789/0 conn(0x7f4e0413a2d0 legacy=0x7f4e040a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 monclient: found mon.noname-a 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e09acc640 1 -- 172.21.15.98:0/1740210473 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e040a9620 con 0x7f4e04139ef0 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.726+0000 7f4e0a2cd640 1 --2- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e04139b10 0x7f4e04072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e09acc640 1 --2- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e04139ef0 0x7f4e040a7a10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009870 tx=0x7f4df401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c3eda9ab416eab server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 1 -- 172.21.15.98:0/1740210473 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4df40b0070 con 0x7f4e04139ef0 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: dump: 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.742 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 1 -- 172.21.15.98:0/1740210473 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df40aa810 con 0x7f4e04139ef0 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 5 monclient: authenticate success, global_id 45171 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 1 -- 172.21.15.98:0/1740210473 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4df40a9300 con 0x7f4e04139ef0 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 4 set_mon_vals no callback set 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e092cb640 10 monclient: dump: 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.744 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.745 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: shutdown 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e04139ef0 msgr2=0x7f4e040a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e04139ef0 0x7f4e040a7a10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4df4009870 tx=0x7f4df401c4b0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- 172.21.15.98:0/1740210473 shutdown_connections 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e04139ef0 0x7f4e040a7a10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1740210473 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f4e04139b10 0x7f4e04072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- 172.21.15.98:0/1740210473 >> 172.21.15.98:0/1740210473 conn(0x7f4e040ac790 msgr2=0x7f4e040a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- 172.21.15.98:0/1740210473 shutdown_connections 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- 172.21.15.98:0/1740210473 wait complete. 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: monmap: 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 librados: starting msgr at 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 librados: starting objecter 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 Processor -- start 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- start start 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 librados: setting wanted keys 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 librados: calling monclient init 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: init 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: _add_conns ranks=[5,2,4,6,0,1,7,3,8] 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4e04139b10 0x7f4e0413a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): picked mon.noname-h con 0x7f4e04139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e04139ef0 0x7f4e0413aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): picked mon.noname-b con 0x7f4e04139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e0413a2d0 0x7f4e0413b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0413a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f4e04144470 con 0x7f4e04139ef0 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4e040a8870 con 0x7f4e0413a2d0 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f4e041445f0 con 0x7f4e04139b10 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient(hunting): authenticate will time out at 1764.614520s 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e09acc640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e04139ef0 msgr2=0x7f4e0413aca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e09acc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e04139ef0 0x7f4e0413aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e0413a2d0 0x7f4e0413b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e0413a2d0 0x7f4e0413b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45724/0 (socket says 172.21.15.98:45724) 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 -- 172.21.15.98:0/1346741822 learned_addr learned my addr 172.21.15.98:0/1346741822 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): get_auth_request con 0x7f4e0413a2d0 auth_method 0 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0a2cd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4e04139b10 0x7f4e0413a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0a2cd640 10 monclient(hunting): get_auth_request con 0x7f4e04139b10 auth_method 0 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0a2cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient(hunting): handle_auth_done global_id 24592 payload 824 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4e04139b10 msgr2=0x7f4e0413a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4e04139b10 0x7f4e0413a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e04139ef0 msgr2=0x7f4e0413aca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f4e04139ef0 0x7f4e0413aca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 monclient: found mon.noname-c 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 -- 172.21.15.98:0/1346741822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0413d8d0 con 0x7f4e0413a2d0 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0aace640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e0413a2d0 0x7f4e0413b1e0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00c710 tx=0x7f4dfc00cbd0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 client.?.objecter ms_handle_connect 0x7f4e0413a2d0 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4dfc0017e0 con 0x7f4e0413a2d0 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: dump: 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dfc00ce20 con 0x7f4e0413a2d0 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 5 monclient: authenticate success, global_id 24592 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: _renew_subs 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0413cc10 con 0x7f4e0413a2d0 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4dfc00acc0 con 0x7f4e0413a2d0 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e08aca640 4 set_mon_vals no callback set 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4dfaffd640 10 monclient: dump: 2024-09-11T03:03:30.754 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.755 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 client.24592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: _renew_subs 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e040abc20 con 0x7f4e0413a2d0 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.730+0000 7f4e0c531640 1 librados: init done 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0c531640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1500 0x7f4e041d18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0c531640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4e0406ca10 con 0x7f4e041d1500 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1500 0x7f4e041d18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4dfc00a850 con 0x7f4e0413a2d0 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient: get_auth_request con 0x7f4e041d1500 auth_method 0 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4de803df40 0x7f4de8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_done global_id 25118 payload 824 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 1 -- 172.21.15.98:0/1346741822 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e0406e680 con 0x7f4e041d1500 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(530..530 src has 1..530) ==== 6657+0+0 (secure 0 0 0) 0x7f4dfc04f700 con 0x7f4e0413a2d0 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 client.24592.objecter ms_dispatch 0x7f4e04000ba0 osd_map(530..530 src has 1..530) 2024-09-11T03:03:30.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 3 client.24592.objecter handle_osd_map got epochs [530,530] > 0 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 3 client.24592.objecter handle_osd_map decoding full epoch 530 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 20 client.24592.objecter dump_active .. 0 homeless 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 client.24592.objecter ms_handle_connect 0x7f4e041d1500 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 client.24592.objecter resend_mon_ops 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e0a2cd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1500 0x7f4e041d18e0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f4e00011600 tx=0x7f4e00011ac0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e09acc640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4de803df40 0x7f4de8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e0000fbb0 con 0x7f4e041d1500 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 monclient: dump: 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.758 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.759 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e09acc640 10 monclient: get_auth_request con 0x7f4de803df40 auth_method 0 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4e0000f740 con 0x7f4e041d1500 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1500 msgr2=0x7f4e041d18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1500 0x7f4e041d18e0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f4e00011600 tx=0x7f4e00011ac0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1500 msgr2=0x7f4e041d18e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1500 0x7f4e041d18e0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4dfaffd640 10 client.24592.objecter ms_handle_connect 0x7f4de803df40 2024-09-11T03:03:30.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.734+0000 7f4e09acc640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4de803df40 0x7f4de8040400 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7f4df40ad000 tx=0x7f4df4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0c531640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:30.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1e20 0x7f4e040a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0c531640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4e0406e870 con 0x7f4e041d1e20 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1e20 0x7f4e040a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 10 monclient: get_auth_request con 0x7f4e041d1e20 auth_method 0 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.738+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0a2cd640 10 monclient(hunting): handle_auth_done global_id 25127 payload 824 2024-09-11T03:03:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0a2cd640 1 -- 172.21.15.98:0/1346741822 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4e04072600 con 0x7f4e041d1e20 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0a2cd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1e20 0x7f4e040a0e60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4e00011600 tx=0x7f4e000046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 10 client.24592.objecter ms_handle_connect 0x7f4e041d1e20 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 10 client.24592.objecter resend_mon_ops 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4e00006570 con 0x7f4e041d1e20 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 10 monclient: dump: 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9257 (secure 0 0 0) 0x7f4e00006100 con 0x7f4e041d1e20 2024-09-11T03:03:30.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1e20 msgr2=0x7f4e040a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1e20 0x7f4e040a0e60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4e00011600 tx=0x7f4e000046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1e20 msgr2=0x7f4e040a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4dfaffd640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4e041d1e20 0x7f4e040a0e60 secure :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4e00011600 tx=0x7f4e000046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:30.766 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 826535, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:03:30.767 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.768 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.769 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.771 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:30.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.773 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.774 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.775 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.776 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.777 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.779 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.780 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.781 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.782 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:30.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.784 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.786 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:30.787 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.788 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:30.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:30.790 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 10 librados: watch_flush enter 2024-09-11T03:03:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 10 librados: watch_flush exit 2024-09-11T03:03:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 20 client.24592.objecter shutdown clearing up homeless session... 2024-09-11T03:03:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 10 client.24592.objecter successfully canceled tick 2024-09-11T03:03:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4de803df40 msgr2=0x7f4de8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4de803df40 0x7f4de8040400 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7f4df40ad000 tx=0x7f4df4008040 comp rx=0 tx=0).stop 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 10 monclient: shutdown 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e0413a2d0 msgr2=0x7f4e0413b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0aace640 1 -- 172.21.15.98:0/1346741822 reap_dead start 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e0413a2d0 0x7f4e0413b1e0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc00c710 tx=0x7f4dfc00cbd0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 shutdown_connections 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 --2- 172.21.15.98:0/1346741822 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4e0413a2d0 0x7f4e0413b1e0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 >> 172.21.15.98:0/1346741822 conn(0x7f4e040ac790 msgr2=0x7f4e040ab290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 shutdown_connections 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 -- 172.21.15.98:0/1346741822 wait complete. 2024-09-11T03:03:30.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.742+0000 7f4e0c531640 1 librados: shutdown 2024-09-11T03:03:30.794 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:30.794 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-09-11T03:03:30.794 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:03:30.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient: get_monmap_and_config 2024-09-11T03:03:30.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient: monmap: 2024-09-11T03:03:30.911 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.911 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:30.913404+0000 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:30.913404+0000 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.912 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:30.913 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:30.913 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 Processor -- start 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 -- start start 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient: init 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient: _add_conns ranks=[10,6,4,8,1,7,9,5,2,0,3] 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa7c80a3de0 0x7fa7c80a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): picked mon.noname-f con 0x7fa7c80a3de0 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c809b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): picked mon.noname-d con 0x7fa7c80a4700 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7c809b5f0 0x7fa7c809ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): picked mon.noname-c con 0x7fa7c809b5f0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa7c809c3d0 con 0x7fa7c80a3de0 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fa7c80a11a0 con 0x7fa7c809b5f0 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa7c80a1d20 con 0x7fa7c80a4700 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7cd2d4640 10 monclient(hunting): authenticate will time out at 1764.793586s 2024-09-11T03:03:30.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c809b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7c809b5f0 0x7fa7c809ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7c809b5f0 0x7fa7c809ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45728/0 (socket says 172.21.15.98:45728) 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 -- 172.21.15.98:0/4052760349 learned_addr learned my addr 172.21.15.98:0/4052760349 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c809b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46618/0 (socket says 172.21.15.98:46618) 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): get_auth_request con 0x7fa7c809b5f0 auth_method 0 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 10 monclient(hunting): get_auth_request con 0x7fa7c80a4700 auth_method 0 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c6ffd640 1 --2- 172.21.15.98:0/4052760349 >> v2:172.21.15.190:3300/0 conn(0x7fa7c80a3de0 0x7fa7c80a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c6ffd640 10 monclient(hunting): get_auth_request con 0x7fa7c80a3de0 auth_method 0 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient(hunting): handle_auth_done global_id 24601 payload 274 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 -- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 msgr2=0x7fa7c809b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 --2- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c809b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 -- 172.21.15.98:0/4052760349 >> v2:172.21.15.190:3300/0 conn(0x7fa7c80a3de0 msgr2=0x7fa7c80a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 --2- 172.21.15.98:0/4052760349 >> v2:172.21.15.190:3300/0 conn(0x7fa7c80a3de0 0x7fa7c80a41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 monclient: found mon.noname-c 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c77fe640 1 -- 172.21.15.98:0/4052760349 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c80a22e0 con 0x7fa7c809b5f0 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c67fc640 1 --2- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c809b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.910+0000 7fa7c6ffd640 1 --2- 172.21.15.98:0/4052760349 >> v2:172.21.15.190:3300/0 conn(0x7fa7c80a3de0 0x7fa7c80a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:03:30.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 1 --2- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7c809b5f0 0x7fa7c809ba50 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009e80 tx=0x7fa7c001c8d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d0a38e4801195f50 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 1 -- 172.21.15.98:0/4052760349 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c00090d0 con 0x7fa7c809b5f0 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: dump: 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.928 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 5 monclient: authenticate success, global_id 24601 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 1 -- 172.21.15.98:0/4052760349 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c00a5990 con 0x7fa7c809b5f0 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 4 set_mon_vals no callback set 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 1 -- 172.21.15.98:0/4052760349 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7c00ad3f0 con 0x7fa7c809b5f0 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c5ffb640 10 monclient: dump: 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.930 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.931 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: shutdown 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7c809b5f0 msgr2=0x7fa7c809ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7c809b5f0 0x7fa7c809ba50 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fa7c0009e80 tx=0x7fa7c001c8d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/4052760349 shutdown_connections 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fa7c809b5f0 0x7fa7c809ba50 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/4052760349 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c809b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/4052760349 >> v2:172.21.15.190:3300/0 conn(0x7fa7c80a3de0 0x7fa7c80a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/4052760349 >> 172.21.15.98:0/4052760349 conn(0x7fa7c80a4ed0 msgr2=0x7fa7c80a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/4052760349 shutdown_connections 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/4052760349 wait complete. 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: build_initial_monmap 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: monmap: 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:30.933 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 librados: starting msgr at 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 librados: starting objecter 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 Processor -- start 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- start start 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 librados: setting wanted keys 2024-09-11T03:03:30.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 librados: calling monclient init 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: init 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: _add_conns ranks=[5,0,6,1,8,3,4,7,2] 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa7c809b5f0 0x7fa7c812dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): picked mon.noname-h con 0x7fa7c809b5f0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c80a3de0 0x7fa7c813a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): picked mon.noname-a con 0x7fa7c80a3de0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c813a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): picked mon.noname-d con 0x7fa7c80a4700 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c80a5d40 con 0x7fa7c80a3de0 2024-09-11T03:03:30.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa7c80a5bc0 con 0x7fa7c809b5f0 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa7c80a2700 con 0x7fa7c80a4700 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient(hunting): authenticate will time out at 1764.798676s 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c813a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c80a3de0 0x7fa7c813a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c813a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46626/0 (socket says 172.21.15.98:46626) 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 1 -- 172.21.15.98:0/468444455 learned_addr learned my addr 172.21.15.98:0/468444455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 10 monclient(hunting): get_auth_request con 0x7fa7c80a4700 auth_method 0 2024-09-11T03:03:30.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): get_auth_request con 0x7fa7c80a3de0 auth_method 0 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c6ffd640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa7c809b5f0 0x7fa7c812dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c6ffd640 10 monclient(hunting): get_auth_request con 0x7fa7c809b5f0 auth_method 0 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient(hunting): handle_auth_done global_id 45180 payload 824 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 msgr2=0x7fa7c813a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c813a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa7c809b5f0 msgr2=0x7fa7c812dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa7c809b5f0 0x7fa7c812dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient: _finish_hunting 0 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 monclient: found mon.noname-a 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:30.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 -- 172.21.15.98:0/468444455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c80a5540 con 0x7fa7c80a3de0 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c77fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa7c80a4700 0x7fa7c813a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c67fc640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c80a3de0 0x7fa7c813a430 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0010e20 tx=0x7fa7b000e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 client.?.objecter ms_handle_connect 0x7fa7c80a3de0 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7b001e070 con 0x7fa7c80a3de0 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: dump: 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.939 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: _finish_auth 0 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7b00118c0 con 0x7fa7c80a3de0 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 5 monclient: authenticate success, global_id 45180 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: _renew_subs 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c812e900 con 0x7fa7c80a3de0 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7b000d350 con 0x7fa7c80a3de0 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:30.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7bf7fe640 10 monclient: dump: 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.942 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7c57fa640 4 set_mon_vals no callback set 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 client.45180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: _renew_subs 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c812ebe0 con 0x7fa7c80a3de0 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.914+0000 7fa7cd2d4640 1 librados: init done 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7cd2d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c813aeb0 0x7fa7c8137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c806ca10 con 0x7fa7c813aeb0 2024-09-11T03:03:30.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa7b000e040 con 0x7fa7c80a3de0 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c813aeb0 0x7fa7c8137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7a803df40 0x7fa7a8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient: get_auth_request con 0x7fa7c813aeb0 auth_method 0 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(531..531 src has 1..531) ==== 6657+0+0 (secure 0 0 0) 0x7fa7b0051ce0 con 0x7fa7c80a3de0 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 client.45180.objecter ms_dispatch 0x7fa7c8000ba0 osd_map(531..531 src has 1..531) 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 3 client.45180.objecter handle_osd_map got epochs [531,531] > 0 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 3 client.45180.objecter handle_osd_map decoding full epoch 531 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 20 client.45180.objecter dump_active .. 0 homeless 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c77fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7a803df40 0x7fa7a8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c77fe640 10 monclient: get_auth_request con 0x7fa7a803df40 auth_method 0 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_done global_id 25957 payload 824 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 1 -- 172.21.15.98:0/468444455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7c813b290 con 0x7fa7c813aeb0 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c6ffd640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c813aeb0 0x7fa7c8137740 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80107c0 tx=0x7fa7b8010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 client.45180.objecter ms_handle_connect 0x7fa7c813aeb0 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 client.45180.objecter resend_mon_ops 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7b801a020 con 0x7fa7c813aeb0 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:30.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 monclient: dump: 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.946 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 client.45180.objecter ms_handle_connect 0x7fa7a803df40 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7c77fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7a803df40 0x7fa7a8040400 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7fa7c001c910 tx=0x7fa7c00a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa7b8010e70 con 0x7fa7c813aeb0 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c813aeb0 msgr2=0x7fa7c8137740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c813aeb0 0x7fa7c8137740 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80107c0 tx=0x7fa7b8010c80 comp rx=0 tx=0).stop 2024-09-11T03:03:30.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c813aeb0 msgr2=0x7fa7c8137740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.918+0000 7fa7bf7fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c813aeb0 0x7fa7c8137740 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:30.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c8137c80 0x7fa7c8138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c806e870 con 0x7fa7c8137c80 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c8137c80 0x7fa7c8138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient: get_auth_request con 0x7fa7c8137c80 auth_method 0 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:30.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 10 monclient(hunting): handle_auth_done global_id 25966 payload 824 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 1 -- 172.21.15.98:0/468444455 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7c80a7c70 con 0x7fa7c8137c80 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 10 client.45180.objecter ms_handle_connect 0x7fa7c8137c80 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 10 client.45180.objecter resend_mon_ops 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c6ffd640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c8137c80 0x7fa7c8138060 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80107c0 tx=0x7fa7b80046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7b801a040 con 0x7fa7c8137c80 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 10 monclient: dump: 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:30.951 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:30.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9424 (secure 0 0 0) 0x7fa7b8015e10 con 0x7fa7c8137c80 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c8137c80 msgr2=0x7fa7c8138060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c8137c80 0x7fa7c8138060 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80107c0 tx=0x7fa7b80046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c8137c80 msgr2=0x7fa7c8138060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7bf7fe640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7c8137c80 0x7fa7c8138060 secure :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7fa7b80107c0 tx=0x7fa7b80046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:30.953 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 826701, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:30.954 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.955 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.956 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:30.957 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:30.958 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.959 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.960 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.961 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.964 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:30.965 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.966 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.968 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.969 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.971 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:30.972 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.973 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.974 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:30.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.976 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:30.977 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:30.978 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:30.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 10 librados: watch_flush enter 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 10 librados: watch_flush exit 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 20 client.45180.objecter shutdown clearing up homeless session... 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 10 client.45180.objecter successfully canceled tick 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7a803df40 msgr2=0x7fa7a8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa7a803df40 0x7fa7a8040400 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7fa7c001c910 tx=0x7fa7c00a7070 comp rx=0 tx=0).stop 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 10 monclient: shutdown 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c80a3de0 msgr2=0x7fa7c813a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 --2- 172.21.15.98:0/468444455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7c80a3de0 0x7fa7c813a430 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0010e20 tx=0x7fa7b000e730 comp rx=0 tx=0).stop 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c77fe640 1 -- 172.21.15.98:0/468444455 reap_dead start 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7c77fe640 1 -- 172.21.15.98:0/468444455 reap_dead start 2024-09-11T03:03:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 shutdown_connections 2024-09-11T03:03:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 >> 172.21.15.98:0/468444455 conn(0x7fa7c80a4ed0 msgr2=0x7fa7c80a6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 shutdown_connections 2024-09-11T03:03:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 -- 172.21.15.98:0/468444455 wait complete. 2024-09-11T03:03:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:30.926+0000 7fa7cd2d4640 1 librados: shutdown 2024-09-11T03:03:30.981 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:30.981 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-09-11T03:03:30.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:03:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.094+0000 7f0924e56640 10 monclient: get_monmap_and_config 2024-09-11T03:03:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.094+0000 7f0924e56640 10 monclient: build_initial_monmap 2024-09-11T03:03:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.094+0000 7f0924e56640 10 monclient: monmap: 2024-09-11T03:03:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:31.100189+0000 2024-09-11T03:03:31.098 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:31.100189+0000 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:31.099 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:31.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.094+0000 7f0924e56640 1 Processor -- start 2024-09-11T03:03:31.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.094+0000 7f0924e56640 1 -- start start 2024-09-11T03:03:31.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.094+0000 7f0924e56640 10 monclient: init 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient: _add_conns ranks=[4,1,2,5,10,7,3,6,0,9,8] 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0920139b80 0x7f0920139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient(hunting): picked mon.noname-c con 0x7f0920139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f092006fce0 addr v1:172.21.15.190:6789/0 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200724b0 0x7f09200a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient(hunting): picked mon.noname-b con 0x7f09200724b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09200a94a0 con 0x7f092006fce0 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0920071850 con 0x7f09200724b0 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0920071aa0 con 0x7f0920139b80 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient(hunting): authenticate will time out at 1764.980426s 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091ed9d640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200724b0 msgr2=0x7f09200a8120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0920139b80 0x7f0920139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0920139b80 0x7f0920139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45732/0 (socket says 172.21.15.98:45732) 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200724b0 0x7f09200a8120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 -- 172.21.15.98:0/3326847907 learned_addr learned my addr 172.21.15.98:0/3326847907 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): get_auth_request con 0x7f0920139b80 auth_method 0 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient(hunting): handle_auth_done global_id 24610 payload 274 2024-09-11T03:03:31.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 -- 172.21.15.98:0/3326847907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200724b0 msgr2=0x7f09200a8120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:03:31.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 --2- 172.21.15.98:0/3326847907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200724b0 0x7f09200a8120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 -- 172.21.15.98:0/3326847907 >> v1:172.21.15.190:6789/0 conn(0x7f092006fce0 legacy=0x7f09200a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:31.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:31.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 monclient: found mon.noname-c 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 -- 172.21.15.98:0/3326847907 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092013a4a0 con 0x7f0920139b80 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091e59c640 1 --2- 172.21.15.98:0/3326847907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0920139b80 0x7f0920139f60 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f09100098a0 tx=0x7f091001c460 comp rx=0 tx=0).ready entity=mon.4 client_cookie=fd094825e49e22ac server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 1 -- 172.21.15.98:0/3326847907 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09100b0070 con 0x7f0920139b80 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: dump: 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.107 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: _finish_auth 0 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: _check_auth_tickets 2024-09-11T03:03:31.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 1 -- 172.21.15.98:0/3326847907 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09100aa810 con 0x7f0920139b80 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 1 -- 172.21.15.98:0/3326847907 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09100a9300 con 0x7f0920139b80 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 5 monclient: authenticate success, global_id 24610 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f0924e56640 4 set_mon_vals no callback set 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:31.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.098+0000 7f091dd9b640 10 monclient: dump: 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.110 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: shutdown 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/3326847907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0920139b80 msgr2=0x7f0920139f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 --2- 172.21.15.98:0/3326847907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0920139b80 0x7f0920139f60 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f09100098a0 tx=0x7f091001c460 comp rx=0 tx=0).stop 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/3326847907 shutdown_connections 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 --2- 172.21.15.98:0/3326847907 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09200724b0 0x7f09200a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 --2- 172.21.15.98:0/3326847907 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0920139b80 0x7f0920139f60 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/3326847907 >> 172.21.15.98:0/3326847907 conn(0x7f09200ac6b0 msgr2=0x7f09200a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/3326847907 shutdown_connections 2024-09-11T03:03:31.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/3326847907 wait complete. 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: build_initial_monmap 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: monmap: 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:31.112 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:31.113 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:31.113 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:31.113 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:31.113 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:31.113 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:31.113 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 librados: starting msgr at 2024-09-11T03:03:31.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 librados: starting objecter 2024-09-11T03:03:31.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 Processor -- start 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- start start 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 librados: setting wanted keys 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 librados: calling monclient init 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: init 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: _add_conns ranks=[3,6,2,1,4,7,5,0,8] 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f092006fce0 0x7f092013a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): picked mon.noname-g con 0x7f092006fce0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09200724b0 0x7f092013acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): picked mon.noname-d con 0x7f09200724b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0920139b80 0x7f092013b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): picked mon.noname-b con 0x7f0920139b80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f09200a8870 con 0x7f0920139b80 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0920070160 con 0x7f092006fce0 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0920144450 con 0x7f09200724b0 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): authenticate will time out at 1764.985989s 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f091ed9d640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0920139b80 msgr2=0x7f092013b210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f091ed9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0920139b80 0x7f092013b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09200724b0 0x7f092013acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09200724b0 0x7f092013acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:46636/0 (socket says 172.21.15.98:46636) 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 -- 172.21.15.98:0/1326638875 learned_addr learned my addr 172.21.15.98:0/1326638875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:31.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): get_auth_request con 0x7f09200724b0 auth_method 0 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f091e59c640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f092006fce0 0x7f092013a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient(hunting): handle_auth_done global_id 84399 payload 824 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f092006fce0 msgr2=0x7f092013a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f092006fce0 0x7f092013a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0920139b80 msgr2=0x7f092013b210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0920139b80 0x7f092013b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:31.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 monclient: found mon.noname-d 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 -- 172.21.15.98:0/1326638875 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f092013d300 con 0x7f09200724b0 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f091e59c640 10 monclient: get_auth_request con 0x7f092006fce0 auth_method 0 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f091e59c640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f092006fce0 0x7f092013a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0917fff640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09200724b0 0x7f092013acd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f090800c440 tx=0x7f090800c900 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 client.?.objecter ms_handle_connect 0x7f09200724b0 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0908016020 con 0x7f09200724b0 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:03:31.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: dump: 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.125 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: _finish_auth 0 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 5 monclient: authenticate success, global_id 84399 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0908012aa0 con 0x7f09200724b0 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: _renew_subs 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0908012c90 con 0x7f09200724b0 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:31.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092013d020 con 0x7f09200724b0 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f091d59a640 4 set_mon_vals no callback set 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 10 monclient: dump: 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.127 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 client.84399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: _renew_subs 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f092013bb00 con 0x7f09200724b0 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 librados: init done 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:31.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920133960 0x7f0920133d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f092006ca10 con 0x7f0920133960 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.102+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0908012630 con 0x7f09200724b0 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920133960 0x7f0920133d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f090003df40 0x7f0900040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient: get_auth_request con 0x7f0920133960 auth_method 0 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(531..531 src has 1..531) ==== 6657+0+0 (secure 0 0 0) 0x7f090804f8f0 con 0x7f09200724b0 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 client.84399.objecter ms_dispatch 0x7f0920000ba0 osd_map(531..531 src has 1..531) 2024-09-11T03:03:31.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 3 client.84399.objecter handle_osd_map got epochs [531,531] > 0 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 3 client.84399.objecter handle_osd_map decoding full epoch 531 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 20 client.84399.objecter dump_active .. 0 homeless 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091ed9d640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f090003df40 0x7f0900040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091ed9d640 10 monclient: get_auth_request con 0x7f090003df40 auth_method 0 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 10 monclient(hunting): handle_auth_done global_id 74433 payload 824 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 1 -- 172.21.15.98:0/1326638875 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0920139f60 con 0x7f0920133960 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 client.84399.objecter ms_handle_connect 0x7f0920133960 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091e59c640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920133960 0x7f0920133d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09100ad000 tx=0x7f0910019200 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 client.84399.objecter resend_mon_ops 2024-09-11T03:03:31.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09100b0030 con 0x7f0920133960 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 monclient: dump: 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.131 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 client.84399.objecter ms_handle_connect 0x7f090003df40 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f091ed9d640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f090003df40 0x7f0900040400 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7f092013ba90 tx=0x7f0918009230 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f09100ba070 con 0x7f0920133960 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:31.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920133960 msgr2=0x7f0920133d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920133960 0x7f0920133d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09100ad000 tx=0x7f0910019200 comp rx=0 tx=0).stop 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920133960 msgr2=0x7f0920133d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.106+0000 7f0916ffd640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920133960 0x7f0920133d40 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09100ad000 tx=0x7f0910019200 comp rx=0 tx=0).stop 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920134280 0x7f09200a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f092006e870 con 0x7f0920134280 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920134280 0x7f09200a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient: get_auth_request con 0x7f0920134280 auth_method 0 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 10 monclient(hunting): handle_auth_done global_id 74442 payload 824 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 1 -- 172.21.15.98:0/1326638875 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0920073570 con 0x7f0920134280 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091e59c640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920134280 0x7f09200a0e60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f091001c290 tx=0x7f09100a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 10 client.84399.objecter ms_handle_connect 0x7f0920134280 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 10 client.84399.objecter resend_mon_ops 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09100b0040 con 0x7f0920134280 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 10 monclient: dump: 2024-09-11T03:03:31.134 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.135 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9424 (secure 0 0 0) 0x7f09100b1040 con 0x7f0920134280 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920134280 msgr2=0x7f09200a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920134280 0x7f09200a0e60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f091001c290 tx=0x7f09100a8000 comp rx=0 tx=0).stop 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920134280 msgr2=0x7f09200a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0916ffd640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0920134280 0x7f09200a0e60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 168047, 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:31.137 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.138 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.139 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:31.140 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:31.141 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.142 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:31.144 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:31.145 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.147 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:31.148 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:31.150 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:31.151 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.152 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.153 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.155 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.159 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:03:31.160 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.161 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:31.162 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 10 librados: watch_flush enter 2024-09-11T03:03:31.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 10 librados: watch_flush exit 2024-09-11T03:03:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 20 client.84399.objecter shutdown clearing up homeless session... 2024-09-11T03:03:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 10 client.84399.objecter successfully canceled tick 2024-09-11T03:03:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f090003df40 msgr2=0x7f0900040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f090003df40 0x7f0900040400 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7f092013ba90 tx=0x7f0918009230 comp rx=0 tx=0).stop 2024-09-11T03:03:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 10 monclient: shutdown 2024-09-11T03:03:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:31.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09200724b0 msgr2=0x7f092013acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09200724b0 0x7f092013acd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f090800c440 tx=0x7f090800c900 comp rx=0 tx=0).stop 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f091ed9d640 1 -- 172.21.15.98:0/1326638875 reap_dead start 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 shutdown_connections 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 --2- 172.21.15.98:0/1326638875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f09200724b0 0x7f092013acd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 >> 172.21.15.98:0/1326638875 conn(0x7f09200ac6b0 msgr2=0x7f0920134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 shutdown_connections 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 -- 172.21.15.98:0/1326638875 wait complete. 2024-09-11T03:03:31.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.114+0000 7f0924e56640 1 librados: shutdown 2024-09-11T03:03:31.165 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:31.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-09-11T03:03:31.166 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:03:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient: get_monmap_and_config 2024-09-11T03:03:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient: build_initial_monmap 2024-09-11T03:03:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient: monmap: 2024-09-11T03:03:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:31.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:31.281456+0000 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:31.281456+0000 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:31.280 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:31.281 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:31.281 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:31.281 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 Processor -- start 2024-09-11T03:03:31.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 -- start start 2024-09-11T03:03:31.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient: init 2024-09-11T03:03:31.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:31.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient: _add_conns ranks=[5,10,4,1,9,3,8,7,0,2,6] 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f15940a46a0 0x7f15940a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): picked mon.noname-h con 0x7f15940a46a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f15940a4fc0 0x7f1594139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): picked mon.noname-f con 0x7f15940a4fc0 addr v2:172.21.15.190:3300/0 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1594139e70 0x7f159413a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): picked mon.noname-c con 0x7f1594139e70 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f15940775b0 con 0x7f15940a4fc0 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f15940a11d0 con 0x7f1594139e70 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f15940a0750 con 0x7f15940a46a0 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1599bdd640 10 monclient(hunting): authenticate will time out at 1765.161765s 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1594139e70 0x7f159413a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1594139e70 0x7f159413a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:45742/0 (socket says 172.21.15.98:45742) 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 -- 172.21.15.98:0/2528296131 learned_addr learned my addr 172.21.15.98:0/2528296131 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): get_auth_request con 0x7f1594139e70 auth_method 0 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 1 --2- 172.21.15.98:0/2528296131 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f15940a46a0 0x7f15940a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1592ffd640 1 --2- 172.21.15.98:0/2528296131 >> v2:172.21.15.190:3300/0 conn(0x7f15940a4fc0 0x7f1594139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 10 monclient(hunting): get_auth_request con 0x7f15940a46a0 auth_method 0 2024-09-11T03:03:31.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1592ffd640 10 monclient(hunting): get_auth_request con 0x7f15940a4fc0 auth_method 0 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1592ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1592ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1592ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient(hunting): handle_auth_done global_id 24619 payload 274 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 -- 172.21.15.98:0/2528296131 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f15940a46a0 msgr2=0x7f15940a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 --2- 172.21.15.98:0/2528296131 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f15940a46a0 0x7f15940a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 -- 172.21.15.98:0/2528296131 >> v2:172.21.15.190:3300/0 conn(0x7f15940a4fc0 msgr2=0x7f1594139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 --2- 172.21.15.98:0/2528296131 >> v2:172.21.15.190:3300/0 conn(0x7f15940a4fc0 0x7f1594139930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 monclient: found mon.noname-c 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1593fff640 1 -- 172.21.15.98:0/2528296131 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1594077160 con 0x7f1594139e70 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.278+0000 7f1592ffd640 1 --2- 172.21.15.98:0/2528296131 >> v2:172.21.15.190:3300/0 conn(0x7f15940a4fc0 0x7f1594139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 1 --2- 172.21.15.98:0/2528296131 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1594139e70 0x7f159413a320 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f1584009870 tx=0x7f158401c540 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7e39b512045856ec server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 1 -- 172.21.15.98:0/2528296131 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f15840a6040 con 0x7f1594139e70 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: dump: 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.290 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: _finish_auth 0 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: _check_auth_tickets 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 1 -- 172.21.15.98:0/2528296131 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15840af070 con 0x7f1594139e70 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 1 -- 172.21.15.98:0/2528296131 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f15840aad20 con 0x7f1594139e70 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 5 monclient: authenticate success, global_id 24619 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 4 set_mon_vals no callback set 2024-09-11T03:03:31.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15927fc640 10 monclient: dump: 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.292 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient: shutdown 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- 172.21.15.98:0/2528296131 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1594139e70 msgr2=0x7f159413a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2528296131 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1594139e70 0x7f159413a320 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f1584009870 tx=0x7f158401c540 comp rx=0 tx=0).stop 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- 172.21.15.98:0/2528296131 shutdown_connections 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2528296131 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1594139e70 0x7f159413a320 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2528296131 >> v2:172.21.15.190:3300/0 conn(0x7f15940a4fc0 0x7f1594139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2528296131 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f15940a46a0 0x7f15940a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- 172.21.15.98:0/2528296131 >> 172.21.15.98:0/2528296131 conn(0x7f159406ed00 msgr2=0x7f159406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- 172.21.15.98:0/2528296131 shutdown_connections 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- 172.21.15.98:0/2528296131 wait complete. 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient: build_initial_monmap 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient: monmap: 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:31.294 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 librados: starting msgr at 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 librados: starting objecter 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 Processor -- start 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- start start 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 librados: setting wanted keys 2024-09-11T03:03:31.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 librados: calling monclient init 2024-09-11T03:03:31.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient: init 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient: _add_conns ranks=[1,2,5,7,6,3,0,8,4] 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15940a46a0 0x7f15941d1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): picked mon.noname-f con 0x7f15940a46a0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15940a4fc0 0x7f15941d1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): picked mon.noname-b con 0x7f15940a4fc0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1594139e70 0x7f159413ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): picked mon.noname-h con 0x7f1594139e70 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1594070090 con 0x7f15940a46a0 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f15940a14b0 con 0x7f15940a4fc0 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f15940a53a0 con 0x7f1594139e70 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1599bdd640 10 monclient(hunting): authenticate will time out at 1765.167396s 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1592ffd640 1 -- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15940a4fc0 msgr2=0x7f15941d1980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3301/0 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1592ffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15940a4fc0 0x7f15941d1980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15940a46a0 0x7f15941d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1594139e70 0x7f159413ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15940a46a0 0x7f15941d1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:40170/0 (socket says 172.21.15.98:40170) 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 1 -- 172.21.15.98:0/2914247825 learned_addr learned my addr 172.21.15.98:0/2914247825 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:31.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 10 monclient(hunting): get_auth_request con 0x7f1594139e70 auth_method 0 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 10 monclient(hunting): get_auth_request con 0x7f15940a46a0 auth_method 0 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f1593fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.282+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 10 monclient(hunting): handle_auth_done global_id 74453 payload 824 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15940a4fc0 msgr2=0x7f15941d1980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f15940a4fc0 0x7f15941d1980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15940a46a0 msgr2=0x7f15941d1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f15940a46a0 0x7f15941d1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 10 monclient: _finish_hunting 0 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 1 monclient: found mon.noname-h 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 1 -- 172.21.15.98:0/2914247825 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1594123f30 con 0x7f1594139e70 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1593fff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1594139e70 0x7f159413ca00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f15840ab000 tx=0x7f15840192e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 client.?.objecter ms_handle_connect 0x7f1594139e70 2024-09-11T03:03:31.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f15840b9070 con 0x7f1594139e70 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: dump: 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.307 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: _finish_auth 0 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: _check_auth_tickets 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1584003bb0 con 0x7f1594139e70 2024-09-11T03:03:31.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 5 monclient: authenticate success, global_id 74453 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 10 monclient: _renew_subs 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1591ffb640 4 set_mon_vals no callback set 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1594124210 con 0x7f1594139e70 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1584003d60 con 0x7f1594139e70 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 10 client.74453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: dump: 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.309 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.310 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 10 monclient: _renew_subs 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f159413a310 con 0x7f1594139e70 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 1 librados: init done 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413cf40 0x7f159413d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f159413f0c0 con 0x7f159413cf40 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413cf40 0x7f159413d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient: get_auth_request con 0x7f159413cf40 auth_method 0 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f15840aa440 con 0x7f1594139e70 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f157403df40 0x7f1574040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 10 monclient(hunting): handle_auth_done global_id 74462 payload 824 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(531..531 src has 1..531) ==== 6657+0+0 (secure 0 0 0) 0x7f15840e9920 con 0x7f1594139e70 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 client.74453.objecter ms_dispatch 0x7f1594000ba0 osd_map(531..531 src has 1..531) 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 3 client.74453.objecter handle_osd_map got epochs [531,531] > 0 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 3 client.74453.objecter handle_osd_map decoding full epoch 531 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 1 -- 172.21.15.98:0/2914247825 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15940a8960 con 0x7f159413cf40 2024-09-11T03:03:31.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 20 client.74453.objecter dump_active .. 0 homeless 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1592ffd640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f157403df40 0x7f1574040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1592ffd640 10 monclient: get_auth_request con 0x7f157403df40 auth_method 0 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 client.74453.objecter ms_handle_connect 0x7f159413cf40 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 client.74453.objecter resend_mon_ops 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f15937fe640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413cf40 0x7f159413d320 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f158c011920 tx=0x7f158c011de0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f158c019070 con 0x7f159413cf40 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 monclient: dump: 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.313 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.314 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 10 client.74453.objecter ms_handle_connect 0x7f157403df40 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f1592ffd640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f157403df40 0x7f1574040400 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7f157c0095c0 tx=0x7f157c009450 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.286+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f158c004240 con 0x7f159413cf40 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.290+0000 7f158bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.290+0000 7f158bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.290+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413cf40 msgr2=0x7f159413d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.290+0000 7f158bfff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413cf40 0x7f159413d320 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f158c011920 tx=0x7f158c011de0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.290+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413cf40 msgr2=0x7f159413d320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.290+0000 7f158bfff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413cf40 0x7f159413d320 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f158c011920 tx=0x7f158c011de0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f1599bdd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413d860 0x7f159409da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f1599bdd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f159406ca10 con 0x7f159413d860 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413d860 0x7f159409da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient: get_auth_request con 0x7f159413d860 auth_method 0 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 10 monclient(hunting): handle_auth_done global_id 74471 payload 824 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 1 -- 172.21.15.98:0/2914247825 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1594069090 con 0x7f159413d860 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f15937fe640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413d860 0x7f159409da60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f158c011920 tx=0x7f158c0046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f158bfff640 10 client.74453.objecter ms_handle_connect 0x7f159413d860 2024-09-11T03:03:31.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f158bfff640 10 client.74453.objecter resend_mon_ops 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f158c019040 con 0x7f159413d860 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f158bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f158bfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.294+0000 7f158bfff640 10 monclient: dump: 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.317 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9257 (secure 0 0 0) 0x7f158c01e450 con 0x7f159413d860 2024-09-11T03:03:31.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f158bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f158bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413d860 msgr2=0x7f159409da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f158bfff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413d860 0x7f159409da60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f158c011920 tx=0x7f158c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f158bfff640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413d860 msgr2=0x7f159409da60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f158bfff640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f159413d860 0x7f159409da60 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f158c011920 tx=0x7f158c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:31.319 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 168227, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.320 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.321 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.322 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:31.323 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.324 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:31.325 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.326 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.327 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.329 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.330 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.332 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.333 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.334 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:31.335 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:31.336 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.337 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.338 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.339 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:31.340 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:31.341 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.342 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:31.343 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:31.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 10 librados: watch_flush enter 2024-09-11T03:03:31.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 10 librados: watch_flush exit 2024-09-11T03:03:31.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 20 client.74453.objecter shutdown clearing up homeless session... 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 10 client.74453.objecter successfully canceled tick 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f157403df40 msgr2=0x7f1574040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f157403df40 0x7f1574040400 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7f157c0095c0 tx=0x7f157c009450 comp rx=0 tx=0).stop 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 10 monclient: shutdown 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1594139e70 msgr2=0x7f159413ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1594139e70 0x7f159413ca00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f15840ab000 tx=0x7f15840192e0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1593fff640 1 -- 172.21.15.98:0/2914247825 reap_dead start 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 shutdown_connections 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 --2- 172.21.15.98:0/2914247825 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1594139e70 0x7f159413ca00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 >> 172.21.15.98:0/2914247825 conn(0x7f159406ed00 msgr2=0x7f1594072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 shutdown_connections 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 -- 172.21.15.98:0/2914247825 wait complete. 2024-09-11T03:03:31.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.298+0000 7f1599bdd640 1 librados: shutdown 2024-09-11T03:03:31.346 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:31.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-09-11T03:03:31.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:03:31.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient: get_monmap_and_config 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient: build_initial_monmap 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient: monmap: 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:03:31.461889+0000 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:03:31.461889+0000 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:03:31.460 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:03:31.461 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 Processor -- start 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 -- start start 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient: init 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient: _add_conns ranks=[5,8,4,1,10,6,7,3,0,2,9] 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac980a52a0 0x7fac9809ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): picked mon.noname-h con 0x7fac980a52a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac98073c20 0x7fac9809cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): picked mon.noname-e con 0x7fac98073c20 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:03:31.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fac9809d4d0 0x7fac9813c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): picked mon.noname-c con 0x7fac9809d4d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fac9809dbf0 con 0x7fac9809d4d0 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fac9809f600 con 0x7fac980a52a0 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fac980a5680 con 0x7fac98073c20 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7faca05e0640 10 monclient(hunting): authenticate will time out at 1765.342659s 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac98073c20 0x7fac9809cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fac9809d4d0 0x7fac9813c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac98073c20 0x7fac9809cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44960/0 (socket says 172.21.15.98:44960) 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): get_auth_request con 0x7fac9809d4d0 auth_method 0 2024-09-11T03:03:31.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 1 -- 172.21.15.98:0/2219356466 learned_addr learned my addr 172.21.15.98:0/2219356466 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 10 monclient(hunting): get_auth_request con 0x7fac98073c20 auth_method 0 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9e37c640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac980a52a0 0x7fac9809ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9e37c640 10 monclient(hunting): get_auth_request con 0x7fac980a52a0 auth_method 0 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9e37c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9e37c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9e37c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient(hunting): handle_auth_done global_id 24628 payload 274 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 1 -- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac98073c20 msgr2=0x7fac9809cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac98073c20 0x7fac9809cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 1 -- 172.21.15.98:0/2219356466 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac980a52a0 msgr2=0x7fac9809ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac980a52a0 0x7fac9809ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient: _finish_hunting 0 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 1 monclient: found mon.noname-c 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:31.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9eb7d640 1 -- 172.21.15.98:0/2219356466 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac980a41a0 con 0x7fac9809d4d0 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.458+0000 7fac9db7b640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac98073c20 0x7fac9809cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9eb7d640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fac9809d4d0 0x7fac9813c480 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fac9000d080 tx=0x7fac9001ea30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=9e551a4bd110ebec server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 1 -- 172.21.15.98:0/2219356466 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac900ad070 con 0x7fac9809d4d0 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: dump: 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.470 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: _finish_auth 0 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: _check_auth_tickets 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 5 monclient: authenticate success, global_id 24628 2024-09-11T03:03:31.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 1 -- 172.21.15.98:0/2219356466 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac900a7b00 con 0x7fac9809d4d0 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient: get_monmap_and_config success 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 4 set_mon_vals no callback set 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 1 -- 172.21.15.98:0/2219356466 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac900a7d30 con 0x7fac9809d4d0 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9d37a640 10 monclient: dump: 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient: shutdown 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:31.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fac9809d4d0 msgr2=0x7fac9813c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fac9809d4d0 0x7fac9813c480 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fac9000d080 tx=0x7fac9001ea30 comp rx=0 tx=0).stop 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- 172.21.15.98:0/2219356466 shutdown_connections 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fac9809d4d0 0x7fac9813c480 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fac98073c20 0x7fac9809cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 --2- 172.21.15.98:0/2219356466 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac980a52a0 0x7fac9809ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- 172.21.15.98:0/2219356466 >> 172.21.15.98:0/2219356466 conn(0x7fac980a4780 msgr2=0x7fac980a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- 172.21.15.98:0/2219356466 shutdown_connections 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- 172.21.15.98:0/2219356466 wait complete. 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient: build_initial_monmap 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient: monmap: 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:03:31.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 librados: starting msgr at 2024-09-11T03:03:31.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 librados: starting objecter 2024-09-11T03:03:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 Processor -- start 2024-09-11T03:03:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- start start 2024-09-11T03:03:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 librados: setting wanted keys 2024-09-11T03:03:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 librados: calling monclient init 2024-09-11T03:03:31.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient: init 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient: _add_conns ranks=[0,5,3,1,2,6,4,8,7] 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98073c20 0x7fac98133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): picked mon.noname-a con 0x7fac98073c20 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac9809d4d0 0x7fac981343f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): picked mon.noname-h con 0x7fac9809d4d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac980a52a0 0x7fac980715d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): picked mon.noname-g con 0x7fac980a52a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:03:31.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac9813eb50 con 0x7fac98073c20 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fac9813e9d0 con 0x7fac980a52a0 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fac9813ecd0 con 0x7fac9809d4d0 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7faca05e0640 10 monclient(hunting): authenticate will time out at 1765.347661s 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98073c20 0x7fac98133eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98073c20 0x7fac98133eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46404/0 (socket says 172.21.15.98:46404) 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 1 -- 172.21.15.98:0/2675553674 learned_addr learned my addr 172.21.15.98:0/2675553674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 10 monclient(hunting): get_auth_request con 0x7fac98073c20 auth_method 0 2024-09-11T03:03:31.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9e37c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9eb7d640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac980a52a0 0x7fac980715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.462+0000 7fac9db7b640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac9809d4d0 0x7fac981343f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9eb7d640 10 monclient(hunting): get_auth_request con 0x7fac980a52a0 auth_method 0 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9eb7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9eb7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9eb7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 10 monclient(hunting): handle_auth_done global_id 45189 payload 824 2024-09-11T03:03:31.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac9809d4d0 msgr2=0x7fac981343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac9809d4d0 0x7fac981343f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac980a52a0 msgr2=0x7fac980715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fac980a52a0 0x7fac980715d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 10 monclient: _finish_hunting 0 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 1 monclient: found mon.noname-a 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 1 -- 172.21.15.98:0/2675553674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac980a6180 con 0x7fac98073c20 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient: get_auth_request con 0x7fac9809d4d0 auth_method 0 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fac9809d4d0 0x7fac981343f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9e37c640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98073c20 0x7fac98133eb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fac9400d620 tx=0x7fac9400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 client.?.objecter ms_handle_connect 0x7fac98073c20 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac94007380 con 0x7fac98073c20 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: dump: 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: _finish_auth 0 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: _check_auth_tickets 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:03:31.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac94019070 con 0x7fac98073c20 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 5 monclient: authenticate success, global_id 45189 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 10 monclient: _renew_subs 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac98072f30 con 0x7fac98073c20 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac94014430 con 0x7fac98073c20 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9cb79640 4 set_mon_vals no callback set 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: dump: 2024-09-11T03:03:31.489 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.490 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 10 client.45189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 10 monclient: _renew_subs 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac9811e510 con 0x7fac98073c20 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 1 librados: init done 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98071b10 0x7fac98071ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fac9806ca10 con 0x7fac98071b10 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fac94013070 con 0x7fac98073c20 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98071b10 0x7fac98071ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient: get_auth_request con 0x7fac98071b10 auth_method 0 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fac7c03df40 0x7fac7c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(531..531 src has 1..531) ==== 6657+0+0 (secure 0 0 0) 0x7fac9400f040 con 0x7fac98073c20 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 client.45189.objecter ms_dispatch 0x7fac98000ba0 osd_map(531..531 src has 1..531) 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 3 client.45189.objecter handle_osd_map got epochs [531,531] > 0 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 3 client.45189.objecter handle_osd_map decoding full epoch 531 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 20 client.45189.objecter dump_active .. 0 homeless 2024-09-11T03:03:31.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9eb7d640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fac7c03df40 0x7fac7c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9eb7d640 10 monclient: get_auth_request con 0x7fac7c03df40 auth_method 0 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 10 monclient(hunting): handle_auth_done global_id 20176 payload 824 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 1 -- 172.21.15.98:0/2675553674 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac98068b40 con 0x7fac98071b10 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 client.45189.objecter ms_handle_connect 0x7fac98071b10 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 client.45189.objecter resend_mon_ops 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9db7b640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98071b10 0x7fac98071ef0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fac98072d00 tx=0x7fac88010c20 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 client.45189.objecter ms_handle_connect 0x7fac7c03df40 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac9eb7d640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fac7c03df40 0x7fac7c040400 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7fac900b2000 tx=0x7fac900a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac8800f040 con 0x7fac98071b10 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: dump: 2024-09-11T03:03:31.493 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.494 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac88010e70 con 0x7fac98071b10 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98071b10 msgr2=0x7fac98071ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98071b10 0x7fac98071ef0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fac98072d00 tx=0x7fac88010c20 comp rx=0 tx=0).stop 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98071b10 msgr2=0x7fac98071ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.466+0000 7fac8effd640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98071b10 0x7fac98071ef0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98072430 0x7fac9813abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:03:31.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fac9806e870 con 0x7fac98072430 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98072430 0x7fac9813abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient: get_auth_request con 0x7fac98072430 auth_method 0 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 10 monclient(hunting): handle_auth_done global_id 20185 payload 824 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 1 -- 172.21.15.98:0/2675553674 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac9809ca50 con 0x7fac98072430 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9db7b640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98072430 0x7fac9813abe0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fac8801b000 tx=0x7fac880046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 10 client.45189.objecter ms_handle_connect 0x7fac98072430 2024-09-11T03:03:31.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 10 client.45189.objecter resend_mon_ops 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac8800ee30 con 0x7fac98072430 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 10 monclient: dump: 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:03:31.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9426 (secure 0 0 0) 0x7fac88025070 con 0x7fac98072430 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:03:31.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98072430 msgr2=0x7fac9813abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98072430 0x7fac9813abe0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fac8801b000 tx=0x7fac880046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98072430 msgr2=0x7fac9813abe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac8effd640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fac98072430 0x7fac9813abe0 secure :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7fac8801b000 tx=0x7fac880046d0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 100, 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 827249, 2024-09-11T03:03:31.499 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:03:31.500 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.501 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.502 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:03:31.503 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:03:31.504 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:03:31.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:03:31.509 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.510 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.511 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.513 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.514 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:03:31.515 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.516 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.517 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:03:31.518 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.519 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:03:31.520 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:03:31.521 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.522 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:03:31.523 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:03:31.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 10 librados: watch_flush enter 2024-09-11T03:03:31.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 10 librados: watch_flush exit 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 20 client.45189.objecter shutdown clearing up homeless session... 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 10 client.45189.objecter successfully canceled tick 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fac7c03df40 msgr2=0x7fac7c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fac7c03df40 0x7fac7c040400 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7fac900b2000 tx=0x7fac900a9070 comp rx=0 tx=0).stop 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 10 monclient: shutdown 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98073c20 msgr2=0x7fac98133eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 --2- 172.21.15.98:0/2675553674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac98073c20 0x7fac98133eb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fac9400d620 tx=0x7fac9400dae0 comp rx=0 tx=0).stop 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9eb7d640 1 -- 172.21.15.98:0/2675553674 reap_dead start 2024-09-11T03:03:31.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7fac9eb7d640 1 -- 172.21.15.98:0/2675553674 reap_dead start 2024-09-11T03:03:31.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 shutdown_connections 2024-09-11T03:03:31.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.474+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 >> 172.21.15.98:0/2675553674 conn(0x7fac980a4780 msgr2=0x7fac9809fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:03:31.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.478+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 shutdown_connections 2024-09-11T03:03:31.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.478+0000 7faca05e0640 1 -- 172.21.15.98:0/2675553674 wait complete. 2024-09-11T03:03:31.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:03:31.478+0000 7faca05e0640 1 librados: shutdown 2024-09-11T03:03:31.527 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-09-11T03:03:31.527 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T03:04:33.512 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9089 ms) 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6073 ms) 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (4851 ms) 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:04:33.513 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (28422 ms) 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (33598 ms) 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12022 ms) 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 00000012 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12049 ms) 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: require_osd_release = squid 2024-09-11T03:04:33.514 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5939 ms) 2024-09-11T03:04:33.515 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (648165 ms total) 2024-09-11T03:04:33.515 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T03:04:33.515 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-11T03:04:33.515 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-11T03:04:33.515 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (957 ms) 2024-09-11T03:04:33.515 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-11T03:04:33.515 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10979 ms) 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18089 ms) 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (8 ms) 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30034 ms total) 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7077 ms) 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (19028 ms) 2024-09-11T03:04:33.516 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: waiting for scrub... 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: done waiting 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (38845 ms) 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10020 ms) 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8096 ms) 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-11T03:04:33.517 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8008 ms) 2024-09-11T03:04:33.518 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-11T03:04:33.518 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9973 ms) 2024-09-11T03:04:33.518 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-11T03:05:01.528 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-11T03:05:01.529 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T03:05:01.529 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T03:05:01.533 INFO:tasks.ceph.mon.b:Started 2024-09-11T03:05:01.533 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:05:01.533 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:01.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.649+0000 7f6ef180d640 10 monclient: get_monmap_and_config 2024-09-11T03:05:01.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.649+0000 7f6ef180d640 10 monclient: build_initial_monmap 2024-09-11T03:05:01.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.649+0000 7f6ef180d640 10 monclient: monmap: 2024-09-11T03:05:01.654 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:01.654 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:01.654 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:01.656647+0000 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:01.656647+0000 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:01.655 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:01.656 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:01.656 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:01.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.649+0000 7f6ef180d640 1 Processor -- start 2024-09-11T03:05:01.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- start start 2024-09-11T03:05:01.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient: init 2024-09-11T03:05:01.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:01.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient: _add_conns ranks=[3,9,7,5,6,8,2,0,1,4,10] 2024-09-11T03:05:01.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6eec137990 0x7f6eec09d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:01.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): picked mon.noname-g con 0x7f6eec137990 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6eec0a2aa0 0x7f6eec137d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): picked mon.noname-a con 0x7f6eec0a2aa0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6eec0a4710 0x7f6eec0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): picked mon.noname-i con 0x7f6eec0a4710 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6eec073da0 con 0x7f6eec0a2aa0 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f6eec0a2f20 con 0x7f6eec137990 2024-09-11T03:05:01.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f6eec09ce60 con 0x7f6eec0a4710 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient(hunting): authenticate will time out at 1855.536542s 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6eec0a2aa0 0x7f6eec137d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6eec0a2aa0 0x7f6eec137d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39954/0 (socket says 172.21.15.98:39954) 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 -- 172.21.15.98:0/3200666117 learned_addr learned my addr 172.21.15.98:0/3200666117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): get_auth_request con 0x7f6eec0a2aa0 auth_method 0 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:01.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eeb7fe640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6eec0a4710 0x7f6eec0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6eec137990 0x7f6eec09d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_done global_id 45198 payload 274 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 -- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6eec0a4710 msgr2=0x7f6eec0a4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6eec0a4710 0x7f6eec0a4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 -- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6eec137990 msgr2=0x7f6eec09d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6eec137990 0x7f6eec09d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient: _finish_hunting 0 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 monclient: found mon.noname-a 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 -- 172.21.15.98:0/3200666117 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eec1382b0 con 0x7f6eec0a2aa0 2024-09-11T03:05:01.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eeaffd640 10 monclient: get_auth_request con 0x7f6eec137990 auth_method 0 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eeb7fe640 10 monclient: get_auth_request con 0x7f6eec0a4710 auth_method 0 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6eec137990 0x7f6eec09d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eeb7fe640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6eec0a4710 0x7f6eec0a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6eea7fc640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec137d70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009870 tx=0x7f6ee001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf2c3f50276ca08 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 1 -- 172.21.15.98:0/3200666117 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ee00ab8d0 con 0x7f6eec0a2aa0 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: dump: 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:01.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 1 -- 172.21.15.98:0/3200666117 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee00b0070 con 0x7f6eec0a2aa0 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 5 monclient: authenticate success, global_id 45198 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 1 -- 172.21.15.98:0/3200666117 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ee00aa3c0 con 0x7f6eec0a2aa0 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 4 set_mon_vals no callback set 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:01.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ee9ffb640 10 monclient: dump: 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:01.671 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 10 monclient: shutdown 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- 172.21.15.98:0/3200666117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 msgr2=0x7f6eec137d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec137d70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0009870 tx=0x7f6ee001c540 comp rx=0 tx=0).stop 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- 172.21.15.98:0/3200666117 shutdown_connections 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec137d70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6eec0a4710 0x7f6eec0a4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 --2- 172.21.15.98:0/3200666117 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f6eec137990 0x7f6eec09d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- 172.21.15.98:0/3200666117 >> 172.21.15.98:0/3200666117 conn(0x7f6eec06ed00 msgr2=0x7f6eec06f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- 172.21.15.98:0/3200666117 shutdown_connections 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.653+0000 7f6ef180d640 1 -- 172.21.15.98:0/3200666117 wait complete. 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: build_initial_monmap 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: monmap: 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:01.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 librados: starting msgr at 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 librados: starting objecter 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 Processor -- start 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 -- start start 2024-09-11T03:05:01.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 librados: setting wanted keys 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 librados: calling monclient init 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: init 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _add_conns ranks=[0,8,4,3,1,2,7,5,6] 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec0a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): picked mon.noname-a con 0x7f6eec0a2aa0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6eec0a4710 0x7f6eec0a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): picked mon.noname-e con 0x7f6eec0a4710 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6eec137990 0x7f6eec0a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): picked mon.noname-c con 0x7f6eec137990 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6eec141340 con 0x7f6eec0a2aa0 2024-09-11T03:05:01.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6eec1411c0 con 0x7f6eec137990 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6eec1414c0 con 0x7f6eec0a4710 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient(hunting): authenticate will time out at 1855.541375s 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec0a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec0a0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39958/0 (socket says 172.21.15.98:39958) 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 1 -- 172.21.15.98:0/2663805472 learned_addr learned my addr 172.21.15.98:0/2663805472 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 10 monclient(hunting): get_auth_request con 0x7f6eec0a2aa0 auth_method 0 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:01.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6eec0a4710 0x7f6eec0a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): get_auth_request con 0x7f6eec0a4710 auth_method 0 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6eec137990 0x7f6eec0a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 10 monclient(hunting): get_auth_request con 0x7f6eec137990 auth_method 0 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:01.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient(hunting): handle_auth_done global_id 25145 payload 824 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 -- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6eec137990 msgr2=0x7f6eec0a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6eec137990 0x7f6eec0a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 -- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 msgr2=0x7f6eec0a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec0a0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient: _finish_hunting 0 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 monclient: found mon.noname-e 2024-09-11T03:05:01.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 -- 172.21.15.98:0/2663805472 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eec11fbd0 con 0x7f6eec0a4710 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec0a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6eea7fc640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6eec0a4710 0x7f6eec0a0740 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00ac000 tx=0x7f6ee00192e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 client.?.objecter ms_handle_connect 0x7f6eec0a4710 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 1 -- 172.21.15.98:0/2663805472 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ee0003bc0 con 0x7f6eec0a4710 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: dump: 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:01.679 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:01.680 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: _finish_auth 0 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: _check_auth_tickets 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 1 -- 172.21.15.98:0/2663805472 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee0003dc0 con 0x7f6eec0a4710 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 1 -- 172.21.15.98:0/2663805472 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ee00abd40 con 0x7f6eec0a4710 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 monclient: dump: 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:01.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:01.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ee97fa640 4 set_mon_vals no callback set 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 5 monclient: authenticate success, global_id 25145 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _renew_subs 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eec0a7ec0 con 0x7f6eec0a4710 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 client.25145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _renew_subs 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eec13d9f0 con 0x7f6eec0a4710 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 librados: init done 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:01.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eec0a4b70 con 0x7f6eec0a4710 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 1 -- 172.21.15.98:0/2663805472 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6ee00ab460 con 0x7f6eec0a4710 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6ed803df40 0x7f6ed8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 1 -- 172.21.15.98:0/2663805472 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(603..603 src has 1..603) ==== 6741+0+0 (secure 0 0 0) 0x7f6ee00e9fe0 con 0x7f6eec0a4710 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 10 client.25145.objecter ms_dispatch 0x7f6eec000ba0 osd_map(603..603 src has 1..603) 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 3 client.25145.objecter handle_osd_map got epochs [603,603] > 0 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 3 client.25145.objecter handle_osd_map decoding full epoch 603 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.657+0000 7f6ed3fff640 20 client.25145.objecter dump_active .. 0 homeless 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.661+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6ed803df40 0x7f6ed8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.661+0000 7f6eeaffd640 10 monclient: get_auth_request con 0x7f6ed803df40 auth_method 0 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.661+0000 7f6ed3fff640 10 client.25145.objecter ms_handle_connect 0x7f6ed803df40 2024-09-11T03:05:01.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.661+0000 7f6eeaffd640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6ed803df40 0x7f6ed8040400 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f6edc006fd0 tx=0x7f6edc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:01.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.665+0000 7f6ed3fff640 1 -- 172.21.15.98:0/2663805472 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6ee0016130 con 0x7f6eec0a4710 2024-09-11T03:05:01.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.665+0000 7f6ed3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:01.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.665+0000 7f6ed3fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.897+0000 7f6ef180d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:01.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.897+0000 7f6ef180d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.897+0000 7f6ef180d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.897+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6eec11f610 con 0x7f6eec0a4710 2024-09-11T03:05:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ed3fff640 1 -- 172.21.15.98:0/2663805472 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3188 (secure 0 0 0) 0x7f6ee00aadd0 con 0x7f6eec0a4710 2024-09-11T03:05:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ed3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:01.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ed3fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:01.904 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":92,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 10 librados: watch_flush enter 2024-09-11T03:05:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 10 librados: watch_flush exit 2024-09-11T03:05:01.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 20 client.25145.objecter shutdown clearing up homeless session... 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 10 client.25145.objecter successfully canceled tick 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6ed803df40 msgr2=0x7f6ed8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6ed803df40 0x7f6ed8040400 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f6edc006fd0 tx=0x7f6edc008040 comp rx=0 tx=0).stop 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 10 monclient: shutdown 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6eec0a4710 msgr2=0x7f6eec0a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6eec0a4710 0x7f6eec0a0740 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00ac000 tx=0x7f6ee00192e0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 shutdown_connections 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6ed803df40 0x7f6ed8040400 unknown :-1 s=CLOSED pgs=3652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f6eec137990 0x7f6eec0a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6eec0a4710 0x7f6eec0a0740 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 --2- 172.21.15.98:0/2663805472 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6eec0a2aa0 0x7f6eec0a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 >> 172.21.15.98:0/2663805472 conn(0x7f6eec06ed00 msgr2=0x7f6eec09d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 shutdown_connections 2024-09-11T03:05:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 -- 172.21.15.98:0/2663805472 wait complete. 2024-09-11T03:05:01.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:01.901+0000 7f6ef180d640 1 librados: shutdown 2024-09-11T03:05:04.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient: get_monmap_and_config 2024-09-11T03:05:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient: build_initial_monmap 2024-09-11T03:05:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient: monmap: 2024-09-11T03:05:05.052 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:05.054519+0000 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:05.054519+0000 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:05.053 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:05.054 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:05.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 1 Processor -- start 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 1 -- start start 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient: init 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient: _add_conns ranks=[1,7,9,3,6,0,5,8,4,2,10] 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fec74139b10 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec7406fe30 0x7fec740725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient(hunting): picked mon.noname-i con 0x7fec7406fe30 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec7413ea00 0x7fec74140df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient(hunting): picked mon.noname-a con 0x7fec7413ea00 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:05.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7413a600 con 0x7fec74139b10 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fec740a9650 con 0x7fec7413ea00 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fec74071350 con 0x7fec7406fe30 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7bee3640 10 monclient(hunting): authenticate will time out at 1858.934813s 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec7413ea00 0x7fec74140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fec7413ea00 0x7fec74140df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:39998/0 (socket says 172.21.15.98:39998) 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 -- 172.21.15.98:0/1209028447 learned_addr learned my addr 172.21.15.98:0/1209028447 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): get_auth_request con 0x7fec7413ea00 auth_method 0 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:05.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7947e640 1 --2- 172.21.15.98:0/1209028447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec7406fe30 0x7fec740725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7947e640 10 monclient(hunting): get_auth_request con 0x7fec7406fe30 auth_method 0 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7947e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7947e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7947e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient(hunting): handle_auth_done global_id 45216 payload 274 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 -- 172.21.15.98:0/1209028447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec7406fe30 msgr2=0x7fec740725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 --2- 172.21.15.98:0/1209028447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec7406fe30 0x7fec740725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 -- 172.21.15.98:0/1209028447 >> v1:172.21.15.190:6789/0 conn(0x7fec74139b10 legacy=0x7fec74139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:05.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient: _finish_hunting 0 2024-09-11T03:05:05.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 monclient: found mon.noname-a 2024-09-11T03:05:05.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:05.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:05.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.049+0000 7fec7a480640 1 -- 172.21.15.98:0/1209028447 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec74071630 con 0x7fec7413ea00 2024-09-11T03:05:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7a480640 1 --2- 172.21.15.98:0/1209028447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec7413ea00 0x7fec74140df0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec6c009870 tx=0x7fec6c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f756af0adaa9e96 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 1 -- 172.21.15.98:0/1209028447 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec6c0b0070 con 0x7fec7413ea00 2024-09-11T03:05:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: dump: 2024-09-11T03:05:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:05.065 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:05.066 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: _finish_auth 0 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: _check_auth_tickets 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 1 -- 172.21.15.98:0/1209028447 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6c0aa810 con 0x7fec7413ea00 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 5 monclient: authenticate success, global_id 45216 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 4 set_mon_vals no callback set 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 1 -- 172.21.15.98:0/1209028447 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec6c0a9300 con 0x7fec7413ea00 2024-09-11T03:05:05.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec78c7d640 10 monclient: dump: 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:05.068 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 10 monclient: shutdown 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 -- 172.21.15.98:0/1209028447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec7413ea00 msgr2=0x7fec74140df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 --2- 172.21.15.98:0/1209028447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec7413ea00 0x7fec74140df0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec6c009870 tx=0x7fec6c01c420 comp rx=0 tx=0).stop 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 -- 172.21.15.98:0/1209028447 shutdown_connections 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 --2- 172.21.15.98:0/1209028447 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec7406fe30 0x7fec740725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 --2- 172.21.15.98:0/1209028447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec7413ea00 0x7fec74140df0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 -- 172.21.15.98:0/1209028447 >> 172.21.15.98:0/1209028447 conn(0x7fec740ac740 msgr2=0x7fec740a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 -- 172.21.15.98:0/1209028447 shutdown_connections 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 -- 172.21.15.98:0/1209028447 wait complete. 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 10 monclient: build_initial_monmap 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 10 monclient: monmap: 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:05.070 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 librados: starting msgr at 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 librados: starting objecter 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 Processor -- start 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 -- start start 2024-09-11T03:05:05.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 librados: setting wanted keys 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 1 librados: calling monclient init 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.053+0000 7fec7bee3640 10 monclient: init 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _add_conns ranks=[7,2,8,5,0,3,6,1,4] 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec74139b10 0x7fec74135780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): picked mon.noname-i con 0x7fec74139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 0x7fec74135cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): picked mon.noname-b con 0x7fec7413ea00 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fec74136200 0x7fec740a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): picked mon.noname-e con 0x7fec74136200 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:05.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fec74143ee0 con 0x7fec7413ea00 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fec74143d60 con 0x7fec74139b10 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fec74144060 con 0x7fec74136200 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient(hunting): authenticate will time out at 1858.940350s 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 0x7fec74135cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fec74136200 0x7fec740a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 0x7fec74135cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:43650/0 (socket says 172.21.15.98:43650) 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 1 -- 172.21.15.98:0/3564653517 learned_addr learned my addr 172.21.15.98:0/3564653517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 10 monclient(hunting): get_auth_request con 0x7fec7413ea00 auth_method 0 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fec74136200 0x7fec740a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:56444/0 (socket says 172.21.15.98:56444) 2024-09-11T03:05:05.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): get_auth_request con 0x7fec74136200 auth_method 0 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 1 -- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 msgr2=0x7fec74135cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 1 -- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 msgr2=0x7fec74135cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 0x7fec74135cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:05.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7947e640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 0x7fec74135cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec74139b10 0x7fec74135780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 10 monclient(hunting): get_auth_request con 0x7fec74139b10 auth_method 0 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient(hunting): handle_auth_done global_id 25154 payload 824 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 -- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec74139b10 msgr2=0x7fec74135780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec74139b10 0x7fec74135780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 -- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 msgr2=0x7fec74135cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 0x7fec74135cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient: _finish_hunting 0 2024-09-11T03:05:05.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 monclient: found mon.noname-e 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 -- 172.21.15.98:0/3564653517 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec740a31d0 con 0x7fec74136200 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec74139b10 0x7fec74135780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7a480640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fec74136200 0x7fec740a0e60 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fec740720e0 tx=0x7fec6c0191c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 client.?.objecter ms_handle_connect 0x7fec74136200 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 1 -- 172.21.15.98:0/3564653517 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec6c0b9070 con 0x7fec74136200 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: dump: 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:05.076 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:05.077 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 5 monclient: authenticate success, global_id 25154 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _renew_subs 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec7413e220 con 0x7fec74136200 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 1 -- 172.21.15.98:0/3564653517 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6c004040 con 0x7fec74136200 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:05.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 1 -- 172.21.15.98:0/3564653517 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec6c0b0040 con 0x7fec74136200 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 monclient: dump: 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:05.079 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec737fe640 4 set_mon_vals no callback set 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 client.25154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _renew_subs 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fec740a2270 con 0x7fec74136200 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 librados: init done 2024-09-11T03:05:05.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec740a3410 con 0x7fec74136200 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 1 -- 172.21.15.98:0/3564653517 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fec6c0041f0 con 0x7fec74136200 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fec5c03df30 0x7fec5c0403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 1 -- 172.21.15.98:0/3564653517 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(606..606 src has 1..606) ==== 6853+0+0 (secure 0 0 0) 0x7fec6c0e9060 con 0x7fec74136200 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 client.25154.objecter ms_dispatch 0x7fec74000ba0 osd_map(606..606 src has 1..606) 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 3 client.25154.objecter handle_osd_map got epochs [606,606] > 0 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 3 client.25154.objecter handle_osd_map decoding full epoch 606 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 20 client.25154.objecter dump_active .. 0 homeless 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fec5c03df30 0x7fec5c0403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 10 monclient: get_auth_request con 0x7fec5c03df30 auth_method 0 2024-09-11T03:05:05.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec727fc640 10 client.25154.objecter ms_handle_connect 0x7fec5c03df30 2024-09-11T03:05:05.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.057+0000 7fec79c7f640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fec5c03df30 0x7fec5c0403f0 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7fec64001750 tx=0x7fec64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:05.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.061+0000 7fec727fc640 1 -- 172.21.15.98:0/3564653517 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fec740a31d0 con 0x7fec74136200 2024-09-11T03:05:05.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.061+0000 7fec727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:05.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.061+0000 7fec727fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:05.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.297+0000 7fec7bee3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:05.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.297+0000 7fec7bee3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:05.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.297+0000 7fec7bee3640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:05.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.297+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec740725d0 con 0x7fec74136200 2024-09-11T03:05:05.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec727fc640 1 -- 172.21.15.98:0/3564653517 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3188 (secure 0 0 0) 0x7fec6c016130 con 0x7fec74136200 2024-09-11T03:05:05.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:05.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec727fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:05.303 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":95,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 10 librados: watch_flush enter 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 10 librados: watch_flush exit 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 20 client.25154.objecter shutdown clearing up homeless session... 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 10 client.25154.objecter successfully canceled tick 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fec5c03df30 msgr2=0x7fec5c0403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fec5c03df30 0x7fec5c0403f0 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7fec64001750 tx=0x7fec64009040 comp rx=0 tx=0).stop 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 10 monclient: shutdown 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fec74136200 msgr2=0x7fec740a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fec74136200 0x7fec740a0e60 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fec740720e0 tx=0x7fec6c0191c0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 shutdown_connections 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fec74136200 0x7fec740a0e60 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fec5c03df30 0x7fec5c0403f0 unknown :-1 s=CLOSED pgs=3653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fec7413ea00 0x7fec74135cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 --2- 172.21.15.98:0/3564653517 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fec74139b10 0x7fec74135780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 >> 172.21.15.98:0/3564653517 conn(0x7fec740ac740 msgr2=0x7fec7413f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 shutdown_connections 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 -- 172.21.15.98:0/3564653517 wait complete. 2024-09-11T03:05:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:05.301+0000 7fec7bee3640 1 librados: shutdown 2024-09-11T03:05:08.322 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient: get_monmap_and_config 2024-09-11T03:05:08.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient: build_initial_monmap 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient: monmap: 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:08.454843+0000 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:08.454843+0000 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:08.453 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:08.454 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 Processor -- start 2024-09-11T03:05:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 -- start start 2024-09-11T03:05:08.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient: init 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient: _add_conns ranks=[10,9,7,8,6,1,3,0,2,5,4] 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f13940a48b0 0x7f13940a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): picked mon.noname-f con 0x7f13940a48b0 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f139409faf0 0x7f13940a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): picked mon.noname-a con 0x7f139409faf0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13940a2a40 0x7f1394137670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): picked mon.noname-i con 0x7f13940a2a40 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f13940a27e0 con 0x7f139409faf0 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:08.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f13940a5710 con 0x7f13940a48b0 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f1394073f60 con 0x7f13940a2a40 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f139ae94640 10 monclient(hunting): authenticate will time out at 1862.335116s 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f139409faf0 0x7f13940a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f139409faf0 0x7f13940a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48012/0 (socket says 172.21.15.98:48012) 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 -- 172.21.15.98:0/3285329568 learned_addr learned my addr 172.21.15.98:0/3285329568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): get_auth_request con 0x7f139409faf0 auth_method 0 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1398c30640 1 --2- 172.21.15.98:0/3285329568 >> v2:172.21.15.190:3300/0 conn(0x7f13940a48b0 0x7f13940a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:08.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1398c30640 10 monclient(hunting): get_auth_request con 0x7f13940a48b0 auth_method 0 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1398c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1398c30640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1398c30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1399431640 1 --2- 172.21.15.98:0/3285329568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13940a2a40 0x7f1394137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient(hunting): handle_auth_done global_id 45225 payload 274 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 -- 172.21.15.98:0/3285329568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13940a2a40 msgr2=0x7f1394137670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 --2- 172.21.15.98:0/3285329568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13940a2a40 0x7f1394137670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 -- 172.21.15.98:0/3285329568 >> v2:172.21.15.190:3300/0 conn(0x7f13940a48b0 msgr2=0x7f13940a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 --2- 172.21.15.98:0/3285329568 >> v2:172.21.15.190:3300/0 conn(0x7f13940a48b0 0x7f13940a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 monclient: found mon.noname-a 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1393fff640 1 -- 172.21.15.98:0/3285329568 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1394077810 con 0x7f139409faf0 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1399431640 10 monclient: get_auth_request con 0x7f13940a2a40 auth_method 0 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1398c30640 1 --2- 172.21.15.98:0/3285329568 >> v2:172.21.15.190:3300/0 conn(0x7f13940a48b0 0x7f13940a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.449+0000 7f1399431640 1 --2- 172.21.15.98:0/3285329568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13940a2a40 0x7f1394137670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f1393fff640 1 --2- 172.21.15.98:0/3285329568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f139409faf0 0x7f13940a51d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1394070410 tx=0x7f137c01c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b2404c5c6fcea6 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 1 -- 172.21.15.98:0/3285329568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f137c0b1070 con 0x7f139409faf0 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: dump: 2024-09-11T03:05:08.462 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:08.463 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 1 -- 172.21.15.98:0/3285329568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137c0a4c90 con 0x7f139409faf0 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 5 monclient: authenticate success, global_id 45225 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 1 -- 172.21.15.98:0/3285329568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f137c0aac20 con 0x7f139409faf0 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:08.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 4 set_mon_vals no callback set 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f13937fe640 10 monclient: dump: 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:08.465 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 10 monclient: shutdown 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 -- 172.21.15.98:0/3285329568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f139409faf0 msgr2=0x7f13940a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 --2- 172.21.15.98:0/3285329568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f139409faf0 0x7f13940a51d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1394070410 tx=0x7f137c01c120 comp rx=0 tx=0).stop 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 -- 172.21.15.98:0/3285329568 shutdown_connections 2024-09-11T03:05:08.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 --2- 172.21.15.98:0/3285329568 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f13940a2a40 0x7f1394137670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 --2- 172.21.15.98:0/3285329568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f139409faf0 0x7f13940a51d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 --2- 172.21.15.98:0/3285329568 >> v2:172.21.15.190:3300/0 conn(0x7f13940a48b0 0x7f13940a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 -- 172.21.15.98:0/3285329568 >> 172.21.15.98:0/3285329568 conn(0x7f139406ed00 msgr2=0x7f139406f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 -- 172.21.15.98:0/3285329568 shutdown_connections 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 -- 172.21.15.98:0/3285329568 wait complete. 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 10 monclient: build_initial_monmap 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 10 monclient: monmap: 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:08.467 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 librados: starting msgr at 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.453+0000 7f139ae94640 1 librados: starting objecter 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 Processor -- start 2024-09-11T03:05:08.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 -- start start 2024-09-11T03:05:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 librados: setting wanted keys 2024-09-11T03:05:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 librados: calling monclient init 2024-09-11T03:05:08.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: init 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _add_conns ranks=[8,4,6,5,3,2,7,1,0] 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f139409faf0 0x7f1394136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): picked mon.noname-e con 0x7f139409faf0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13940a2a40 0x7f1394136860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): picked mon.noname-c con 0x7f13940a2a40 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13940a48b0 0x7f1394136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): picked mon.noname-d con 0x7f13940a48b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:08.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1394140e70 con 0x7f13940a2a40 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1394140cf0 con 0x7f13940a48b0 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1394140ff0 con 0x7f139409faf0 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient(hunting): authenticate will time out at 1862.340657s 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13940a2a40 0x7f1394136860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f139409faf0 0x7f1394136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13940a48b0 0x7f1394136da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13940a2a40 0x7f1394136860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59626/0 (socket says 172.21.15.98:59626) 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f139409faf0 0x7f1394136320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55912/0 (socket says 172.21.15.98:55912) 2024-09-11T03:05:08.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 1 -- 172.21.15.98:0/883538107 learned_addr learned my addr 172.21.15.98:0/883538107 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 10 monclient(hunting): get_auth_request con 0x7f13940a2a40 auth_method 0 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): get_auth_request con 0x7f13940a48b0 auth_method 0 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient(hunting): get_auth_request con 0x7f139409faf0 auth_method 0 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:08.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient(hunting): handle_auth_done global_id 84408 payload 824 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 -- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f139409faf0 msgr2=0x7f1394136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f139409faf0 0x7f1394136320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 -- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13940a2a40 msgr2=0x7f1394136860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13940a2a40 0x7f1394136860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient: _finish_hunting 0 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 monclient: found mon.noname-d 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 -- 172.21.15.98:0/883538107 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1394121e70 con 0x7f13940a48b0 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1393fff640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13940a2a40 0x7f1394136860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f139409faf0 0x7f1394136320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1399431640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13940a48b0 0x7f1394136da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f13940a2e50 tx=0x7f138c00a710 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 client.?.objecter ms_handle_connect 0x7f13940a48b0 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 1 -- 172.21.15.98:0/883538107 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f138c0158b0 con 0x7f13940a48b0 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:05:08.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: dump: 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:08.478 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: _finish_auth 0 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: _check_auth_tickets 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 1 -- 172.21.15.98:0/883538107 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138c015a90 con 0x7f13940a48b0 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 5 monclient: authenticate success, global_id 84408 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _renew_subs 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139409e620 con 0x7f13940a48b0 2024-09-11T03:05:08.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 1 -- 172.21.15.98:0/883538107 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f138c015c80 con 0x7f13940a48b0 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 monclient: dump: 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:08.480 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1392ffd640 4 set_mon_vals no callback set 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 client.84408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _renew_subs 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f139413d6c0 con 0x7f13940a48b0 2024-09-11T03:05:08.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 librados: init done 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1394137670 con 0x7f13940a48b0 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 1 -- 172.21.15.98:0/883538107 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f138c015440 con 0x7f13940a48b0 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f137403df40 0x7f1374040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 1 -- 172.21.15.98:0/883538107 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(609..609 src has 1..609) ==== 6416+0+0 (secure 0 0 0) 0x7f138c0500f0 con 0x7f13940a48b0 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 10 client.84408.objecter ms_dispatch 0x7f1394000ba0 osd_map(609..609 src has 1..609) 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 3 client.84408.objecter handle_osd_map got epochs [609,609] > 0 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 3 client.84408.objecter handle_osd_map decoding full epoch 609 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f13917fa640 20 client.84408.objecter dump_active .. 0 homeless 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f137403df40 0x7f1374040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:08.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.457+0000 7f1398c30640 10 monclient: get_auth_request con 0x7f137403df40 auth_method 0 2024-09-11T03:05:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.461+0000 7f13917fa640 10 client.84408.objecter ms_handle_connect 0x7f137403df40 2024-09-11T03:05:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.461+0000 7f1398c30640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f137403df40 0x7f1374040400 secure :-1 s=READY pgs=3655 cs=0 l=1 rev1=1 crypto rx=0x7f1384006e90 tx=0x7f1384008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.461+0000 7f13917fa640 1 -- 172.21.15.98:0/883538107 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f138c015e20 con 0x7f13940a48b0 2024-09-11T03:05:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.461+0000 7f13917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:08.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.461+0000 7f13917fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:08.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1394069120 con 0x7f13940a48b0 2024-09-11T03:05:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f13917fa640 1 -- 172.21.15.98:0/883538107 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3188 (secure 0 0 0) 0x7f138c00dc80 con 0x7f13940a48b0 2024-09-11T03:05:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f13917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:08.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f13917fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:08.720 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":98,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 10 librados: watch_flush enter 2024-09-11T03:05:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 10 librados: watch_flush exit 2024-09-11T03:05:08.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 20 client.84408.objecter shutdown clearing up homeless session... 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 10 client.84408.objecter successfully canceled tick 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f137403df40 msgr2=0x7f1374040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f137403df40 0x7f1374040400 secure :-1 s=READY pgs=3655 cs=0 l=1 rev1=1 crypto rx=0x7f1384006e90 tx=0x7f1384008040 comp rx=0 tx=0).stop 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 10 monclient: shutdown 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13940a48b0 msgr2=0x7f1394136da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13940a48b0 0x7f1394136da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f13940a2e50 tx=0x7f138c00a710 comp rx=0 tx=0).stop 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 shutdown_connections 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f137403df40 0x7f1374040400 unknown :-1 s=CLOSED pgs=3655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f13940a48b0 0x7f1394136da0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f13940a2a40 0x7f1394136860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 --2- 172.21.15.98:0/883538107 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f139409faf0 0x7f1394136320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 >> 172.21.15.98:0/883538107 conn(0x7f139406ed00 msgr2=0x7f13940a7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 shutdown_connections 2024-09-11T03:05:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 -- 172.21.15.98:0/883538107 wait complete. 2024-09-11T03:05:08.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:08.717+0000 7f139ae94640 1 librados: shutdown 2024-09-11T03:05:11.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:11.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient: get_monmap_and_config 2024-09-11T03:05:11.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient: build_initial_monmap 2024-09-11T03:05:11.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient: monmap: 2024-09-11T03:05:11.872 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:11.872 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:11.872 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:11.874597+0000 2024-09-11T03:05:11.872 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:11.874597+0000 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:11.873 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:11.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 Processor -- start 2024-09-11T03:05:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 -- start start 2024-09-11T03:05:11.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient: init 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient: _add_conns ranks=[10,9,7,1,5,3,4,6,0,8,2] 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe23806f6c0 0x7fe238072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): picked mon.noname-f con 0x7fe23806f6c0 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe23806faa0 0x7fe23806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): picked mon.noname-a con 0x7fe23806faa0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe2380703c0 0x7fe2381397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): picked mon.noname-i con 0x7fe2380703c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fe238071ff0 con 0x7fe23806faa0 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe2380a8750 con 0x7fe23806f6c0 2024-09-11T03:05:11.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe2380716c0 con 0x7fe2380703c0 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23fee8640 10 monclient(hunting): authenticate will time out at 1865.754860s 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe23806faa0 0x7fe23806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fe23806faa0 0x7fe23806fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48022/0 (socket says 172.21.15.98:48022) 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 -- 172.21.15.98:0/3296486722 learned_addr learned my addr 172.21.15.98:0/3296486722 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): get_auth_request con 0x7fe23806faa0 auth_method 0 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:11.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient(hunting): handle_auth_done global_id 45234 payload 274 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 -- 172.21.15.98:0/3296486722 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe2380703c0 msgr2=0x7fe2381397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 --2- 172.21.15.98:0/3296486722 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe2380703c0 0x7fe2381397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 -- 172.21.15.98:0/3296486722 >> v2:172.21.15.190:3300/0 conn(0x7fe23806f6c0 msgr2=0x7fe238072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23dc84640 1 --2- 172.21.15.98:0/3296486722 >> v2:172.21.15.190:3300/0 conn(0x7fe23806f6c0 0x7fe238072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 --2- 172.21.15.98:0/3296486722 >> v2:172.21.15.190:3300/0 conn(0x7fe23806f6c0 0x7fe238072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient: _finish_hunting 0 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 monclient: found mon.noname-a 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23d483640 1 -- 172.21.15.98:0/3296486722 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2380a64a0 con 0x7fe23806faa0 2024-09-11T03:05:11.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23dc84640 10 monclient: get_auth_request con 0x7fe23806f6c0 auth_method 0 2024-09-11T03:05:11.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.869+0000 7fe23dc84640 1 --2- 172.21.15.98:0/3296486722 >> v2:172.21.15.190:3300/0 conn(0x7fe23806f6c0 0x7fe238072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23d483640 1 --2- 172.21.15.98:0/3296486722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23806faa0 0x7fe23806fe80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe2380712e0 tx=0x7fe23401c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13dcdaccc1ce7c49 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:11.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 1 -- 172.21.15.98:0/3296486722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe2340ac020 con 0x7fe23806faa0 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: dump: 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:11.885 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: _finish_auth 0 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: _check_auth_tickets 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 5 monclient: authenticate success, global_id 45234 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 1 -- 172.21.15.98:0/3296486722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe234009110 con 0x7fe23806faa0 2024-09-11T03:05:11.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 4 set_mon_vals no callback set 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 1 -- 172.21.15.98:0/3296486722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe2340a6d80 con 0x7fe23806faa0 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23cc82640 10 monclient: dump: 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:11.887 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 10 monclient: shutdown 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 -- 172.21.15.98:0/3296486722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23806faa0 msgr2=0x7fe23806fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:11.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 --2- 172.21.15.98:0/3296486722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23806faa0 0x7fe23806fe80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe2380712e0 tx=0x7fe23401c5f0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 -- 172.21.15.98:0/3296486722 shutdown_connections 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 --2- 172.21.15.98:0/3296486722 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe2380703c0 0x7fe2381397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 --2- 172.21.15.98:0/3296486722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe23806faa0 0x7fe23806fe80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 --2- 172.21.15.98:0/3296486722 >> v2:172.21.15.190:3300/0 conn(0x7fe23806f6c0 0x7fe238072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 -- 172.21.15.98:0/3296486722 >> 172.21.15.98:0/3296486722 conn(0x7fe2380acac0 msgr2=0x7fe2380a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 -- 172.21.15.98:0/3296486722 shutdown_connections 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 -- 172.21.15.98:0/3296486722 wait complete. 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 10 monclient: build_initial_monmap 2024-09-11T03:05:11.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 10 monclient: monmap: 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:11.890 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 librados: starting msgr at 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 librados: starting objecter 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 Processor -- start 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 -- start start 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 librados: setting wanted keys 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 1 librados: calling monclient init 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.873+0000 7fe23fee8640 10 monclient: init 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _add_conns ranks=[4,3,8,0,5,2,6,1,7] 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe23806f6c0 0x7fe238136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:11.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): picked mon.noname-c con 0x7fe23806f6c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe23806faa0 0x7fe2381366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): picked mon.noname-g con 0x7fe23806faa0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe2380703c0 0x7fe2380a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): picked mon.noname-e con 0x7fe2380703c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe238143840 con 0x7fe23806faa0 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe2381436c0 con 0x7fe23806f6c0 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe2381439c0 con 0x7fe2380703c0 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:11.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient(hunting): authenticate will time out at 1865.760224s 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe23806f6c0 0x7fe238136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe23806f6c0 0x7fe238136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59642/0 (socket says 172.21.15.98:59642) 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 -- 172.21.15.98:0/857322019 learned_addr learned my addr 172.21.15.98:0/857322019 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): get_auth_request con 0x7fe23806f6c0 auth_method 0 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe2380703c0 0x7fe2380a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 10 monclient(hunting): get_auth_request con 0x7fe2380703c0 auth_method 0 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:11.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe23806faa0 0x7fe2381366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 10 monclient(hunting): get_auth_request con 0x7fe23806faa0 auth_method 0 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient(hunting): handle_auth_done global_id 24655 payload 824 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 -- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe2380703c0 msgr2=0x7fe2380a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe2380703c0 0x7fe2380a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 -- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe23806faa0 msgr2=0x7fe2381366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe23806faa0 0x7fe2381366c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient: _finish_hunting 0 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 monclient: found mon.noname-c 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 -- 172.21.15.98:0/857322019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2380a5620 con 0x7fe23806f6c0 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23e485640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe2380703c0 0x7fe2380a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23dc84640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe23806f6c0 0x7fe238136180 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe22c00c7c0 tx=0x7fe22c00cc80 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 client.?.objecter ms_handle_connect 0x7fe23806f6c0 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 1 -- 172.21.15.98:0/857322019 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe22c007ca0 con 0x7fe23806f6c0 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:11.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: dump: 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:11.896 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 5 monclient: authenticate success, global_id 24655 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _renew_subs 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 1 -- 172.21.15.98:0/857322019 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe22c0113f0 con 0x7fe23806f6c0 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe238140290 con 0x7fe23806f6c0 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:11.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 1 -- 172.21.15.98:0/857322019 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe22c011a10 con 0x7fe23806f6c0 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22bfff640 4 set_mon_vals no callback set 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 client.24655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 monclient: dump: 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _renew_subs 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2380a5060 con 0x7fe23806f6c0 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 librados: init done 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2380a5860 con 0x7fe23806f6c0 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 1 -- 172.21.15.98:0/857322019 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe22c011bb0 con 0x7fe23806f6c0 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe21c03df40 0x7fe21c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 1 -- 172.21.15.98:0/857322019 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(613..613 src has 1..613) ==== 6797+0+0 (secure 0 0 0) 0x7fe22c04fbe0 con 0x7fe23806f6c0 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 client.24655.objecter ms_dispatch 0x7fe238000ba0 osd_map(613..613 src has 1..613) 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 3 client.24655.objecter handle_osd_map got epochs [613,613] > 0 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 3 client.24655.objecter handle_osd_map decoding full epoch 613 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 20 client.24655.objecter dump_active .. 0 homeless 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe21c03df40 0x7fe21c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 10 monclient: get_auth_request con 0x7fe21c03df40 auth_method 0 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe22a7fc640 10 client.24655.objecter ms_handle_connect 0x7fe21c03df40 2024-09-11T03:05:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.877+0000 7fe23d483640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe21c03df40 0x7fe21c040400 secure :-1 s=READY pgs=3656 cs=0 l=1 rev1=1 crypto rx=0x7fe2340ad000 tx=0x7fe2340a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.881+0000 7fe22a7fc640 1 -- 172.21.15.98:0/857322019 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe22c015020 con 0x7fe23806f6c0 2024-09-11T03:05:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.881+0000 7fe22a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:11.881+0000 7fe22a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.117+0000 7fe23fee8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.117+0000 7fe23fee8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.117+0000 7fe23fee8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:12.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.117+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe23806fe80 con 0x7fe23806f6c0 2024-09-11T03:05:12.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.117+0000 7fe22a7fc640 1 -- 172.21.15.98:0/857322019 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3189 (secure 0 0 0) 0x7fe22c018d60 con 0x7fe23806f6c0 2024-09-11T03:05:12.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.117+0000 7fe22a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:12.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.117+0000 7fe22a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:12.120 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":102,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 10 librados: watch_flush enter 2024-09-11T03:05:12.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 10 librados: watch_flush exit 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 20 client.24655.objecter shutdown clearing up homeless session... 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 10 client.24655.objecter successfully canceled tick 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe21c03df40 msgr2=0x7fe21c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe21c03df40 0x7fe21c040400 secure :-1 s=READY pgs=3656 cs=0 l=1 rev1=1 crypto rx=0x7fe2340ad000 tx=0x7fe2340a7000 comp rx=0 tx=0).stop 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 10 monclient: shutdown 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe23806f6c0 msgr2=0x7fe238136180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe23806f6c0 0x7fe238136180 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe22c00c7c0 tx=0x7fe22c00cc80 comp rx=0 tx=0).stop 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 shutdown_connections 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe21c03df40 0x7fe21c040400 unknown :-1 s=CLOSED pgs=3656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe2380703c0 0x7fe2380a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:12.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe23806faa0 0x7fe2381366c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 --2- 172.21.15.98:0/857322019 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe23806f6c0 0x7fe238136180 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 >> 172.21.15.98:0/857322019 conn(0x7fe2380acac0 msgr2=0x7fe2380a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 shutdown_connections 2024-09-11T03:05:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 -- 172.21.15.98:0/857322019 wait complete. 2024-09-11T03:05:12.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:12.121+0000 7fe23fee8640 1 librados: shutdown 2024-09-11T03:05:15.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:15.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient: get_monmap_and_config 2024-09-11T03:05:15.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient: build_initial_monmap 2024-09-11T03:05:15.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient: monmap: 2024-09-11T03:05:15.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:15.274674+0000 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:15.274674+0000 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:15.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:15.274 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:15.274 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:15.274 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:15.274 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:15.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 Processor -- start 2024-09-11T03:05:15.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 -- start start 2024-09-11T03:05:15.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient: init 2024-09-11T03:05:15.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient: _add_conns ranks=[6,2,9,0,5,4,8,7,1,3,10] 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806f6c0 0x7f89b8072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): picked mon.noname-d con 0x7f89b806f6c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 0x7f89b806fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): picked mon.noname-b con 0x7f89b806faa0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89b80703c0 0x7f89b81397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): picked mon.noname-a con 0x7f89b80703c0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f89b8071ff0 con 0x7f89b80703c0 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f89b80a8750 con 0x7f89b806faa0 2024-09-11T03:05:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f89b80716c0 con 0x7f89b806f6c0 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89be94e640 10 monclient(hunting): authenticate will time out at 1869.155205s 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89b80703c0 0x7f89b81397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806f6c0 0x7f89b8072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806f6c0 0x7f89b8072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:60996/0 (socket says 172.21.15.98:60996) 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f89b80703c0 0x7f89b81397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48040/0 (socket says 172.21.15.98:48040) 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 -- 172.21.15.98:0/509734711 learned_addr learned my addr 172.21.15.98:0/509734711 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): get_auth_request con 0x7f89b806f6c0 auth_method 0 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:15.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 10 monclient(hunting): get_auth_request con 0x7f89b80703c0 auth_method 0 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 0x7f89b806fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 10 monclient(hunting): get_auth_request con 0x7f89b806faa0 auth_method 0 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:15.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 1 -- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 msgr2=0x7f89b806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 1 -- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 msgr2=0x7f89b806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 0x7f89b806fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b77fe640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 0x7f89b806fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient(hunting): handle_auth_done global_id 84417 payload 274 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 -- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 msgr2=0x7f89b806fe80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 0x7f89b806fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 -- 172.21.15.98:0/509734711 >> v2:172.21.15.98:3300/0 conn(0x7f89b80703c0 msgr2=0x7f89b81397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 --2- 172.21.15.98:0/509734711 >> v2:172.21.15.98:3300/0 conn(0x7f89b80703c0 0x7f89b81397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:15.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 monclient: found mon.noname-d 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89b7fff640 1 -- 172.21.15.98:0/509734711 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b80a64a0 con 0x7f89b806f6c0 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.269+0000 7f89bceeb640 1 --2- 172.21.15.98:0/509734711 >> v2:172.21.15.98:3300/0 conn(0x7f89b80703c0 0x7f89b81397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b7fff640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806f6c0 0x7f89b8072940 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009870 tx=0x7f89a801c160 comp rx=0 tx=0).ready entity=mon.6 client_cookie=21727b11c0122d2a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 1 -- 172.21.15.98:0/509734711 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89a80b0070 con 0x7f89b806f6c0 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: dump: 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:15.288 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:15.289 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 1 -- 172.21.15.98:0/509734711 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a80aa810 con 0x7f89b806f6c0 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 5 monclient: authenticate success, global_id 84417 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 1 -- 172.21.15.98:0/509734711 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89a80a9300 con 0x7f89b806f6c0 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89b6ffd640 10 monclient: dump: 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:15.291 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 4 set_mon_vals no callback set 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 10 monclient: shutdown 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 -- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806f6c0 msgr2=0x7f89b8072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806f6c0 0x7f89b8072940 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009870 tx=0x7f89a801c160 comp rx=0 tx=0).stop 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 -- 172.21.15.98:0/509734711 shutdown_connections 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 --2- 172.21.15.98:0/509734711 >> v2:172.21.15.98:3300/0 conn(0x7f89b80703c0 0x7f89b81397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89b806faa0 0x7f89b806fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 --2- 172.21.15.98:0/509734711 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806f6c0 0x7f89b8072940 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 -- 172.21.15.98:0/509734711 >> 172.21.15.98:0/509734711 conn(0x7f89b80acac0 msgr2=0x7f89b80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 -- 172.21.15.98:0/509734711 shutdown_connections 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 -- 172.21.15.98:0/509734711 wait complete. 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 10 monclient: build_initial_monmap 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 10 monclient: monmap: 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:15.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 librados: starting msgr at 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 librados: starting objecter 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 Processor -- start 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 -- start start 2024-09-11T03:05:15.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 librados: setting wanted keys 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 1 librados: calling monclient init 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.273+0000 7f89be94e640 10 monclient: init 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _add_conns ranks=[4,6,0,7,3,5,2,8,1] 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89b806f6c0 0x7f89b81d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): picked mon.noname-c con 0x7f89b806f6c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806faa0 0x7f89b8138df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): picked mon.noname-d con 0x7f89b806faa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b80703c0 0x7f89b813dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): picked mon.noname-a con 0x7f89b80703c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89b8143840 con 0x7f89b80703c0 2024-09-11T03:05:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f89b81436c0 con 0x7f89b806f6c0 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f89b81439c0 con 0x7f89b806faa0 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient(hunting): authenticate will time out at 1869.160511s 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b80703c0 0x7f89b813dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89b806f6c0 0x7f89b81d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89b806f6c0 0x7f89b81d19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59664/0 (socket says 172.21.15.98:59664) 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b80703c0 0x7f89b813dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48046/0 (socket says 172.21.15.98:48046) 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 -- 172.21.15.98:0/1724756619 learned_addr learned my addr 172.21.15.98:0/1724756619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): get_auth_request con 0x7f89b806f6c0 auth_method 0 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806faa0 0x7f89b8138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 10 monclient(hunting): get_auth_request con 0x7f89b80703c0 auth_method 0 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient(hunting): get_auth_request con 0x7f89b806faa0 auth_method 0 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient(hunting): handle_auth_done global_id 24664 payload 824 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 -- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806faa0 msgr2=0x7f89b8138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806faa0 0x7f89b8138df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 -- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b80703c0 msgr2=0x7f89b813dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b80703c0 0x7f89b813dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 monclient: found mon.noname-c 2024-09-11T03:05:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 -- 172.21.15.98:0/1724756619 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b8124790 con 0x7f89b806f6c0 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89bceeb640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b80703c0 0x7f89b813dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b7fff640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89b806f6c0 0x7f89b81d19c0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f89a80ae040 tx=0x7f89a8018f00 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 client.?.objecter ms_handle_connect 0x7f89b806f6c0 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 1 -- 172.21.15.98:0/1724756619 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89a80b9070 con 0x7f89b806f6c0 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: dump: 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: _finish_auth 0 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 1 -- 172.21.15.98:0/1724756619 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89a80044b0 con 0x7f89b806f6c0 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 1 -- 172.21.15.98:0/1724756619 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f89a80b0040 con 0x7f89b806f6c0 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 monclient: dump: 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b67fc640 4 set_mon_vals no callback set 2024-09-11T03:05:15.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 5 monclient: authenticate success, global_id 24664 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _renew_subs 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b813fab0 con 0x7f89b806f6c0 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 client.24664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _renew_subs 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b81241d0 con 0x7f89b806f6c0 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 librados: init done 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b806e820 con 0x7f89b806f6c0 2024-09-11T03:05:15.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 1 -- 172.21.15.98:0/1724756619 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f89a8004040 con 0x7f89b806f6c0 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f89a403df70 0x7f89a4040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 1 -- 172.21.15.98:0/1724756619 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(616..616 src has 1..616) ==== 6789+0+0 (secure 0 0 0) 0x7f89a80e8bc0 con 0x7f89b806f6c0 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 10 client.24664.objecter ms_dispatch 0x7f89b8000ba0 osd_map(616..616 src has 1..616) 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 3 client.24664.objecter handle_osd_map got epochs [616,616] > 0 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 3 client.24664.objecter handle_osd_map decoding full epoch 616 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b4ff9640 20 client.24664.objecter dump_active .. 0 homeless 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f89a403df70 0x7f89a4040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.277+0000 7f89b77fe640 10 monclient: get_auth_request con 0x7f89a403df70 auth_method 0 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.281+0000 7f89b4ff9640 10 client.24664.objecter ms_handle_connect 0x7f89a403df70 2024-09-11T03:05:15.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.281+0000 7f89b77fe640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f89a403df70 0x7f89a4040430 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 crypto rx=0x7f89b813f720 tx=0x7f89ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:15.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.281+0000 7f89b4ff9640 1 -- 172.21.15.98:0/1724756619 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f89b806e820 con 0x7f89b806f6c0 2024-09-11T03:05:15.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.281+0000 7f89b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:15.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.281+0000 7f89b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.525+0000 7f89be94e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.525+0000 7f89be94e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.525+0000 7f89be94e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.525+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f89b806fe80 con 0x7f89b806f6c0 2024-09-11T03:05:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.525+0000 7f89b4ff9640 1 -- 172.21.15.98:0/1724756619 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3189 (secure 0 0 0) 0x7f89a8016130 con 0x7f89b806f6c0 2024-09-11T03:05:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.525+0000 7f89b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.525+0000 7f89b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:15.529 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":105,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 10 librados: watch_flush enter 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 10 librados: watch_flush exit 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 20 client.24664.objecter shutdown clearing up homeless session... 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 10 client.24664.objecter successfully canceled tick 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f89a403df70 msgr2=0x7f89a4040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f89a403df70 0x7f89a4040430 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 crypto rx=0x7f89b813f720 tx=0x7f89ac008040 comp rx=0 tx=0).stop 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 10 monclient: shutdown 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89b806f6c0 msgr2=0x7f89b81d19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89b806f6c0 0x7f89b81d19c0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f89a80ae040 tx=0x7f89a8018f00 comp rx=0 tx=0).stop 2024-09-11T03:05:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 shutdown_connections 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f89a403df70 0x7f89a4040430 unknown :-1 s=CLOSED pgs=3658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89b80703c0 0x7f89b813dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f89b806faa0 0x7f89b8138df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 --2- 172.21.15.98:0/1724756619 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89b806f6c0 0x7f89b81d19c0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 >> 172.21.15.98:0/1724756619 conn(0x7f89b80acac0 msgr2=0x7f89b81400c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 shutdown_connections 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 -- 172.21.15.98:0/1724756619 wait complete. 2024-09-11T03:05:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:15.529+0000 7f89be94e640 1 librados: shutdown 2024-09-11T03:05:18.550 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:18.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient: get_monmap_and_config 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient: build_initial_monmap 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient: monmap: 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:18.675055+0000 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:18.675055+0000 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:18.674 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 1 Processor -- start 2024-09-11T03:05:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 1 -- start start 2024-09-11T03:05:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient: init 2024-09-11T03:05:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient: _add_conns ranks=[6,1,0,2,5,4,9,7,8,10,3] 2024-09-11T03:05:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48b006f650 0x7f48b0072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient(hunting): picked mon.noname-d con 0x7f48b006f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f48b00a8380 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48b0070140 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48b0071a80 con 0x7f48b0070140 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48b0071d60 con 0x7f48b00a8380 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f48b00a8800 con 0x7f48b006f650 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.669+0000 7f48b7c18640 10 monclient(hunting): authenticate will time out at 1872.555907s 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48b006f650 0x7f48b0072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48b006f650 0x7f48b0072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49194/0 (socket says 172.21.15.98:49194) 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b61b5640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f48b0070140 0x7f48b0140c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:52762/0 (socket says 172.21.15.98:52762) 2024-09-11T03:05:18.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 1 -- 172.21.15.98:0/3846088875 learned_addr learned my addr 172.21.15.98:0/3846088875 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): get_auth_request con 0x7f48b006f650 auth_method 0 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b49b2640 1 -- 172.21.15.98:0/3846088875 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 364718959 0 0) 0x7f48b0071a80 con 0x7f48b0070140 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient(hunting): handle_auth_done global_id 84435 payload 274 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 1 -- 172.21.15.98:0/3846088875 >> v1:172.21.15.190:6789/0 conn(0x7f48b00a8380 legacy=0x7f48b006fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 1 -- 172.21.15.98:0/3846088875 >> v1:172.21.15.98:6789/0 conn(0x7f48b0070140 legacy=0x7f48b0140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient: _finish_hunting 0 2024-09-11T03:05:18.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 1 monclient: found mon.noname-d 2024-09-11T03:05:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:05:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b59b4640 1 -- 172.21.15.98:0/3846088875 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b00714c0 con 0x7f48b006f650 2024-09-11T03:05:18.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.673+0000 7f48b49b2640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:05:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b59b4640 1 --2- 172.21.15.98:0/3846088875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48b006f650 0x7f48b0072640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0098a0 tx=0x7f48ac01c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8517aef197607a5f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 1 -- 172.21.15.98:0/3846088875 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48ac0ab8d0 con 0x7f48b006f650 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: dump: 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: _finish_auth 0 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: _check_auth_tickets 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 1 -- 172.21.15.98:0/3846088875 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48ac0b0070 con 0x7f48b006f650 2024-09-11T03:05:18.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 5 monclient: authenticate success, global_id 84435 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 1 -- 172.21.15.98:0/3846088875 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48ac0aa3c0 con 0x7f48b006f650 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b49b2640 10 monclient: dump: 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:18.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 4 set_mon_vals no callback set 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: shutdown 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:18.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- 172.21.15.98:0/3846088875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48b006f650 msgr2=0x7f48b0072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 --2- 172.21.15.98:0/3846088875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48b006f650 0x7f48b0072640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0098a0 tx=0x7f48ac01c440 comp rx=0 tx=0).stop 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- 172.21.15.98:0/3846088875 shutdown_connections 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 --2- 172.21.15.98:0/3846088875 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f48b006f650 0x7f48b0072640 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- 172.21.15.98:0/3846088875 >> 172.21.15.98:0/3846088875 conn(0x7f48b00ac8e0 msgr2=0x7f48b00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- 172.21.15.98:0/3846088875 shutdown_connections 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- 172.21.15.98:0/3846088875 wait complete. 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: build_initial_monmap 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: monmap: 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:18.689 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 librados: starting msgr at 2024-09-11T03:05:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 librados: starting objecter 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 Processor -- start 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- start start 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 librados: setting wanted keys 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 librados: calling monclient init 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: init 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: _add_conns ranks=[7,8,2,6,5,4,1,0,3] 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f48b006f650 0x7f48b0128af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): picked mon.noname-i con 0x7f48b006f650 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48b0070140 0x7f48b01d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): picked mon.noname-e con 0x7f48b0070140 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 0x7f48b0126520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): picked mon.noname-b con 0x7f48b00a8380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f48b00517e0 con 0x7f48b00a8380 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f48b0144d70 con 0x7f48b006f650 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f48b0144ef0 con 0x7f48b0070140 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient(hunting): authenticate will time out at 1872.562790s 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 0x7f48b0126520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48b0070140 0x7f48b01d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 0x7f48b0126520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:46780/0 (socket says 172.21.15.98:46780) 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 1 -- 172.21.15.98:0/1102281739 learned_addr learned my addr 172.21.15.98:0/1102281739 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48b0070140 0x7f48b01d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:40526/0 (socket says 172.21.15.98:40526) 2024-09-11T03:05:18.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): get_auth_request con 0x7f48b0070140 auth_method 0 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 10 monclient(hunting): get_auth_request con 0x7f48b00a8380 auth_method 0 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:18.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 1 -- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 msgr2=0x7f48b0126520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 1 -- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 msgr2=0x7f48b0126520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 0x7f48b0126520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b61b5640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 0x7f48b0126520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b59b4640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f48b006f650 0x7f48b0128af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b59b4640 10 monclient(hunting): get_auth_request con 0x7f48b006f650 auth_method 0 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b59b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b59b4640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b59b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient(hunting): handle_auth_done global_id 25181 payload 824 2024-09-11T03:05:18.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 -- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f48b006f650 msgr2=0x7f48b0128af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f48b006f650 0x7f48b0128af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 -- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 msgr2=0x7f48b0126520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 0x7f48b0126520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient: _finish_hunting 0 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 monclient: found mon.noname-e 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 -- 172.21.15.98:0/1102281739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b012a010 con 0x7f48b0070140 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b51b3640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48b0070140 0x7f48b01d13b0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c3c0 tx=0x7f48a000c880 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 client.?.objecter ms_handle_connect 0x7f48b0070140 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 1 -- 172.21.15.98:0/1102281739 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48a0017070 con 0x7f48b0070140 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:18.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: dump: 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:18.696 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 5 monclient: authenticate success, global_id 25181 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 1 -- 172.21.15.98:0/1102281739 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48a0003600 con 0x7f48b0070140 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 1 -- 172.21.15.98:0/1102281739 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48a0003c20 con 0x7f48b0070140 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: _renew_subs 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b012a2f0 con 0x7f48b0070140 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a77fe640 4 set_mon_vals no callback set 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 client.25181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48a67fc640 10 monclient: dump: 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: _renew_subs 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b0129480 con 0x7f48b0070140 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.677+0000 7f48b7c18640 1 librados: init done 2024-09-11T03:05:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48b7c18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48b7c18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48b7c18640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48b0072640 con 0x7f48b0070140 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 1 -- 172.21.15.98:0/1102281739 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f48a00037b0 con 0x7f48b0070140 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f489403df40 0x7f4894040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 1 -- 172.21.15.98:0/1102281739 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(620..620 src has 1..620) ==== 6774+0+0 (secure 0 0 0) 0x7f48a004eac0 con 0x7f48b0070140 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 10 client.25181.objecter ms_dispatch 0x7f48b0000ba0 osd_map(620..620 src has 1..620) 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 3 client.25181.objecter handle_osd_map got epochs [620,620] > 0 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 3 client.25181.objecter handle_osd_map decoding full epoch 620 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 20 client.25181.objecter dump_active .. 0 homeless 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48b59b4640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f489403df40 0x7f4894040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48b59b4640 10 monclient: get_auth_request con 0x7f489403df40 auth_method 0 2024-09-11T03:05:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48a67fc640 10 client.25181.objecter ms_handle_connect 0x7f489403df40 2024-09-11T03:05:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.681+0000 7f48b59b4640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f489403df40 0x7f4894040400 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0af000 tx=0x7f48ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.685+0000 7f48a67fc640 1 -- 172.21.15.98:0/1102281739 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f48a0007020 con 0x7f48b0070140 2024-09-11T03:05:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.685+0000 7f48a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.685+0000 7f48a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:18.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.929+0000 7f48b7c18640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:18.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.929+0000 7f48b7c18640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:18.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.929+0000 7f48b7c18640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:18.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.929+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f48b0127f90 con 0x7f48b0070140 2024-09-11T03:05:18.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.929+0000 7f48a67fc640 1 -- 172.21.15.98:0/1102281739 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3189 (secure 0 0 0) 0x7f48a00073a0 con 0x7f48b0070140 2024-09-11T03:05:18.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.929+0000 7f48a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:18.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.929+0000 7f48a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:18.934 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":109,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:18.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 10 librados: watch_flush enter 2024-09-11T03:05:18.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 10 librados: watch_flush exit 2024-09-11T03:05:18.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 20 client.25181.objecter shutdown clearing up homeless session... 2024-09-11T03:05:18.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 10 client.25181.objecter successfully canceled tick 2024-09-11T03:05:18.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f489403df40 msgr2=0x7f4894040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f489403df40 0x7f4894040400 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7f48ac0af000 tx=0x7f48ac009040 comp rx=0 tx=0).stop 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 10 monclient: shutdown 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48b0070140 msgr2=0x7f48b01d13b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48b0070140 0x7f48b01d13b0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c3c0 tx=0x7f48a000c880 comp rx=0 tx=0).stop 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 shutdown_connections 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f489403df40 0x7f4894040400 unknown :-1 s=CLOSED pgs=3659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f48b00a8380 0x7f48b0126520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f48b0070140 0x7f48b01d13b0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 --2- 172.21.15.98:0/1102281739 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f48b006f650 0x7f48b0128af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 >> 172.21.15.98:0/1102281739 conn(0x7f48b00ac8e0 msgr2=0x7f48b00a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:18.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 shutdown_connections 2024-09-11T03:05:18.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 -- 172.21.15.98:0/1102281739 wait complete. 2024-09-11T03:05:18.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:18.933+0000 7f48b7c18640 1 librados: shutdown 2024-09-11T03:05:21.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.077+0000 7f43dd60e640 10 monclient: get_monmap_and_config 2024-09-11T03:05:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.077+0000 7f43dd60e640 10 monclient: build_initial_monmap 2024-09-11T03:05:22.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.077+0000 7f43dd60e640 10 monclient: monmap: 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:22.084680+0000 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:22.084680+0000 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:22.083 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:22.084 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:22.084 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:22.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 1 Processor -- start 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 1 -- start start 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient: init 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient: _add_conns ranks=[8,0,9,1,5,10,3,7,2,6,4] 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43d806f6c0 0x7f43d8072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient(hunting): picked mon.noname-e con 0x7f43d806f6c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43d806faa0 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43d813ec80 0x7f43d8141070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient(hunting): picked mon.noname-a con 0x7f43d813ec80 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:22.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d80a8750 con 0x7f43d806faa0 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f43d8071ff0 con 0x7f43d813ec80 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f43d80716c0 con 0x7f43d806f6c0 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient(hunting): authenticate will time out at 1875.964911s 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43d806f6c0 0x7f43d8072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43d813ec80 0x7f43d8141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f43d813ec80 0x7f43d8141070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37324/0 (socket says 172.21.15.98:37324) 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43d806f6c0 0x7f43d8072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:40546/0 (socket says 172.21.15.98:40546) 2024-09-11T03:05:22.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 -- 172.21.15.98:0/3153321282 learned_addr learned my addr 172.21.15.98:0/3153321282 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): get_auth_request con 0x7f43d813ec80 auth_method 0 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 10 monclient(hunting): get_auth_request con 0x7f43d806f6c0 auth_method 0 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:22.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient(hunting): handle_auth_done global_id 45270 payload 274 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 -- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43d806f6c0 msgr2=0x7f43d8072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 --2- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43d806f6c0 0x7f43d8072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 -- 172.21.15.98:0/3153321282 >> v1:172.21.15.98:6789/0 conn(0x7f43d806faa0 legacy=0x7f43d806fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient: _finish_hunting 0 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 monclient: found mon.noname-a 2024-09-11T03:05:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 1 -- 172.21.15.98:0/3153321282 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713329121 0 0) 0x7f43d80a8750 con 0x7f43d806faa0 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 -- 172.21.15.98:0/3153321282 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d80a64a0 con 0x7f43d813ec80 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d6ffd640 1 --2- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43d806f6c0 0x7f43d8072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d77fe640 1 --2- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d813ec80 0x7f43d8141070 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f43d0009e50 tx=0x7f43d001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8162947b587c2cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 1 -- 172.21.15.98:0/3153321282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43d00090d0 con 0x7f43d813ec80 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: dump: 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 1 -- 172.21.15.98:0/3153321282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d00a5c40 con 0x7f43d813ec80 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 5 monclient: authenticate success, global_id 45270 2024-09-11T03:05:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 1 -- 172.21.15.98:0/3153321282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43d00a5df0 con 0x7f43d813ec80 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43dd60e640 4 set_mon_vals no callback set 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.081+0000 7f43d67fc640 10 monclient: dump: 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:22.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: shutdown 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d813ec80 msgr2=0x7f43d8141070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 --2- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d813ec80 0x7f43d8141070 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f43d0009e50 tx=0x7f43d001c8f0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/3153321282 shutdown_connections 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 --2- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d813ec80 0x7f43d8141070 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 --2- 172.21.15.98:0/3153321282 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f43d806f6c0 0x7f43d8072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/3153321282 >> 172.21.15.98:0/3153321282 conn(0x7f43d80acac0 msgr2=0x7f43d80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/3153321282 shutdown_connections 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/3153321282 wait complete. 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: build_initial_monmap 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: monmap: 2024-09-11T03:05:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:22.104 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 librados: starting msgr at 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 librados: starting objecter 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 Processor -- start 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- start start 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 librados: setting wanted keys 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 librados: calling monclient init 2024-09-11T03:05:22.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: init 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _add_conns ranks=[0,1,6,2,4,5,7,8,3] 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d806f6c0 0x7f43d81235f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): picked mon.noname-a con 0x7f43d806f6c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d806faa0 0x7f43d8123b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): picked mon.noname-f con 0x7f43d806faa0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43d813ec80 0x7f43d8124070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): picked mon.noname-d con 0x7f43d813ec80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8144160 con 0x7f43d806f6c0 2024-09-11T03:05:22.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8143fe0 con 0x7f43d806faa0 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f43d81442e0 con 0x7f43d813ec80 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient(hunting): authenticate will time out at 1875.970012s 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d806f6c0 0x7f43d81235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d806f6c0 0x7f43d81235f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37336/0 (socket says 172.21.15.98:37336) 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43d813ec80 0x7f43d8124070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 -- 172.21.15.98:0/4284596250 learned_addr learned my addr 172.21.15.98:0/4284596250 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): get_auth_request con 0x7f43d806f6c0 auth_method 0 2024-09-11T03:05:22.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 10 monclient(hunting): get_auth_request con 0x7f43d813ec80 auth_method 0 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:22.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient(hunting): handle_auth_done global_id 45288 payload 824 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cffff640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d806faa0 0x7f43d8123b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 -- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43d813ec80 msgr2=0x7f43d8124070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43d813ec80 0x7f43d8124070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 -- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d806faa0 msgr2=0x7f43d8123b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d806faa0 0x7f43d8123b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 monclient: found mon.noname-a 2024-09-11T03:05:22.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 -- 172.21.15.98:0/4284596250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d81263d0 con 0x7f43d806f6c0 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cffff640 10 monclient: get_auth_request con 0x7f43d806faa0 auth_method 0 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d77fe640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43d813ec80 0x7f43d8124070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cffff640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d806faa0 0x7f43d8123b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d6ffd640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d806f6c0 0x7f43d81235f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f43c800ebe0 tx=0x7f43c800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 client.?.objecter ms_handle_connect 0x7f43d806f6c0 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 1 -- 172.21.15.98:0/4284596250 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43c8017070 con 0x7f43d806f6c0 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:22.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: dump: 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:22.111 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:22.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 1 -- 172.21.15.98:0/4284596250 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c8013470 con 0x7f43d806f6c0 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 5 monclient: authenticate success, global_id 45288 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _renew_subs 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d8127240 con 0x7f43d806f6c0 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 1 -- 172.21.15.98:0/4284596250 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43c8013a90 con 0x7f43d806f6c0 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 10 monclient: dump: 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:22.113 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:22.114 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43d5ffb640 4 set_mon_vals no callback set 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 client.45288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _renew_subs 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d80aa2b0 con 0x7f43d806f6c0 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 librados: init done 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:22.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d8072940 con 0x7f43d806f6c0 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.085+0000 7f43cf7fe640 1 -- 172.21.15.98:0/4284596250 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f43c8013c30 con 0x7f43d806f6c0 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cf7fe640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f43b803df40 0x7f43b8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cf7fe640 1 -- 172.21.15.98:0/4284596250 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(621..621 src has 1..621) ==== 6774+0+0 (secure 0 0 0) 0x7f43c800f040 con 0x7f43d806f6c0 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cf7fe640 10 client.45288.objecter ms_dispatch 0x7f43d8000ba0 osd_map(621..621 src has 1..621) 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cf7fe640 3 client.45288.objecter handle_osd_map got epochs [621,621] > 0 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cf7fe640 3 client.45288.objecter handle_osd_map decoding full epoch 621 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cf7fe640 20 client.45288.objecter dump_active .. 0 homeless 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cffff640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f43b803df40 0x7f43b8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:22.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cffff640 10 monclient: get_auth_request con 0x7f43b803df40 auth_method 0 2024-09-11T03:05:22.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cffff640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f43b803df40 0x7f43b8040400 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f43c0007fd0 tx=0x7f43c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:22.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.089+0000 7f43cf7fe640 10 client.45288.objecter ms_handle_connect 0x7f43b803df40 2024-09-11T03:05:22.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.093+0000 7f43cf7fe640 1 -- 172.21.15.98:0/4284596250 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f43c8021070 con 0x7f43d806f6c0 2024-09-11T03:05:22.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.093+0000 7f43cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:22.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.093+0000 7f43cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:22.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f43d806e870 con 0x7f43d806f6c0 2024-09-11T03:05:22.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43cf7fe640 1 -- 172.21.15.98:0/4284596250 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3189 (secure 0 0 0) 0x7f43c801ac60 con 0x7f43d806f6c0 2024-09-11T03:05:22.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:22.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43cf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:22.328 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":112,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 10 librados: watch_flush enter 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 10 librados: watch_flush exit 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 20 client.45288.objecter shutdown clearing up homeless session... 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 10 client.45288.objecter successfully canceled tick 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f43b803df40 msgr2=0x7f43b8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f43b803df40 0x7f43b8040400 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f43c0007fd0 tx=0x7f43c0009040 comp rx=0 tx=0).stop 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 10 monclient: shutdown 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:22.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d806f6c0 msgr2=0x7f43d81235f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d806f6c0 0x7f43d81235f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f43c800ebe0 tx=0x7f43c800c520 comp rx=0 tx=0).stop 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 shutdown_connections 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f43b803df40 0x7f43b8040400 unknown :-1 s=CLOSED pgs=3660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f43d813ec80 0x7f43d8124070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43d806faa0 0x7f43d8123b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 --2- 172.21.15.98:0/4284596250 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43d806f6c0 0x7f43d81235f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 >> 172.21.15.98:0/4284596250 conn(0x7f43d80acac0 msgr2=0x7f43d80a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 shutdown_connections 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 -- 172.21.15.98:0/4284596250 wait complete. 2024-09-11T03:05:22.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:22.325+0000 7f43dd60e640 1 librados: shutdown 2024-09-11T03:05:25.346 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient: get_monmap_and_config 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient: build_initial_monmap 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient: monmap: 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:25.475316+0000 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:25.475316+0000 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:25.473 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:25.474 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 1 Processor -- start 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 1 -- start start 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient: init 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient: _add_conns ranks=[1,2,4,5,7,3,8,9,6,0,10] 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ffae806f650 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 0x7ffae8072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient(hunting): picked mon.noname-b con 0x7ffae80a8380 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffae8070140 0x7ffae8140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient(hunting): picked mon.noname-c con 0x7ffae8070140 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:25.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae8071a80 con 0x7ffae806f650 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7ffae80a8800 con 0x7ffae80a8380 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ffae8071d60 con 0x7ffae8070140 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaee44e640 10 monclient(hunting): authenticate will time out at 1879.355550s 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffae8070140 0x7ffae8140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 0x7ffae8072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffae8070140 0x7ffae8140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:56658/0 (socket says 172.21.15.98:56658) 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 1 -- 172.21.15.98:0/202218977 learned_addr learned my addr 172.21.15.98:0/202218977 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 10 monclient(hunting): get_auth_request con 0x7ffae8070140 auth_method 0 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:25.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 10 monclient(hunting): get_auth_request con 0x7ffae80a8380 auth_method 0 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 1 -- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 msgr2=0x7ffae8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 1 -- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 msgr2=0x7ffae8072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffae77fe640 1 --2- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 0x7ffae8072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.469+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae77fe640 1 --2- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 0x7ffae8072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_done global_id 24673 payload 274 2024-09-11T03:05:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 1 -- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 msgr2=0x7ffae8072640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 1 --2- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 0x7ffae8072640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 1 -- 172.21.15.98:0/202218977 >> v1:172.21.15.190:6789/0 conn(0x7ffae806f650 legacy=0x7ffae806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 10 monclient: _finish_hunting 0 2024-09-11T03:05:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 1 monclient: found mon.noname-c 2024-09-11T03:05:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 1 -- 172.21.15.98:0/202218977 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae80714c0 con 0x7ffae8070140 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaec9eb640 1 --2- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffae8070140 0x7ffae8140c20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009e80 tx=0x7ffae001c910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3c3f61bbd9563571 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 1 -- 172.21.15.98:0/202218977 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffae00090d0 con 0x7ffae8070140 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: dump: 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:25.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:25.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 1 -- 172.21.15.98:0/202218977 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae00a5c20 con 0x7ffae8070140 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 5 monclient: authenticate success, global_id 24673 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 1 -- 172.21.15.98:0/202218977 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffae00a5e10 con 0x7ffae8070140 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 4 set_mon_vals no callback set 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:25.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffae6ffd640 10 monclient: dump: 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:25.489 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 10 monclient: shutdown 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 -- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffae8070140 msgr2=0x7ffae8140c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 --2- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffae8070140 0x7ffae8140c20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7ffae0009e80 tx=0x7ffae001c910 comp rx=0 tx=0).stop 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 -- 172.21.15.98:0/202218977 shutdown_connections 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 --2- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ffae8070140 0x7ffae8140c20 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 --2- 172.21.15.98:0/202218977 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7ffae80a8380 0x7ffae8072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 -- 172.21.15.98:0/202218977 >> 172.21.15.98:0/202218977 conn(0x7ffae80ac8e0 msgr2=0x7ffae80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 -- 172.21.15.98:0/202218977 shutdown_connections 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 1 -- 172.21.15.98:0/202218977 wait complete. 2024-09-11T03:05:25.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 10 monclient: build_initial_monmap 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.473+0000 7ffaee44e640 10 monclient: monmap: 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:25.491 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 librados: starting msgr at 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 librados: starting objecter 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 Processor -- start 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 -- start start 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 librados: setting wanted keys 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 librados: calling monclient init 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: init 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _add_conns ranks=[1,6,0,8,7,2,5,3,4] 2024-09-11T03:05:25.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae806f650 0x7ffae8128be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): picked mon.noname-f con 0x7ffae806f650 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffae8070140 0x7ffae81d1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): picked mon.noname-d con 0x7ffae8070140 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae80a8380 0x7ffae8126690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): picked mon.noname-a con 0x7ffae80a8380 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8143d10 con 0x7ffae80a8380 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffae8143b90 con 0x7ffae806f650 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ffae8143e90 con 0x7ffae8070140 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient(hunting): authenticate will time out at 1879.360914s 2024-09-11T03:05:25.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffae8070140 0x7ffae81d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffae8070140 0x7ffae81d1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:49222/0 (socket says 172.21.15.98:49222) 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 1 -- 172.21.15.98:0/1104729361 learned_addr learned my addr 172.21.15.98:0/1104729361 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 10 monclient(hunting): get_auth_request con 0x7ffae8070140 auth_method 0 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae80a8380 0x7ffae8126690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): get_auth_request con 0x7ffae80a8380 auth_method 0 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae806f650 0x7ffae8128be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:25.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient(hunting): get_auth_request con 0x7ffae806f650 auth_method 0 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient(hunting): handle_auth_done global_id 45297 payload 824 2024-09-11T03:05:25.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 -- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffae8070140 msgr2=0x7ffae81d1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffae8070140 0x7ffae81d1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 -- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae806f650 msgr2=0x7ffae8128be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae806f650 0x7ffae8128be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient: _finish_hunting 0 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 monclient: found mon.noname-a 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 -- 172.21.15.98:0/1104729361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae812a100 con 0x7ffae80a8380 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae77fe640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffae8070140 0x7ffae81d1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaec9eb640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae80a8380 0x7ffae8126690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffae00ab000 tx=0x7ffae80a7f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 client.?.objecter ms_handle_connect 0x7ffae80a8380 2024-09-11T03:05:25.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 1 -- 172.21.15.98:0/1104729361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffae00bc020 con 0x7ffae80a8380 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: dump: 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:25.497 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: _finish_auth 0 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 1 -- 172.21.15.98:0/1104729361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae0003bb0 con 0x7ffae80a8380 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 1 -- 172.21.15.98:0/1104729361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffae0003de0 con 0x7ffae80a8380 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 5 monclient: authenticate success, global_id 45297 2024-09-11T03:05:25.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _renew_subs 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae812a3e0 con 0x7ffae80a8380 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae67fc640 4 set_mon_vals no callback set 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 client.45297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 monclient: dump: 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:25.499 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _renew_subs 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:25.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae80a7580 con 0x7ffae80a8380 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 librados: init done 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae8072640 con 0x7ffae80a8380 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 1 -- 172.21.15.98:0/1104729361 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ffae0004040 con 0x7ffae80a8380 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ffad403df40 0x7ffad4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 1 -- 172.21.15.98:0/1104729361 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(624..624 src has 1..624) ==== 6769+0+0 (secure 0 0 0) 0x7ffae00ead30 con 0x7ffae80a8380 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 10 client.45297.objecter ms_dispatch 0x7ffae8000ba0 osd_map(624..624 src has 1..624) 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 3 client.45297.objecter handle_osd_map got epochs [624,624] > 0 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 3 client.45297.objecter handle_osd_map decoding full epoch 624 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae4ff9640 20 client.45297.objecter dump_active .. 0 homeless 2024-09-11T03:05:25.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ffad403df40 0x7ffad4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:25.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.477+0000 7ffae7fff640 10 monclient: get_auth_request con 0x7ffad403df40 auth_method 0 2024-09-11T03:05:25.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.481+0000 7ffae4ff9640 10 client.45297.objecter ms_handle_connect 0x7ffad403df40 2024-09-11T03:05:25.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.481+0000 7ffae7fff640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ffad403df40 0x7ffad4040400 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7ffad8007fd0 tx=0x7ffad8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:25.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.481+0000 7ffae4ff9640 1 -- 172.21.15.98:0/1104729361 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7ffae00c6030 con 0x7ffae80a8380 2024-09-11T03:05:25.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.481+0000 7ffae4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:25.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.481+0000 7ffae4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:25.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.725+0000 7ffaee44e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:25.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.725+0000 7ffaee44e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:25.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.725+0000 7ffaee44e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:25.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.725+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffae8129570 con 0x7ffae80a8380 2024-09-11T03:05:25.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.725+0000 7ffae4ff9640 1 -- 172.21.15.98:0/1104729361 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3189 (secure 0 0 0) 0x7ffae0016620 con 0x7ffae80a8380 2024-09-11T03:05:25.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.725+0000 7ffae4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:25.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.725+0000 7ffae4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:25.729 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":100,"quorum":[0,1,3,4,5,6,7,8],"quorum_names":["a","f","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":115,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 10 librados: watch_flush enter 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 10 librados: watch_flush exit 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 20 client.45297.objecter shutdown clearing up homeless session... 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 10 client.45297.objecter successfully canceled tick 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ffad403df40 msgr2=0x7ffad4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ffad403df40 0x7ffad4040400 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7ffad8007fd0 tx=0x7ffad8009040 comp rx=0 tx=0).stop 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 10 monclient: shutdown 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae80a8380 msgr2=0x7ffae8126690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae80a8380 0x7ffae8126690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ffae00ab000 tx=0x7ffae80a7f40 comp rx=0 tx=0).stop 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 shutdown_connections 2024-09-11T03:05:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ffad403df40 0x7ffad4040400 unknown :-1 s=CLOSED pgs=3661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffae80a8380 0x7ffae8126690 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ffae8070140 0x7ffae81d1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 --2- 172.21.15.98:0/1104729361 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffae806f650 0x7ffae8128be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:25.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 >> 172.21.15.98:0/1104729361 conn(0x7ffae80ac8e0 msgr2=0x7ffae80a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:25.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 shutdown_connections 2024-09-11T03:05:25.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 -- 172.21.15.98:0/1104729361 wait complete. 2024-09-11T03:05:25.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:25.729+0000 7ffaee44e640 1 librados: shutdown 2024-09-11T03:05:28.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient: get_monmap_and_config 2024-09-11T03:05:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient: build_initial_monmap 2024-09-11T03:05:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient: monmap: 2024-09-11T03:05:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:28.876 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:28.878606+0000 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:28.878606+0000 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:28.877 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:28.878 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:28.878 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 1 Processor -- start 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 1 -- start start 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient: init 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient: _add_conns ranks=[9,0,6,7,5,2,10,1,8,4,3] 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f146806f650 0x7f1468072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient(hunting): picked mon.noname-a con 0x7f146806f650 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:28.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14680a8380 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1468070140 0x7f1468140c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient(hunting): picked mon.noname-d con 0x7f1468070140 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1468071a80 con 0x7f14680a8380 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f14680a81b0 con 0x7f146806f650 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f14680a8800 con 0x7f1468070140 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f146d76c640 10 monclient(hunting): authenticate will time out at 1882.759024s 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1468070140 0x7f1468140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1468070140 0x7f1468140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:56998/0 (socket says 172.21.15.98:56998) 2024-09-11T03:05:28.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 1 -- 172.21.15.98:0/1073490607 learned_addr learned my addr 172.21.15.98:0/1073490607 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 1 --2- 172.21.15.98:0/1073490607 >> v2:172.21.15.98:3300/0 conn(0x7f146806f650 0x7f1468072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 10 monclient(hunting): get_auth_request con 0x7f1468070140 auth_method 0 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): get_auth_request con 0x7f146806f650 auth_method 0 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:28.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient(hunting): handle_auth_done global_id 45306 payload 274 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 1 -- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1468070140 msgr2=0x7f1468140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 1 --2- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1468070140 0x7f1468140c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 1 -- 172.21.15.98:0/1073490607 >> v1:172.21.15.98:6789/0 conn(0x7f14680a8380 legacy=0x7f146806fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 1 monclient: found mon.noname-a 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1465ffb640 1 -- 172.21.15.98:0/1073490607 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1311250797 0 0) 0x7f1468071a80 con 0x7f14680a8380 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:28.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1466ffd640 1 -- 172.21.15.98:0/1073490607 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1468071d60 con 0x7f146806f650 2024-09-11T03:05:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f1465ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:05:28.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.873+0000 7f14677fe640 1 --2- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1468070140 0x7f1468140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1466ffd640 1 --2- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f146806f650 0x7f1468072640 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f14500097c0 tx=0x7f145001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9959528909498f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 1 -- 172.21.15.98:0/1073490607 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f14500b0070 con 0x7f146806f650 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: dump: 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.890 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:28.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 1 -- 172.21.15.98:0/1073490607 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14500aa810 con 0x7f146806f650 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 5 monclient: authenticate success, global_id 45306 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 4 set_mon_vals no callback set 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 1 -- 172.21.15.98:0/1073490607 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f14500a9300 con 0x7f146806f650 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f1465ffb640 10 monclient: dump: 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:28.892 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:28.893 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 10 monclient: shutdown 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 -- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f146806f650 msgr2=0x7f1468072640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 --2- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f146806f650 0x7f1468072640 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f14500097c0 tx=0x7f145001c360 comp rx=0 tx=0).stop 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 -- 172.21.15.98:0/1073490607 shutdown_connections 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 --2- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1468070140 0x7f1468140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 --2- 172.21.15.98:0/1073490607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f146806f650 0x7f1468072640 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 -- 172.21.15.98:0/1073490607 >> 172.21.15.98:0/1073490607 conn(0x7f14680ac8e0 msgr2=0x7f14680a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 -- 172.21.15.98:0/1073490607 shutdown_connections 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 1 -- 172.21.15.98:0/1073490607 wait complete. 2024-09-11T03:05:28.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 10 monclient: build_initial_monmap 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.877+0000 7f146d76c640 10 monclient: monmap: 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:28.895 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 librados: starting msgr at 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 librados: starting objecter 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 Processor -- start 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 -- start start 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 librados: setting wanted keys 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 librados: calling monclient init 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: init 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:28.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _add_conns ranks=[6,4,7,1,5,2,8,0,3] 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f146806f650 0x7f1468128da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): picked mon.noname-d con 0x7f146806f650 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1468070140 0x7f14681d1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): picked mon.noname-c con 0x7f1468070140 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f14680a8380 0x7f1468126780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): picked mon.noname-i con 0x7f14680a8380 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f1468143d10 con 0x7f1468070140 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:28.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f1468143b90 con 0x7f146806f650 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f1468143e90 con 0x7f14680a8380 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient(hunting): authenticate will time out at 1882.764824s 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f146806f650 0x7f1468128da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f146806f650 0x7f1468128da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57004/0 (socket says 172.21.15.98:57004) 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 -- 172.21.15.98:0/3144702361 learned_addr learned my addr 172.21.15.98:0/3144702361 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): get_auth_request con 0x7f146806f650 auth_method 0 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1468070140 0x7f14681d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 10 monclient(hunting): get_auth_request con 0x7f1468070140 auth_method 0 2024-09-11T03:05:28.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient(hunting): handle_auth_done global_id 84471 payload 824 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 -- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f14680a8380 msgr2=0x7f1468126780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f14680a8380 0x7f1468126780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 -- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1468070140 msgr2=0x7f14681d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:28.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1468070140 0x7f14681d1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 monclient: found mon.noname-d 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 -- 172.21.15.98:0/3144702361 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14680aa000 con 0x7f146806f650 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1468070140 0x7f14681d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f1466ffd640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f146806f650 0x7f1468128da0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14500ae040 tx=0x7f1450019100 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 client.?.objecter ms_handle_connect 0x7f146806f650 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 1 -- 172.21.15.98:0/3144702361 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f14500b0040 con 0x7f146806f650 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:28.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: dump: 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:28.901 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 1 -- 172.21.15.98:0/3144702361 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14500b9070 con 0x7f146806f650 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 5 monclient: authenticate success, global_id 84471 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 1 -- 172.21.15.98:0/3144702361 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f14500044b0 con 0x7f146806f650 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:28.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 monclient: dump: 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:28.903 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14657fa640 4 set_mon_vals no callback set 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _renew_subs 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f146812a2c0 con 0x7f146806f650 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 client.84471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _renew_subs 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f14681c8d20 con 0x7f146806f650 2024-09-11T03:05:28.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 librados: init done 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1468129730 con 0x7f146806f650 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 1 -- 172.21.15.98:0/3144702361 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f14500b3070 con 0x7f146806f650 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f144803df40 0x7f1448040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 1 -- 172.21.15.98:0/3144702361 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(628..628 src has 1..628) ==== 6761+0+0 (secure 0 0 0) 0x7f14500ea050 con 0x7f146806f650 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 10 client.84471.objecter ms_dispatch 0x7f1468000ba0 osd_map(628..628 src has 1..628) 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 3 client.84471.objecter handle_osd_map got epochs [628,628] > 0 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 3 client.84471.objecter handle_osd_map decoding full epoch 628 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f145f7fe640 20 client.84471.objecter dump_active .. 0 homeless 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.881+0000 7f14667fc640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f144803df40 0x7f1448040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:28.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.885+0000 7f14667fc640 10 monclient: get_auth_request con 0x7f144803df40 auth_method 0 2024-09-11T03:05:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.885+0000 7f14667fc640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f144803df40 0x7f1448040400 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7f1468128d30 tx=0x7f1460009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.885+0000 7f145f7fe640 10 client.84471.objecter ms_handle_connect 0x7f144803df40 2024-09-11T03:05:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.885+0000 7f145f7fe640 1 -- 172.21.15.98:0/3144702361 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f14680aa000 con 0x7f146806f650 2024-09-11T03:05:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.885+0000 7f145f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:28.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:28.885+0000 7f145f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.121+0000 7f146d76c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.121+0000 7f146d76c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.121+0000 7f146d76c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:29.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.121+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1468128240 con 0x7f146806f650 2024-09-11T03:05:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.121+0000 7f145f7fe640 1 -- 172.21.15.98:0/3144702361 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f14500db070 con 0x7f146806f650 2024-09-11T03:05:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.121+0000 7f145f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:29.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.121+0000 7f145f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:29.126 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":106,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 10 librados: watch_flush enter 2024-09-11T03:05:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 10 librados: watch_flush exit 2024-09-11T03:05:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 20 client.84471.objecter shutdown clearing up homeless session... 2024-09-11T03:05:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 10 client.84471.objecter successfully canceled tick 2024-09-11T03:05:29.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f144803df40 msgr2=0x7f1448040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f144803df40 0x7f1448040400 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7f1468128d30 tx=0x7f1460009040 comp rx=0 tx=0).stop 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 10 monclient: shutdown 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f146806f650 msgr2=0x7f1468128da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f146806f650 0x7f1468128da0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f14500ae040 tx=0x7f1450019100 comp rx=0 tx=0).stop 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 shutdown_connections 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f144803df40 0x7f1448040400 unknown :-1 s=CLOSED pgs=3662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f1468070140 0x7f14681d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f14680a8380 0x7f1468126780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 --2- 172.21.15.98:0/3144702361 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f146806f650 0x7f1468128da0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 >> 172.21.15.98:0/3144702361 conn(0x7f14680ac8e0 msgr2=0x7f14680a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:29.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 shutdown_connections 2024-09-11T03:05:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 -- 172.21.15.98:0/3144702361 wait complete. 2024-09-11T03:05:29.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.125+0000 7f146d76c640 1 librados: shutdown 2024-09-11T03:05:29.146 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:05:29.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T03:05:29.146 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:05:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.261+0000 7fb0d975f640 10 monclient: get_monmap_and_config 2024-09-11T03:05:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.261+0000 7fb0d975f640 10 monclient: build_initial_monmap 2024-09-11T03:05:29.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.261+0000 7fb0d975f640 10 monclient: monmap: 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:29.268071+0000 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:29.268071+0000 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:29.266 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:29.267 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:29.267 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.261+0000 7fb0d975f640 1 Processor -- start 2024-09-11T03:05:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.261+0000 7fb0d975f640 1 -- start start 2024-09-11T03:05:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.261+0000 7fb0d975f640 10 monclient: init 2024-09-11T03:05:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient: _add_conns ranks=[5,9,0,2,8,1,7,6,3,4,10] 2024-09-11T03:05:29.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb0d4136780 0x7fb0d4073750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient(hunting): picked mon.noname-h con 0x7fb0d4136780 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0d4136b60 0x7fb0d4136f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient(hunting): picked mon.noname-a con 0x7fb0d4136b60 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0d40a4610 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0d40700d0 con 0x7fb0d40a4610 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fb0d4072db0 con 0x7fb0d4136b60 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb0d40713e0 con 0x7fb0d4136780 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient(hunting): authenticate will time out at 1883.148478s 2024-09-11T03:05:29.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fb0d4136b60 0x7fb0d4136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb0d40a4610 0x7fb0d40a4ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56800/0 (socket says 172.21.15.98:56800) 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d37fe640 1 -- 172.21.15.98:0/1717137313 learned_addr learned my addr 172.21.15.98:0/1717137313 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): get_auth_request con 0x7fb0d4136b60 auth_method 0 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d2ffd640 1 --2- 172.21.15.98:0/1717137313 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb0d4136780 0x7fb0d4073750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d2ffd640 10 monclient(hunting): get_auth_request con 0x7fb0d4136780 auth_method 0 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_done global_id 45324 payload 274 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 1 -- 172.21.15.98:0/1717137313 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb0d4136780 msgr2=0x7fb0d4073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 1 --2- 172.21.15.98:0/1717137313 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb0d4136780 0x7fb0d4073750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 1 -- 172.21.15.98:0/1717137313 >> v1:172.21.15.98:6789/0 conn(0x7fb0d40a4610 legacy=0x7fb0d40a4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient: _finish_hunting 0 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 1 monclient: found mon.noname-a 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 1 -- 172.21.15.98:0/1717137313 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d4073c90 con 0x7fb0d4136b60 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 1 -- 172.21.15.98:0/1717137313 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241892993 0 0) 0x7fb0d40700d0 con 0x7fb0d40a4610 2024-09-11T03:05:29.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d27fc640 1 --2- 172.21.15.98:0/1717137313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d4136b60 0x7fb0d4136f40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0097c0 tx=0x7fb0bc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e23d57a4cd3dae server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 1 -- 172.21.15.98:0/1717137313 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0bc0b0070 con 0x7fb0d4136b60 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: dump: 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 1 -- 172.21.15.98:0/1717137313 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0bc0aa810 con 0x7fb0d4136b60 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:29.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 1 -- 172.21.15.98:0/1717137313 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0bc0a9300 con 0x7fb0d4136b60 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d1ffb640 10 monclient: dump: 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 5 monclient: authenticate success, global_id 45324 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 4 set_mon_vals no callback set 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 10 monclient: shutdown 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- 172.21.15.98:0/1717137313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d4136b60 msgr2=0x7fb0d4136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 --2- 172.21.15.98:0/1717137313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d4136b60 0x7fb0d4136f40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc0097c0 tx=0x7fb0bc01c360 comp rx=0 tx=0).stop 2024-09-11T03:05:29.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- 172.21.15.98:0/1717137313 shutdown_connections 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 --2- 172.21.15.98:0/1717137313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d4136b60 0x7fb0d4136f40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 --2- 172.21.15.98:0/1717137313 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb0d4136780 0x7fb0d4073750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- 172.21.15.98:0/1717137313 >> 172.21.15.98:0/1717137313 conn(0x7fb0d406ed00 msgr2=0x7fb0d406f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- 172.21.15.98:0/1717137313 shutdown_connections 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.265+0000 7fb0d975f640 1 -- 172.21.15.98:0/1717137313 wait complete. 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: build_initial_monmap 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: monmap: 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:29.279 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:29.280 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 librados: starting msgr at 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 librados: starting objecter 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 Processor -- start 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 -- start start 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 librados: setting wanted keys 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 librados: calling monclient init 2024-09-11T03:05:29.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: init 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: _add_conns ranks=[4,8,1,6,7,3,2,5,0] 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0d40a4610 0x7fb0d41d14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): picked mon.noname-c con 0x7fb0d40a4610 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb0d4136780 0x7fb0d41d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): picked mon.noname-e con 0x7fb0d4136780 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d4136b60 0x7fb0d409cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): picked mon.noname-f con 0x7fb0d4136b60 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:29.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d41412b0 con 0x7fb0d4136b60 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb0d4141130 con 0x7fb0d40a4610 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fb0d4141430 con 0x7fb0d4136780 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): authenticate will time out at 1883.153510s 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0d40a4610 0x7fb0d41d14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb0d4136780 0x7fb0d41d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0d40a4610 0x7fb0d41d14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54674/0 (socket says 172.21.15.98:54674) 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 1 -- 172.21.15.98:0/4052029561 learned_addr learned my addr 172.21.15.98:0/4052029561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:29.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): get_auth_request con 0x7fb0d40a4610 auth_method 0 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): get_auth_request con 0x7fb0d4136780 auth_method 0 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d37fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d4136b60 0x7fb0d409cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d37fe640 10 monclient(hunting): get_auth_request con 0x7fb0d4136b60 auth_method 0 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient(hunting): handle_auth_done global_id 25199 payload 824 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0d40a4610 msgr2=0x7fb0d41d14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0d40a4610 0x7fb0d41d14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d4136b60 msgr2=0x7fb0d409cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0d4136b60 0x7fb0d409cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient: _finish_hunting 0 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 monclient: found mon.noname-e 2024-09-11T03:05:29.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 -- 172.21.15.98:0/4052029561 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d413da50 con 0x7fb0d4136780 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb0d40a4610 0x7fb0d41d14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d27fc640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb0d4136780 0x7fb0d41d1b80 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4073140 tx=0x7fb0bc019100 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb0d4136780 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0bc0b0040 con 0x7fb0d4136780 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: dump: 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.292 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.293 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0bc0044b0 con 0x7fb0d4136780 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0bc0aad70 con 0x7fb0d4136780 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 monclient: dump: 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.294 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d17fa640 4 set_mon_vals no callback set 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 5 monclient: authenticate success, global_id 25199 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: _renew_subs 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d413dd30 con 0x7fb0d4136780 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 client.25199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: _renew_subs 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d40a1c90 con 0x7fb0d4136780 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 librados: init done 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d41d20c0 0x7fb0d409d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d413c450 con 0x7fb0d41d20c0 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb0bc0b3070 con 0x7fb0d4136780 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d41d20c0 0x7fb0d409d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient: get_auth_request con 0x7fb0d41d20c0 auth_method 0 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0b403df40 0x7fb0b4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_done global_id 45342 payload 824 2024-09-11T03:05:29.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(628..628 src has 1..628) ==== 6761+0+0 (secure 0 0 0) 0x7fb0bc0e9140 con 0x7fb0d4136780 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 1 -- 172.21.15.98:0/4052029561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0d40a8770 con 0x7fb0d41d20c0 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 client.25199.objecter ms_dispatch 0x7fb0d4000ba0 osd_map(628..628 src has 1..628) 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 3 client.25199.objecter handle_osd_map got epochs [628,628] > 0 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 3 client.25199.objecter handle_osd_map decoding full epoch 628 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 20 client.25199.objecter dump_active .. 0 homeless 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 client.25199.objecter ms_handle_connect 0x7fb0d41d20c0 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0d2ffd640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d41d20c0 0x7fb0d409d080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4007590 tx=0x7fb0c4007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.269+0000 7fb0cb7fe640 10 client.25199.objecter resend_mon_ops 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0c4015a60 con 0x7fb0d41d20c0 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0d37fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0b403df40 0x7fb0b4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:29.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 10 monclient: dump: 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0d37fe640 10 monclient: get_auth_request con 0x7fb0b403df40 auth_method 0 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0c401b070 con 0x7fb0d41d20c0 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d41d20c0 msgr2=0x7fb0d409d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d41d20c0 0x7fb0d409d080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4007590 tx=0x7fb0c4007a50 comp rx=0 tx=0).stop 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d41d20c0 msgr2=0x7fb0d409d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d41d20c0 0x7fb0d409d080 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4007590 tx=0x7fb0c4007a50 comp rx=0 tx=0).stop 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0cb7fe640 10 client.25199.objecter ms_handle_connect 0x7fb0b403df40 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.273+0000 7fb0d37fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0b403df40 0x7fb0b4040400 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc000f70 tx=0x7fb0cc007af0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d409d5c0 0x7fb0d41384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d406ca10 con 0x7fb0d409d5c0 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d409d5c0 0x7fb0d41384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient: get_auth_request con 0x7fb0d409d5c0 auth_method 0 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 10 monclient(hunting): handle_auth_done global_id 45351 payload 824 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 1 -- 172.21.15.98:0/4052029561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb0d409d9d0 con 0x7fb0d409d5c0 2024-09-11T03:05:29.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d2ffd640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d409d5c0 0x7fb0d41384c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4007590 tx=0x7fb0c4004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 10 client.25199.objecter ms_handle_connect 0x7fb0d409d5c0 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 10 client.25199.objecter resend_mon_ops 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0c4015e10 con 0x7fb0d409d5c0 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 10 monclient: dump: 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.302 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9602 (secure 0 0 0) 0x7fb0c4015e10 con 0x7fb0d409d5c0 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d409d5c0 msgr2=0x7fb0d41384c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d409d5c0 0x7fb0d41384c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4007590 tx=0x7fb0c4004a50 comp rx=0 tx=0).stop 2024-09-11T03:05:29.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d409d5c0 msgr2=0x7fb0d41384c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0cb7fe640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb0d409d5c0 0x7fb0d41384c0 secure :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4007590 tx=0x7fb0c4004a50 comp rx=0 tx=0).stop 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 555717, 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:29.304 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:29.305 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:29.306 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:29.307 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:29.308 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:29.309 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.310 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:29.311 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.314 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:29.315 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.317 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.318 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.320 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.321 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.322 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.323 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.324 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:29.326 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:29.327 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:29.328 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:29.328 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:29.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 10 librados: watch_flush enter 2024-09-11T03:05:29.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 10 librados: watch_flush exit 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 20 client.25199.objecter shutdown clearing up homeless session... 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 10 client.25199.objecter successfully canceled tick 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0b403df40 msgr2=0x7fb0b4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb0b403df40 0x7fb0b4040400 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc000f70 tx=0x7fb0cc007af0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 10 monclient: shutdown 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb0d4136780 msgr2=0x7fb0d41d1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 --2- 172.21.15.98:0/4052029561 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fb0d4136780 0x7fb0d41d1b80 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4073140 tx=0x7fb0bc019100 comp rx=0 tx=0).stop 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d37fe640 1 -- 172.21.15.98:0/4052029561 reap_dead start 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d37fe640 1 -- 172.21.15.98:0/4052029561 reap_dead start 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 shutdown_connections 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 >> 172.21.15.98:0/4052029561 conn(0x7fb0d406ed00 msgr2=0x7fb0d4076b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:29.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 shutdown_connections 2024-09-11T03:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.277+0000 7fb0d975f640 1 -- 172.21.15.98:0/4052029561 wait complete. 2024-09-11T03:05:29.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.281+0000 7fb0d975f640 1 librados: shutdown 2024-09-11T03:05:29.331 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:29.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T03:05:29.331 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:05:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient: get_monmap_and_config 2024-09-11T03:05:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient: build_initial_monmap 2024-09-11T03:05:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient: monmap: 2024-09-11T03:05:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.447 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:29.449706+0000 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:29.449706+0000 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:29.448 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:29.449 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:29.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 Processor -- start 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 -- start start 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient: init 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient: _add_conns ranks=[7,8,4,5,3,6,1,2,10,0,9] 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31300a7aa0 0x7f31300a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): picked mon.noname-i con 0x7f31300a7aa0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:29.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31300a12a0 0x7f31300a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): picked mon.noname-e con 0x7f31300a12a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f3130139c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): picked mon.noname-c con 0x7f3130139850 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31300a8900 con 0x7f3130139850 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f31300a1720 con 0x7f31300a7aa0 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f31300a27c0 con 0x7f31300a12a0 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:29.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f313810e640 10 monclient(hunting): authenticate will time out at 1883.330492s 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f3130139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31300a12a0 0x7f31300a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f3130139c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54690/0 (socket says 172.21.15.98:54690) 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31300a12a0 0x7f31300a83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55710/0 (socket says 172.21.15.98:55710) 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 1 -- 172.21.15.98:0/1613728613 learned_addr learned my addr 172.21.15.98:0/1613728613 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 10 monclient(hunting): get_auth_request con 0x7f3130139850 auth_method 0 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): get_auth_request con 0x7f31300a12a0 auth_method 0 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f3135eaa640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31300a7aa0 0x7f31300a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f3135eaa640 10 monclient(hunting): get_auth_request con 0x7f31300a7aa0 auth_method 0 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f3135eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f3135eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f3135eaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient(hunting): handle_auth_done global_id 25208 payload 274 2024-09-11T03:05:29.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 -- 172.21.15.98:0/1613728613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31300a7aa0 msgr2=0x7f31300a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31300a7aa0 0x7f31300a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 -- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 msgr2=0x7f3130139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f3130139c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient: _finish_hunting 0 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 monclient: found mon.noname-e 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31356a9640 1 -- 172.21.15.98:0/1613728613 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31300a9590 con 0x7f31300a12a0 2024-09-11T03:05:29.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.445+0000 7f31366ab640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f3130139c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31356a9640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31300a12a0 0x7f31300a83c0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f31200097f0 tx=0x7f312001c430 comp rx=0 tx=0).ready entity=mon.8 client_cookie=64ad832664460a7f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 1 -- 172.21.15.98:0/1613728613 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31200b0070 con 0x7f31300a12a0 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: dump: 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.461 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: _finish_auth 0 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: _check_auth_tickets 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 1 -- 172.21.15.98:0/1613728613 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31200aa810 con 0x7f31300a12a0 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 5 monclient: authenticate success, global_id 25208 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 1 -- 172.21.15.98:0/1613728613 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31200a9300 con 0x7f31300a12a0 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 4 set_mon_vals no callback set 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3134ea8640 10 monclient: dump: 2024-09-11T03:05:29.463 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.464 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient: shutdown 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31300a12a0 msgr2=0x7f31300a83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31300a12a0 0x7f31300a83c0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f31200097f0 tx=0x7f312001c430 comp rx=0 tx=0).stop 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- 172.21.15.98:0/1613728613 shutdown_connections 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f3130139c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f31300a12a0 0x7f31300a83c0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 --2- 172.21.15.98:0/1613728613 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31300a7aa0 0x7f31300a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- 172.21.15.98:0/1613728613 >> 172.21.15.98:0/1613728613 conn(0x7f31300ac690 msgr2=0x7f31300a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- 172.21.15.98:0/1613728613 shutdown_connections 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- 172.21.15.98:0/1613728613 wait complete. 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient: build_initial_monmap 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient: monmap: 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:29.466 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 librados: starting msgr at 2024-09-11T03:05:29.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 librados: starting objecter 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 Processor -- start 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- start start 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 librados: setting wanted keys 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 librados: calling monclient init 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient: init 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient: _add_conns ranks=[5,1,4,3,6,7,8,0,2] 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31300a12a0 0x7f3130138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): picked mon.noname-h con 0x7f31300a12a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31300a7aa0 0x7f3130138760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): picked mon.noname-f con 0x7f31300a7aa0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f313013d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): picked mon.noname-c con 0x7f3130139850 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3130143440 con 0x7f31300a7aa0 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31301432c0 con 0x7f3130139850 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f31301435c0 con 0x7f31300a12a0 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f313810e640 10 monclient(hunting): authenticate will time out at 1883.335505s 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f313013d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f313013d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54694/0 (socket says 172.21.15.98:54694) 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 -- 172.21.15.98:0/2731538714 learned_addr learned my addr 172.21.15.98:0/2731538714 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): get_auth_request con 0x7f3130139850 auth_method 0 2024-09-11T03:05:29.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31356a9640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31300a7aa0 0x7f3130138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient(hunting): handle_auth_done global_id 24709 payload 824 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31300a12a0 msgr2=0x7f3130138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31300a12a0 0x7f3130138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f31300a12a0 0x7f3130138220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31300a7aa0 msgr2=0x7f3130138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:29.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31300a7aa0 0x7f3130138760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient: _finish_hunting 0 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 1 monclient: found mon.noname-c 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.449+0000 7f31366ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31366ab640 1 -- 172.21.15.98:0/2731538714 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3130123d20 con 0x7f3130139850 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31356a9640 10 monclient: get_auth_request con 0x7f31300a7aa0 auth_method 0 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31356a9640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f31300a7aa0 0x7f3130138760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31366ab640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f313013d5b0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f3128009e50 tx=0x7f312800c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 client.?.objecter ms_handle_connect 0x7f3130139850 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3128014480 con 0x7f3130139850 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: dump: 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3128019070 con 0x7f3130139850 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 5 monclient: authenticate success, global_id 24709 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 10 monclient: _renew_subs 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3130139400 con 0x7f3130139850 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:29.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3128014ab0 con 0x7f3130139850 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 monclient: dump: 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:29.474 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 10 client.24709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 10 monclient: _renew_subs 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f313013fac0 con 0x7f3130139850 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3127fff640 4 set_mon_vals no callback set 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 1 librados: init done 2024-09-11T03:05:29.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f3130122800 con 0x7f313013daf0 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 10 monclient: get_auth_request con 0x7f313013daf0 auth_method 0 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:29.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3128014c70 con 0x7f3130139850 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f311403df70 0x7f3114040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(628..628 src has 1..628) ==== 6761+0+0 (secure 0 0 0) 0x7f3128050160 con 0x7f3130139850 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 client.24709.objecter ms_dispatch 0x7f3130000ba0 osd_map(628..628 src has 1..628) 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 3 client.24709.objecter handle_osd_map got epochs [628,628] > 0 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 3 client.24709.objecter handle_osd_map decoding full epoch 628 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 20 client.24709.objecter dump_active .. 0 homeless 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31356a9640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f311403df70 0x7f3114040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31356a9640 10 monclient: get_auth_request con 0x7f311403df70 auth_method 0 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31267fc640 10 client.24709.objecter ms_handle_connect 0x7f311403df70 2024-09-11T03:05:29.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.453+0000 7f31356a9640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f311403df70 0x7f3114040430 secure :-1 s=READY pgs=3664 cs=0 l=1 rev1=1 crypto rx=0x7f31200ad000 tx=0x7f31200a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:29.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 10 monclient: get_auth_request con 0x7f313013daf0 auth_method 0 2024-09-11T03:05:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T03:05:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:05:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:29.661 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:29.657+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:30.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 10 monclient: get_auth_request con 0x7f313013daf0 auth_method 0 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:30.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.061+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:05:30.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 10 monclient: get_auth_request con 0x7f313013daf0 auth_method 0 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:30.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:30.865+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:05:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.449+0000 7f31257fa640 10 monclient: tick 2024-09-11T03:05:32.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.449+0000 7f31257fa640 10 monclient: _check_auth_tickets 2024-09-11T03:05:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.449+0000 7f31257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:32.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.449+0000 7f31257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient: get_auth_request con 0x7f313013daf0 auth_method 0 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 10 monclient(hunting): handle_auth_done global_id 94106 payload 824 2024-09-11T03:05:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31300a5030 con 0x7f313013daf0 2024-09-11T03:05:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 10 client.24709.objecter ms_handle_connect 0x7f313013daf0 2024-09-11T03:05:32.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 10 client.24709.objecter resend_mon_ops 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f312c016700 tx=0x7f312c016bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f312c00f040 con 0x7f313013daf0 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 10 monclient: dump: 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.474 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.475 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f312c01e070 con 0x7f313013daf0 2024-09-11T03:05:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f312c016700 tx=0x7f312c016bc0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 msgr2=0x7f313013ded0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:32.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.469+0000 7f31267fc640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013daf0 0x7f313013ded0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f312c016700 tx=0x7f312c016bc0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f313810e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f313810e640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013e410 0x7f3130071710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f313810e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f313006ca10 con 0x7f313013e410 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013e410 0x7f3130071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient: get_auth_request con 0x7f313013e410 auth_method 0 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 10 monclient(hunting): handle_auth_done global_id 94115 payload 824 2024-09-11T03:05:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.477+0000 7f3135eaa640 1 -- 172.21.15.98:0/2731538714 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3130139c50 con 0x7f313013e410 2024-09-11T03:05:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 10 client.24709.objecter ms_handle_connect 0x7f313013e410 2024-09-11T03:05:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 10 client.24709.objecter resend_mon_ops 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f3135eaa640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013e410 0x7f3130071710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f312c008010 tx=0x7f312c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f312c003dc0 con 0x7f313013e410 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 10 monclient: dump: 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f312c003950 con 0x7f313013e410 2024-09-11T03:05:32.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:32.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:32.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013e410 msgr2=0x7f3130071710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013e410 0x7f3130071710 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f312c008010 tx=0x7f312c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013e410 msgr2=0x7f3130071710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31267fc640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f313013e410 0x7f3130071710 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f312c008010 tx=0x7f312c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 30867, 2024-09-11T03:05:32.487 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:32.488 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:32.489 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:32.491 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.492 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:32.494 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.495 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.496 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.497 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:32.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.499 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.500 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.501 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.502 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.504 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.506 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:32.507 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:32.508 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.510 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:32.510 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 10 librados: watch_flush enter 2024-09-11T03:05:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 10 librados: watch_flush exit 2024-09-11T03:05:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 20 client.24709.objecter shutdown clearing up homeless session... 2024-09-11T03:05:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 10 client.24709.objecter successfully canceled tick 2024-09-11T03:05:32.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f311403df70 msgr2=0x7f3114040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f311403df70 0x7f3114040430 secure :-1 s=READY pgs=3664 cs=0 l=1 rev1=1 crypto rx=0x7f31200ad000 tx=0x7f31200a6000 comp rx=0 tx=0).stop 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 10 monclient: shutdown 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 msgr2=0x7f313013d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 --2- 172.21.15.98:0/2731538714 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3130139850 0x7f313013d5b0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f3128009e50 tx=0x7f312800c490 comp rx=0 tx=0).stop 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31366ab640 1 -- 172.21.15.98:0/2731538714 reap_dead start 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f31366ab640 1 -- 172.21.15.98:0/2731538714 reap_dead start 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 shutdown_connections 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 >> 172.21.15.98:0/2731538714 conn(0x7f31300ac690 msgr2=0x7f31300a1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 shutdown_connections 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 -- 172.21.15.98:0/2731538714 wait complete. 2024-09-11T03:05:32.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.481+0000 7f313810e640 1 librados: shutdown 2024-09-11T03:05:32.513 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:32.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T03:05:32.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:05:32.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.637+0000 7fee9393b640 10 monclient: get_monmap_and_config 2024-09-11T03:05:32.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.637+0000 7fee9393b640 10 monclient: build_initial_monmap 2024-09-11T03:05:32.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.637+0000 7fee9393b640 10 monclient: monmap: 2024-09-11T03:05:32.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:32.642838+0000 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:32.642838+0000 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:32.641 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:32.642 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:32.642 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:32.642 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:32.642 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:32.642 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:32.642 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.637+0000 7fee9393b640 1 Processor -- start 2024-09-11T03:05:32.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.637+0000 7fee9393b640 1 -- start start 2024-09-11T03:05:32.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.637+0000 7fee9393b640 10 monclient: init 2024-09-11T03:05:32.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient: _add_conns ranks=[0,5,8,6,3,4,1,2,10,7,9] 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee8c0a5830 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fee8c0a1770 0x7fee8c13ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient(hunting): picked mon.noname-h con 0x7fee8c0a1770 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c071ed0 0x7fee8c13ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient(hunting): picked mon.noname-e con 0x7fee8c071ed0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee8c071c90 con 0x7fee8c0a5830 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fee8c0739b0 con 0x7fee8c0a1770 2024-09-11T03:05:32.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fee8c073c90 con 0x7fee8c071ed0 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee9393b640 10 monclient(hunting): authenticate will time out at 1886.525073s 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee916d7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fee8c0a5830 0x7fee8c0a5c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56816/0 (socket says 172.21.15.98:56816) 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee916d7640 1 -- 172.21.15.98:0/4086171968 learned_addr learned my addr 172.21.15.98:0/4086171968 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 1 --2- 172.21.15.98:0/4086171968 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c071ed0 0x7fee8c13ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): get_auth_request con 0x7fee8c071ed0 auth_method 0 2024-09-11T03:05:32.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee90ed6640 1 --2- 172.21.15.98:0/4086171968 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fee8c0a1770 0x7fee8c13ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee90ed6640 10 monclient(hunting): get_auth_request con 0x7fee8c0a1770 auth_method 0 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee90ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee90ed6640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee90ed6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient(hunting): handle_auth_done global_id 25217 payload 274 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 1 -- 172.21.15.98:0/4086171968 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fee8c0a1770 msgr2=0x7fee8c13ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 1 --2- 172.21.15.98:0/4086171968 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fee8c0a1770 0x7fee8c13ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 1 -- 172.21.15.98:0/4086171968 >> v1:172.21.15.98:6789/0 conn(0x7fee8c0a5830 legacy=0x7fee8c0a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient: _finish_hunting 0 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 1 monclient: found mon.noname-e 2024-09-11T03:05:32.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:32.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:32.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 1 -- 172.21.15.98:0/4086171968 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8c072350 con 0x7fee8c071ed0 2024-09-11T03:05:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee91ed8640 1 --2- 172.21.15.98:0/4086171968 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c071ed0 0x7fee8c13ef80 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fee84009df0 tx=0x7fee8401c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=2db862bb91d32ee5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.641+0000 7fee83fff640 1 -- 172.21.15.98:0/4086171968 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee840a6950 con 0x7fee8c071ed0 2024-09-11T03:05:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:32.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: dump: 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.661 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: _finish_auth 0 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: _check_auth_tickets 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 5 monclient: authenticate success, global_id 25217 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 1 -- 172.21.15.98:0/4086171968 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee840a6b10 con 0x7fee8c071ed0 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:32.662 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 4 set_mon_vals no callback set 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 1 -- 172.21.15.98:0/4086171968 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee840a6cc0 con 0x7fee8c071ed0 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee83fff640 10 monclient: dump: 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.663 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: shutdown 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- 172.21.15.98:0/4086171968 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c071ed0 msgr2=0x7fee8c13ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 --2- 172.21.15.98:0/4086171968 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c071ed0 0x7fee8c13ef80 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fee84009df0 tx=0x7fee8401c850 comp rx=0 tx=0).stop 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- 172.21.15.98:0/4086171968 shutdown_connections 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 --2- 172.21.15.98:0/4086171968 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fee8c0a1770 0x7fee8c13ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 --2- 172.21.15.98:0/4086171968 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c071ed0 0x7fee8c13ef80 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- 172.21.15.98:0/4086171968 >> 172.21.15.98:0/4086171968 conn(0x7fee8c0aa420 msgr2=0x7fee8c0a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- 172.21.15.98:0/4086171968 shutdown_connections 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- 172.21.15.98:0/4086171968 wait complete. 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: build_initial_monmap 2024-09-11T03:05:32.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: monmap: 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:32.666 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 librados: starting msgr at 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 librados: starting objecter 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 Processor -- start 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- start start 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 librados: setting wanted keys 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 librados: calling monclient init 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: init 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:32.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: _add_conns ranks=[1,8,3,7,0,4,2,5,6] 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee8c071ed0 0x7fee8c1cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): picked mon.noname-f con 0x7fee8c071ed0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c0a5830 0x7fee8c1cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): picked mon.noname-e con 0x7fee8c0a5830 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8c1cfc10 0x7fee8c124550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): picked mon.noname-g con 0x7fee8c1cfc10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c142070 con 0x7fee8c071ed0 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fee8c141ef0 con 0x7fee8c1cfc10 2024-09-11T03:05:32.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fee8c1421f0 con 0x7fee8c0a5830 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient(hunting): authenticate will time out at 1886.530380s 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c0a5830 0x7fee8c1cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c0a5830 0x7fee8c1cf6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55730/0 (socket says 172.21.15.98:55730) 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 -- 172.21.15.98:0/2972342905 learned_addr learned my addr 172.21.15.98:0/2972342905 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): get_auth_request con 0x7fee8c0a5830 auth_method 0 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee916d7640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee8c071ed0 0x7fee8c1cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee91ed8640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8c1cfc10 0x7fee8c124550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee916d7640 10 monclient(hunting): get_auth_request con 0x7fee8c071ed0 auth_method 0 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee916d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee916d7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee916d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient(hunting): handle_auth_done global_id 25226 payload 824 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8c1cfc10 msgr2=0x7fee8c124550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8c1cfc10 0x7fee8c124550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee8c071ed0 msgr2=0x7fee8c1cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee8c071ed0 0x7fee8c1cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient: _finish_hunting 0 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 monclient: found mon.noname-e 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 -- 172.21.15.98:0/2972342905 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8c127c70 con 0x7fee8c0a5830 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee91ed8640 10 monclient: get_auth_request con 0x7fee8c1cfc10 auth_method 0 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee91ed8640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fee8c1cfc10 0x7fee8c124550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee90ed6640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c0a5830 0x7fee8c1cf6d0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00d800 tx=0x7fee7c00dcc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 client.?.objecter ms_handle_connect 0x7fee8c0a5830 2024-09-11T03:05:32.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee7c00bb30 con 0x7fee8c0a5830 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: dump: 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.672 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee7c019070 con 0x7fee8c0a5830 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 5 monclient: authenticate success, global_id 25226 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: _renew_subs 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee8c127f50 con 0x7fee8c0a5830 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee7c00bcf0 con 0x7fee8c0a5830 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee81ffb640 10 monclient: dump: 2024-09-11T03:05:32.674 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.675 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee837fe640 4 set_mon_vals no callback set 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 client.25226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: _renew_subs 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:32.676 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fee8c128230 con 0x7fee8c0a5830 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 librados: init done 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.645+0000 7fee9393b640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c0a1750 0x7fee8c124a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee9393b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fee8c125f60 con 0x7fee8c0a1750 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fee7c013070 con 0x7fee8c0a5830 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c0a1750 0x7fee8c124a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient: get_auth_request con 0x7fee8c0a1750 auth_method 0 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.677 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee7003df40 0x7fee70040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(632..632 src has 1..632) ==== 6769+0+0 (secure 0 0 0) 0x7fee7c00f040 con 0x7fee8c0a5830 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 client.25226.objecter ms_dispatch 0x7fee8c000ba0 osd_map(632..632 src has 1..632) 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 3 client.25226.objecter handle_osd_map got epochs [632,632] > 0 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 3 client.25226.objecter handle_osd_map decoding full epoch 632 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 10 monclient(hunting): handle_auth_done global_id 24718 payload 824 2024-09-11T03:05:32.678 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 1 -- 172.21.15.98:0/2972342905 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee8c13f160 con 0x7fee8c0a1750 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 20 client.25226.objecter dump_active .. 0 homeless 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 client.25226.objecter ms_handle_connect 0x7fee8c0a1750 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 client.25226.objecter resend_mon_ops 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee916d7640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c0a1750 0x7fee8c124a90 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fee88000f70 tx=0x7fee8800e890 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee91ed8640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee7003df40 0x7fee70040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee91ed8640 10 monclient: get_auth_request con 0x7fee7003df40 auth_method 0 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee88019070 con 0x7fee8c0a1750 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:32.679 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 monclient: dump: 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.680 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fee88005a70 con 0x7fee8c0a1750 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c0a1750 msgr2=0x7fee8c124a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c0a1750 0x7fee8c124a90 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fee88000f70 tx=0x7fee8800e890 comp rx=0 tx=0).stop 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c0a1750 msgr2=0x7fee8c124a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c0a1750 0x7fee8c124a90 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee81ffb640 10 client.25226.objecter ms_handle_connect 0x7fee7003df40 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.649+0000 7fee91ed8640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee7003df40 0x7fee70040400 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7fee84009df0 tx=0x7fee840a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee9393b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee9393b640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c124fd0 0x7fee8c09ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee9393b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fee8c06ca10 con 0x7fee8c124fd0 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c124fd0 0x7fee8c09ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient: get_auth_request con 0x7fee8c124fd0 auth_method 0 2024-09-11T03:05:32.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 10 monclient(hunting): handle_auth_done global_id 24727 payload 824 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 1 -- 172.21.15.98:0/2972342905 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee8c127990 con 0x7fee8c124fd0 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee81ffb640 10 client.25226.objecter ms_handle_connect 0x7fee8c124fd0 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee81ffb640 10 client.25226.objecter resend_mon_ops 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.653+0000 7fee916d7640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c124fd0 0x7fee8c09ea60 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fee8800efd0 tx=0x7fee88000ee0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fee88019030 con 0x7fee8c124fd0 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 10 monclient: dump: 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9435 (secure 0 0 0) 0x7fee8801b830 con 0x7fee8c124fd0 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:32.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c124fd0 msgr2=0x7fee8c09ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c124fd0 0x7fee8c09ea60 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fee8800efd0 tx=0x7fee88000ee0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c124fd0 msgr2=0x7fee8c09ea60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:32.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee81ffb640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fee8c124fd0 0x7fee8c09ea60 secure :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fee8800efd0 tx=0x7fee88000ee0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.686 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:32.686 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 948441, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:32.687 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.688 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.689 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.690 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:32.691 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.693 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:32.694 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:32.695 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:32.696 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:32.697 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.698 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:32.699 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.701 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.702 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:32.703 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:32.705 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:32.707 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:32.709 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:32.711 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:32.712 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:32.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:32.714 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:32.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 10 librados: watch_flush enter 2024-09-11T03:05:32.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 10 librados: watch_flush exit 2024-09-11T03:05:32.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 20 client.25226.objecter shutdown clearing up homeless session... 2024-09-11T03:05:32.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 10 client.25226.objecter successfully canceled tick 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee7003df40 msgr2=0x7fee70040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fee7003df40 0x7fee70040400 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7fee84009df0 tx=0x7fee840a7000 comp rx=0 tx=0).stop 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 10 monclient: shutdown 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c0a5830 msgr2=0x7fee8c1cf6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 --2- 172.21.15.98:0/2972342905 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fee8c0a5830 0x7fee8c1cf6d0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7fee7c00d800 tx=0x7fee7c00dcc0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee91ed8640 1 -- 172.21.15.98:0/2972342905 reap_dead start 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee91ed8640 1 -- 172.21.15.98:0/2972342905 reap_dead start 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 shutdown_connections 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 >> 172.21.15.98:0/2972342905 conn(0x7fee8c0aa420 msgr2=0x7fee8c0a7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:32.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 shutdown_connections 2024-09-11T03:05:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 -- 172.21.15.98:0/2972342905 wait complete. 2024-09-11T03:05:32.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.657+0000 7fee9393b640 1 librados: shutdown 2024-09-11T03:05:32.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:32.718 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T03:05:32.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:05:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient: get_monmap_and_config 2024-09-11T03:05:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient: build_initial_monmap 2024-09-11T03:05:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient: monmap: 2024-09-11T03:05:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.835 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:32.837334+0000 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:32.837334+0000 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:32.836 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:32.837 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:32.837 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 Processor -- start 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 -- start start 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient: init 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient: _add_conns ranks=[5,8,9,4,6,10,1,7,2,3,0] 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139b10 0x7f1228072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): picked mon.noname-h con 0x7f1228139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1228139ef0 0x7f12280a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): picked mon.noname-e con 0x7f1228139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f122813a2d0 0x7f12280a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): picked mon.noname-a con 0x7f122813a2d0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f12280702b0 con 0x7f122813a2d0 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f1228071bf0 con 0x7f1228139b10 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f1228071350 con 0x7f1228139ef0 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122f4f3640 10 monclient(hunting): authenticate will time out at 1886.717586s 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1228139ef0 0x7f12280a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f122813a2d0 0x7f12280a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1228139ef0 0x7f12280a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55744/0 (socket says 172.21.15.98:55744) 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f122813a2d0 0x7f12280a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36002/0 (socket says 172.21.15.98:36002) 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 1 -- 172.21.15.98:0/3310506673 learned_addr learned my addr 172.21.15.98:0/3310506673 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 10 monclient(hunting): get_auth_request con 0x7f1228139ef0 auth_method 0 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): get_auth_request con 0x7f122813a2d0 auth_method 0 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122d28f640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139b10 0x7f1228072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122d28f640 10 monclient(hunting): get_auth_request con 0x7f1228139b10 auth_method 0 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122d28f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122d28f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122d28f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient(hunting): handle_auth_done global_id 45369 payload 274 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 -- 172.21.15.98:0/3310506673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1228139ef0 msgr2=0x7f12280a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1228139ef0 0x7f12280a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 -- 172.21.15.98:0/3310506673 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139b10 msgr2=0x7f1228072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139b10 0x7f1228072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient: _finish_hunting 0 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 monclient: found mon.noname-a 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:32.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.833+0000 7f122da90640 1 -- 172.21.15.98:0/3310506673 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12280a9620 con 0x7f122813a2d0 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122da90640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122813a2d0 0x7f12280a7f50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1218009dc0 tx=0x7f121801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968e2f39c746babe server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 1 -- 172.21.15.98:0/3310506673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12180ac070 con 0x7f122813a2d0 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: dump: 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.852 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: _finish_auth 0 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: _check_auth_tickets 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 1 -- 172.21.15.98:0/3310506673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12180a68d0 con 0x7f122813a2d0 2024-09-11T03:05:32.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 1 -- 172.21.15.98:0/3310506673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12180a7440 con 0x7f122813a2d0 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1227fff640 10 monclient: dump: 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.854 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 5 monclient: authenticate success, global_id 45369 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 4 set_mon_vals no callback set 2024-09-11T03:05:32.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: shutdown 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- 172.21.15.98:0/3310506673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122813a2d0 msgr2=0x7f12280a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122813a2d0 0x7f12280a7f50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1218009dc0 tx=0x7f121801c830 comp rx=0 tx=0).stop 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- 172.21.15.98:0/3310506673 shutdown_connections 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f1228139ef0 0x7f12280a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f122813a2d0 0x7f12280a7f50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 --2- 172.21.15.98:0/3310506673 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139b10 0x7f1228072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- 172.21.15.98:0/3310506673 >> 172.21.15.98:0/3310506673 conn(0x7f12280ac790 msgr2=0x7f12280a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- 172.21.15.98:0/3310506673 shutdown_connections 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- 172.21.15.98:0/3310506673 wait complete. 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: build_initial_monmap 2024-09-11T03:05:32.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: monmap: 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 librados: starting msgr at 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 librados: starting objecter 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 Processor -- start 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- start start 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 librados: setting wanted keys 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 librados: calling monclient init 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: init 2024-09-11T03:05:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: _add_conns ranks=[2,5,7,0,4,3,6,1,8] 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1228139b10 0x7f1228134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): picked mon.noname-b con 0x7f1228139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139ef0 0x7f122813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): picked mon.noname-h con 0x7f1228139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f122813a2d0 0x7f122813abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): picked mon.noname-i con 0x7f122813a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f1228143460 con 0x7f1228139b10 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f12280a88a0 con 0x7f1228139ef0 2024-09-11T03:05:32.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f12281435e0 con 0x7f122813a2d0 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient(hunting): authenticate will time out at 1886.723071s 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1228139b10 0x7f1228134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1228139b10 0x7f1228134ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49496/0 (socket says 172.21.15.98:49496) 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 -- 172.21.15.98:0/1604901665 learned_addr learned my addr 172.21.15.98:0/1604901665 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): get_auth_request con 0x7f1228139b10 auth_method 0 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient(hunting): handle_auth_done global_id 94124 payload 824 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122ca8e640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139ef0 0x7f122813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f122813a2d0 msgr2=0x7f122813abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f122813a2d0 0x7f122813abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139ef0 msgr2=0x7f122813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f1228139ef0 0x7f122813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient: _finish_hunting 0 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 monclient: found mon.noname-b 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 -- 172.21.15.98:0/1604901665 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122813e320 con 0x7f1228139b10 2024-09-11T03:05:32.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122d28f640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1228139b10 0x7f1228134ec0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f122000d800 tx=0x7f122000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 client.?.objecter ms_handle_connect 0x7f1228139b10 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f122000bb30 con 0x7f1228139b10 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: dump: 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.862 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:32.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1220019070 con 0x7f1228139b10 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 5 monclient: authenticate success, global_id 94124 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: _renew_subs 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1228135810 con 0x7f1228139b10 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f122000bcf0 con 0x7f1228139b10 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f1225ffb640 10 monclient: dump: 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.865 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 client.94124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: _renew_subs 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f12277fe640 4 set_mon_vals no callback set 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12280a9e00 con 0x7f1228139b10 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.837+0000 7f122f4f3640 1 librados: init done 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122f4f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122f4f3640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1228135400 0x7f122813b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f122806ca10 con 0x7f1228135400 2024-09-11T03:05:32.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1228135400 0x7f122813b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient: get_auth_request con 0x7f1228135400 auth_method 0 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f1220013070 con 0x7f1228139b10 2024-09-11T03:05:32.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 10 monclient(hunting): handle_auth_done global_id 84480 payload 824 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 1 -- 172.21.15.98:0/1604901665 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f122813c020 con 0x7f1228135400 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f121403df40 0x7f1214040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(632..632 src has 1..632) ==== 6769+0+0 (secure 0 0 0) 0x7f122000f040 con 0x7f1228139b10 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 client.94124.objecter ms_dispatch 0x7f1228000ba0 osd_map(632..632 src has 1..632) 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 3 client.94124.objecter handle_osd_map got epochs [632,632] > 0 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 3 client.94124.objecter handle_osd_map decoding full epoch 632 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122ca8e640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1228135400 0x7f122813b130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f12280a5ed0 tx=0x7f121c010b60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 20 client.94124.objecter dump_active .. 0 homeless 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 client.94124.objecter ms_handle_connect 0x7f1228135400 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 client.94124.objecter resend_mon_ops 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f121c00f040 con 0x7f1228135400 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 monclient: dump: 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.869 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122da90640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f121403df40 0x7f1214040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122da90640 10 monclient: get_auth_request con 0x7f121403df40 auth_method 0 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f121c01c400 con 0x7f1228135400 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1228135400 msgr2=0x7f122813b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1228135400 0x7f122813b130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f12280a5ed0 tx=0x7f121c010b60 comp rx=0 tx=0).stop 2024-09-11T03:05:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1228135400 msgr2=0x7f122813b130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f1228135400 0x7f122813b130 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f12280a5ed0 tx=0x7f121c010b60 comp rx=0 tx=0).stop 2024-09-11T03:05:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f122da90640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f121403df40 0x7f1214040400 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7f12180ae000 tx=0x7f1218004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.841+0000 7f1225ffb640 10 client.94124.objecter ms_handle_connect 0x7f121403df40 2024-09-11T03:05:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122f4f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:32.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122f4f3640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f122813b670 0x7f12280a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122f4f3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f122806e870 con 0x7f122813b670 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122ca8e640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f122813b670 0x7f12280a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122ca8e640 10 monclient: get_auth_request con 0x7f122813b670 auth_method 0 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122ca8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122ca8e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.845+0000 7f122ca8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122ca8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122ca8e640 10 monclient(hunting): handle_auth_done global_id 84489 payload 824 2024-09-11T03:05:32.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122ca8e640 1 -- 172.21.15.98:0/1604901665 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f122806e680 con 0x7f122813b670 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 10 client.94124.objecter ms_handle_connect 0x7f122813b670 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122ca8e640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f122813b670 0x7f12280a0e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f12280a5ed0 tx=0x7f121c0046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 10 client.94124.objecter resend_mon_ops 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f121c01cae0 con 0x7f122813b670 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 10 monclient: dump: 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:32.874 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7f121c01c670 con 0x7f122813b670 2024-09-11T03:05:32.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f122813b670 msgr2=0x7f12280a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f122813b670 0x7f12280a0e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f12280a5ed0 tx=0x7f121c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f122813b670 msgr2=0x7f12280a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f1225ffb640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f122813b670 0x7f12280a0e60 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f12280a5ed0 tx=0x7f121c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:32.876 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 158662, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.877 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.878 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.879 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:32.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:32.881 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:32.882 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:32.885 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:32.886 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:32.888 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:32.889 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:32.890 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.891 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.893 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:32.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.895 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:32.896 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:32.898 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:32.899 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 10 librados: watch_flush enter 2024-09-11T03:05:32.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 10 librados: watch_flush exit 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 20 client.94124.objecter shutdown clearing up homeless session... 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 10 client.94124.objecter successfully canceled tick 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f121403df40 msgr2=0x7f1214040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f121403df40 0x7f1214040400 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7f12180ae000 tx=0x7f1218004ac0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 10 monclient: shutdown 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1228139b10 msgr2=0x7f1228134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 --2- 172.21.15.98:0/1604901665 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f1228139b10 0x7f1228134ec0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f122000d800 tx=0x7f122000dcc0 comp rx=0 tx=0).stop 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122da90640 1 -- 172.21.15.98:0/1604901665 reap_dead start 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122da90640 1 -- 172.21.15.98:0/1604901665 reap_dead start 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 shutdown_connections 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 >> 172.21.15.98:0/1604901665 conn(0x7f12280ac790 msgr2=0x7f12280aa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 shutdown_connections 2024-09-11T03:05:32.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 -- 172.21.15.98:0/1604901665 wait complete. 2024-09-11T03:05:32.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:32.849+0000 7f122f4f3640 1 librados: shutdown 2024-09-11T03:05:32.903 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:32.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T03:05:32.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:05:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient: get_monmap_and_config 2024-09-11T03:05:33.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient: monmap: 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:33.025941+0000 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:33.025941+0000 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:33.024 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:33.025 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 Processor -- start 2024-09-11T03:05:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 -- start start 2024-09-11T03:05:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient: init 2024-09-11T03:05:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient: _add_conns ranks=[6,5,9,10,2,7,3,1,8,0,4] 2024-09-11T03:05:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): picked mon.noname-d con 0x7f40f0139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f0139ef0 0x7f40f00a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): picked mon.noname-h con 0x7f40f0139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f40f013a2d0 0x7f40f00a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): picked mon.noname-a con 0x7f40f013a2d0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f40f00702b0 con 0x7f40f013a2d0 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f40f0071bf0 con 0x7f40f0139ef0 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f40f0071350 con 0x7f40f0139b10 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f6c1b640 10 monclient(hunting): authenticate will time out at 1886.906661s 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57030/0 (socket says 172.21.15.98:57030) 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 -- 172.21.15.98:0/1177284114 learned_addr learned my addr 172.21.15.98:0/1177284114 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): get_auth_request con 0x7f40f0139b10 auth_method 0 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 1 --2- 172.21.15.98:0/1177284114 >> v2:172.21.15.98:3300/0 conn(0x7f40f013a2d0 0x7f40f00a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 10 monclient(hunting): get_auth_request con 0x7f40f013a2d0 auth_method 0 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40effff640 1 --2- 172.21.15.98:0/1177284114 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f0139ef0 0x7f40f00a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f51b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40effff640 10 monclient(hunting): get_auth_request con 0x7f40f0139ef0 auth_method 0 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient(hunting): handle_auth_done global_id 84498 payload 274 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 -- 172.21.15.98:0/1177284114 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f0139ef0 msgr2=0x7f40f00a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 --2- 172.21.15.98:0/1177284114 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f0139ef0 0x7f40f00a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 -- 172.21.15.98:0/1177284114 >> v2:172.21.15.98:3300/0 conn(0x7f40f013a2d0 msgr2=0x7f40f00a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 --2- 172.21.15.98:0/1177284114 >> v2:172.21.15.98:3300/0 conn(0x7f40f013a2d0 0x7f40f00a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 monclient: found mon.noname-d 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:05:33.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.021+0000 7f40f49b7640 1 -- 172.21.15.98:0/1177284114 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f00a9620 con 0x7f40f0139b10 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f49b7640 1 --2- 172.21.15.98:0/1177284114 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0072600 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40e8009e50 tx=0x7f40e801c8f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b78e8c1c00e99d91 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 1 -- 172.21.15.98:0/1177284114 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40e80090d0 con 0x7f40f0139b10 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: dump: 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 1 -- 172.21.15.98:0/1177284114 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e80a5ab0 con 0x7f40f0139b10 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 1 -- 172.21.15.98:0/1177284114 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40e80a5c60 con 0x7f40f0139b10 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40ef7fe640 10 monclient: dump: 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 5 monclient: authenticate success, global_id 84498 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 4 set_mon_vals no callback set 2024-09-11T03:05:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 10 monclient: shutdown 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1177284114 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 msgr2=0x7f40f0072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1177284114 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0072600 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f40e8009e50 tx=0x7f40e801c8f0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1177284114 shutdown_connections 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1177284114 >> v2:172.21.15.98:3300/0 conn(0x7f40f013a2d0 0x7f40f00a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1177284114 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f0139ef0 0x7f40f00a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1177284114 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0072600 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1177284114 >> 172.21.15.98:0/1177284114 conn(0x7f40f00ac790 msgr2=0x7f40f00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1177284114 shutdown_connections 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.025+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1177284114 wait complete. 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: monmap: 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 librados: starting msgr at 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 librados: starting objecter 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 Processor -- start 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 -- start start 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 librados: setting wanted keys 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 librados: calling monclient init 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: init 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: _add_conns ranks=[6,7,5,3,8,1,4,0,2] 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): picked mon.noname-d con 0x7f40f0139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f40f0139ef0 0x7f40f0134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): picked mon.noname-i con 0x7f40f0139ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f013a2d0 0x7f40f013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): picked mon.noname-h con 0x7f40f013a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f40f0143460 con 0x7f40f013a2d0 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f40f00a88a0 con 0x7f40f0139b10 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f40f01435e0 con 0x7f40f0139ef0 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): authenticate will time out at 1886.913279s 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0134a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57040/0 (socket says 172.21.15.98:57040) 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 -- 172.21.15.98:0/1583427570 learned_addr learned my addr 172.21.15.98:0/1583427570 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): get_auth_request con 0x7f40f0139b10 auth_method 0 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f51b8640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f013a2d0 0x7f40f013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f40f0139ef0 0x7f40f0134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient(hunting): handle_auth_done global_id 84507 payload 824 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f40f0139ef0 msgr2=0x7f40f0134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f40f0139ef0 0x7f40f0134f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f013a2d0 msgr2=0x7f40f013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f40f013a2d0 0x7f40f013a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 monclient: found mon.noname-d 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 -- 172.21.15.98:0/1583427570 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f013e210 con 0x7f40f0139b10 2024-09-11T03:05:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f49b7640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0134a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40e801d190 tx=0x7f40e8019690 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 client.?.objecter ms_handle_connect 0x7f40f0139b10 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40e80bc020 con 0x7f40f0139b10 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: dump: 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.053 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 5 monclient: authenticate success, global_id 84507 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e8003e10 con 0x7f40f0139b10 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: _renew_subs 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f00aa200 con 0x7f40f0139b10 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40e80044b0 con 0x7f40f0139b10 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ee7fc640 4 set_mon_vals no callback set 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: dump: 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.055 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 client.84507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: _renew_subs 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f013beb0 con 0x7f40f0139b10 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 librados: init done 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013ad10 0x7f40f013b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f40f006ca10 con 0x7f40f013ad10 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013ad10 0x7f40f013b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient: get_auth_request con 0x7f40f013ad10 auth_method 0 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f40e8004650 con 0x7f40f0139b10 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 10 monclient(hunting): handle_auth_done global_id 25244 payload 824 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40d003df60 0x7f40d0040420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 1 -- 172.21.15.98:0/1583427570 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40f006e680 con 0x7f40f013ad10 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(632..632 src has 1..632) ==== 6769+0+0 (secure 0 0 0) 0x7f40e80eabb0 con 0x7f40f0139b10 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 client.84507.objecter ms_dispatch 0x7f40f0000ba0 osd_map(632..632 src has 1..632) 2024-09-11T03:05:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 3 client.84507.objecter handle_osd_map got epochs [632,632] > 0 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 3 client.84507.objecter handle_osd_map decoding full epoch 632 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 20 client.84507.objecter dump_active .. 0 homeless 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 client.84507.objecter ms_handle_connect 0x7f40f013ad10 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 client.84507.objecter resend_mon_ops 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40effff640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013ad10 0x7f40f013b0f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f40f013c520 tx=0x7f40d800e990 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f51b8640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40d003df60 0x7f40d0040420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40d8017070 con 0x7f40f013ad10 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40ed7fa640 10 monclient: dump: 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.059 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.060 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.029+0000 7f40f51b8640 10 monclient: get_auth_request con 0x7f40d003df60 auth_method 0 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40d801c450 con 0x7f40f013ad10 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013ad10 msgr2=0x7f40f013b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013ad10 0x7f40f013b0f0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f40f013c520 tx=0x7f40d800e990 comp rx=0 tx=0).stop 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013ad10 msgr2=0x7f40f013b0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013ad10 0x7f40f013b0f0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40ed7fa640 10 client.84507.objecter ms_handle_connect 0x7f40d003df60 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.033+0000 7f40f51b8640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40d003df60 0x7f40d0040420 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f40f0140900 tx=0x7f40e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:33.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013b630 0x7f40f00a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f40f006e870 con 0x7f40f013b630 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013b630 0x7f40f00a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient: get_auth_request con 0x7f40f013b630 auth_method 0 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 10 monclient(hunting): handle_auth_done global_id 25253 payload 824 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 1 -- 172.21.15.98:0/1583427570 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40f013db00 con 0x7f40f013b630 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40effff640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013b630 0x7f40f00a0e60 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f40f013c520 tx=0x7f40d8004740 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 10 client.84507.objecter ms_handle_connect 0x7f40f013b630 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 10 client.84507.objecter resend_mon_ops 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40d800f040 con 0x7f40f013b630 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 10 monclient: dump: 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.063 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.064 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7f40d8013d30 con 0x7f40f013b630 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013b630 msgr2=0x7f40f00a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013b630 0x7f40f00a0e60 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f40f013c520 tx=0x7f40d8004740 comp rx=0 tx=0).stop 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013b630 msgr2=0x7f40f00a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40ed7fa640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f40f013b630 0x7f40f00a0e60 secure :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f40f013c520 tx=0x7f40d8004740 comp rx=0 tx=0).stop 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:33.065 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 948829, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:33.066 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 5, 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.067 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.068 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:33.069 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:33.070 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.071 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.073 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.076 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:33.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.078 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.079 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.080 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.081 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.082 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.083 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:33.084 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:33.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.086 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.088 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.089 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:33.090 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 10 librados: watch_flush enter 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 10 librados: watch_flush exit 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 20 client.84507.objecter shutdown clearing up homeless session... 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 10 client.84507.objecter successfully canceled tick 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40d003df60 msgr2=0x7f40d0040420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f40d003df60 0x7f40d0040420 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f40f0140900 tx=0x7f40e0008040 comp rx=0 tx=0).stop 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 10 monclient: shutdown 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 msgr2=0x7f40f0134a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0134a00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f40e801d190 tx=0x7f40e8019690 comp rx=0 tx=0).stop 2024-09-11T03:05:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f51b8640 1 -- 172.21.15.98:0/1583427570 reap_dead start 2024-09-11T03:05:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 shutdown_connections 2024-09-11T03:05:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 --2- 172.21.15.98:0/1583427570 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f40f0139b10 0x7f40f0134a00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 >> 172.21.15.98:0/1583427570 conn(0x7f40f00ac790 msgr2=0x7f40f00aa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 shutdown_connections 2024-09-11T03:05:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 -- 172.21.15.98:0/1583427570 wait complete. 2024-09-11T03:05:33.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.037+0000 7f40f6c1b640 1 librados: shutdown 2024-09-11T03:05:33.093 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:33.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T03:05:33.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient: get_monmap_and_config 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient: monmap: 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:33.221495+0000 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:33.221495+0000 2024-09-11T03:05:33.219 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:33.220 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 Processor -- start 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 -- start start 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient: init 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient: _add_conns ranks=[3,5,2,7,10,1,4,0,9,6,8] 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7130139b80 0x7f7130139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): picked mon.noname-g con 0x7f7130139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f713006fe50 0x7f71300a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): picked mon.noname-h con 0x7f713006fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:33.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7130072640 0x7f71300a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): picked mon.noname-b con 0x7f7130072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f71300719c0 con 0x7f7130072640 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7130071c10 con 0x7f7130139b80 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f71300702d0 con 0x7f713006fe50 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f7135f77640 10 monclient(hunting): authenticate will time out at 1887.101706s 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7130072640 0x7f71300a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7130072640 0x7f71300a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49502/0 (socket says 172.21.15.98:49502) 2024-09-11T03:05:33.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 -- 172.21.15.98:0/603259270 learned_addr learned my addr 172.21.15.98:0/603259270 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): get_auth_request con 0x7f7130072640 auth_method 0 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712effd640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f713006fe50 0x7f71300a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712f7fe640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7130139b80 0x7f7130139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712f7fe640 10 monclient(hunting): get_auth_request con 0x7f7130139b80 auth_method 0 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712effd640 10 monclient(hunting): get_auth_request con 0x7f713006fe50 auth_method 0 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient(hunting): handle_auth_done global_id 94133 payload 274 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 -- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f713006fe50 msgr2=0x7f71300a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f713006fe50 0x7f71300a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 -- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7130139b80 msgr2=0x7f7130139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7130139b80 0x7f7130139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 monclient: found mon.noname-b 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:05:33.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.217+0000 7f712ffff640 1 -- 172.21.15.98:0/603259270 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7130071370 con 0x7f7130072640 2024-09-11T03:05:33.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712ffff640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7130072640 0x7f71300a7f50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71280098d0 tx=0x7f712801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc071151f7ebcc5b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 1 -- 172.21.15.98:0/603259270 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71280b0070 con 0x7f7130072640 2024-09-11T03:05:33.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:05:33.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: dump: 2024-09-11T03:05:33.233 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.234 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 1 -- 172.21.15.98:0/603259270 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71280aa810 con 0x7f7130072640 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 5 monclient: authenticate success, global_id 94133 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 1 -- 172.21.15.98:0/603259270 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71280a9300 con 0x7f7130072640 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 4 set_mon_vals no callback set 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f712e7fc640 10 monclient: dump: 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.236 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 10 monclient: shutdown 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 -- 172.21.15.98:0/603259270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7130072640 msgr2=0x7f71300a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7130072640 0x7f71300a7f50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f71280098d0 tx=0x7f712801c440 comp rx=0 tx=0).stop 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 -- 172.21.15.98:0/603259270 shutdown_connections 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7130072640 0x7f71300a7f50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f713006fe50 0x7f71300a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 --2- 172.21.15.98:0/603259270 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7130139b80 0x7f7130139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 -- 172.21.15.98:0/603259270 >> 172.21.15.98:0/603259270 conn(0x7f71300ac730 msgr2=0x7f71300a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 -- 172.21.15.98:0/603259270 shutdown_connections 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 -- 172.21.15.98:0/603259270 wait complete. 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 10 monclient: monmap: 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:33.238 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 librados: starting msgr at 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.221+0000 7f7135f77640 1 librados: starting objecter 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 Processor -- start 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 -- start start 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 librados: setting wanted keys 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 librados: calling monclient init 2024-09-11T03:05:33.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient: init 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient: _add_conns ranks=[2,8,1,5,0,3,6,7,4] 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f713006fe50 0x7f7130133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): picked mon.noname-b con 0x7f713006fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7130072640 0x7f7130133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): picked mon.noname-e con 0x7f7130072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7130139b80 0x7f71301341e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): picked mon.noname-f con 0x7f7130139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f71301435c0 con 0x7f7130139b80 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7130143440 con 0x7f713006fe50 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7130143740 con 0x7f7130072640 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): authenticate will time out at 1887.108546s 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f713006fe50 0x7f7130133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7130072640 0x7f7130133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f713006fe50 0x7f7130133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49504/0 (socket says 172.21.15.98:49504) 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7130072640 0x7f7130133ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55758/0 (socket says 172.21.15.98:55758) 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 1 -- 172.21.15.98:0/1067333802 learned_addr learned my addr 172.21.15.98:0/1067333802 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): get_auth_request con 0x7f713006fe50 auth_method 0 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): get_auth_request con 0x7f7130072640 auth_method 0 2024-09-11T03:05:33.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712ffff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7130139b80 0x7f71301341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712ffff640 10 monclient(hunting): get_auth_request con 0x7f7130139b80 auth_method 0 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient(hunting): handle_auth_done global_id 25262 payload 824 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f713006fe50 msgr2=0x7f7130133760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f713006fe50 0x7f7130133760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7130139b80 msgr2=0x7f71301341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7130139b80 0x7f71301341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 monclient: found mon.noname-e 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 -- 172.21.15.98:0/1067333802 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713013d140 con 0x7f7130072640 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f713006fe50 0x7f7130133760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712effd640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7130072640 0x7f7130133ca0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f7120007e40 tx=0x7f712000e4b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 client.?.objecter ms_handle_connect 0x7f7130072640 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:33.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f712000ee90 con 0x7f7130072640 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: dump: 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.244 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7120014890 con 0x7f7130072640 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7120013380 con 0x7f7130072640 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:33.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712dffb640 4 set_mon_vals no callback set 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 monclient: dump: 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.246 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 5 monclient: authenticate success, global_id 25262 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient: _renew_subs 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71300aa460 con 0x7f7130072640 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 client.25262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient: _renew_subs 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:33.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f71301358f0 con 0x7f7130072640 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 librados: init done 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013cd20 0x7f713013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f713006ca10 con 0x7f713013cd20 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f7120014420 con 0x7f7130072640 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f711003e0e0 0x7f71100405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013cd20 0x7f713013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(632..632 src has 1..632) ==== 6769+0+0 (secure 0 0 0) 0x7f7120019020 con 0x7f7130072640 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 10 client.25262.objecter ms_dispatch 0x7f7130000ba0 osd_map(632..632 src has 1..632) 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 3 client.25262.objecter handle_osd_map got epochs [632,632] > 0 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 3 client.25262.objecter handle_osd_map decoding full epoch 632 2024-09-11T03:05:33.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient: get_auth_request con 0x7f713013cd20 auth_method 0 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f7127fff640 20 client.25262.objecter dump_active .. 0 homeless 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712ffff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f711003e0e0 0x7f71100405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712ffff640 10 monclient: get_auth_request con 0x7f711003e0e0 auth_method 0 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 10 monclient(hunting): handle_auth_done global_id 26056 payload 824 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.225+0000 7f712f7fe640 1 -- 172.21.15.98:0/1067333802 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7130139f60 con 0x7f713013cd20 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f712f7fe640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013cd20 0x7f713013a6b0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f7118010b90 tx=0x7f711800e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 client.25262.objecter ms_handle_connect 0x7f713013cd20 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 client.25262.objecter resend_mon_ops 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 client.25262.objecter ms_handle_connect 0x7f711003e0e0 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f712ffff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f711003e0e0 0x7f71100405a0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f71280ad000 tx=0x7f71280a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7118010e90 con 0x7f713013cd20 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 monclient: dump: 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.250 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f711800ee80 con 0x7f713013cd20 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:33.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013cd20 msgr2=0x7f713013a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013cd20 0x7f713013a6b0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f7118010b90 tx=0x7f711800e5a0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013cd20 msgr2=0x7f713013a6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.229+0000 7f7127fff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013cd20 0x7f713013a6b0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f7135f77640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f7135f77640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013abf0 0x7f713013b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f7135f77640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f713006e870 con 0x7f713013abf0 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013abf0 0x7f713013b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 10 monclient: get_auth_request con 0x7f713013abf0 auth_method 0 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.233+0000 7f712f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f712f7fe640 10 monclient(hunting): handle_auth_done global_id 26065 payload 824 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f712f7fe640 1 -- 172.21.15.98:0/1067333802 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f713013b540 con 0x7f713013abf0 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 10 client.25262.objecter ms_handle_connect 0x7f713013abf0 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 10 client.25262.objecter resend_mon_ops 2024-09-11T03:05:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f712f7fe640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013abf0 0x7f713013b000 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f7118019000 tx=0x7f71180046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7118014ab0 con 0x7f713013abf0 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 10 monclient: dump: 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.255 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9435 (secure 0 0 0) 0x7f7118014640 con 0x7f713013abf0 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013abf0 msgr2=0x7f713013b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013abf0 0x7f713013b000 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f7118019000 tx=0x7f71180046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013abf0 msgr2=0x7f713013b000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7127fff640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f713013abf0 0x7f713013b000 secure :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f7118019000 tx=0x7f71180046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 949014, 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:33.257 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.258 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:33.259 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:33.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.262 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:33.263 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.264 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.265 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:33.266 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.267 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.268 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.272 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.273 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:33.275 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.277 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:33.278 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:05:33.279 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:33.280 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 10 librados: watch_flush enter 2024-09-11T03:05:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 10 librados: watch_flush exit 2024-09-11T03:05:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 20 client.25262.objecter shutdown clearing up homeless session... 2024-09-11T03:05:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 10 client.25262.objecter successfully canceled tick 2024-09-11T03:05:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f711003e0e0 msgr2=0x7f71100405a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f711003e0e0 0x7f71100405a0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f71280ad000 tx=0x7f71280a6000 comp rx=0 tx=0).stop 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 10 monclient: shutdown 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7130072640 msgr2=0x7f7130133ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 --2- 172.21.15.98:0/1067333802 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7130072640 0x7f7130133ca0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f7120007e40 tx=0x7f712000e4b0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f712ffff640 1 -- 172.21.15.98:0/1067333802 reap_dead start 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f712ffff640 1 -- 172.21.15.98:0/1067333802 reap_dead start 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 shutdown_connections 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 >> 172.21.15.98:0/1067333802 conn(0x7f71300ac730 msgr2=0x7f71300a6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 shutdown_connections 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 -- 172.21.15.98:0/1067333802 wait complete. 2024-09-11T03:05:33.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.237+0000 7f7135f77640 1 librados: shutdown 2024-09-11T03:05:33.284 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:33.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T03:05:33.284 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:05:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient: get_monmap_and_config 2024-09-11T03:05:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient: monmap: 2024-09-11T03:05:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.399 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:33.401606+0000 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:33.401606+0000 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:33.400 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:33.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 Processor -- start 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 -- start start 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient: init 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient: _add_conns ranks=[2,8,4,7,1,3,6,9,0,5,10] 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fbc139b10 0x7f7fbc072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): picked mon.noname-b con 0x7f7fbc139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7fbc139ef0 0x7f7fbc0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): picked mon.noname-e con 0x7f7fbc139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7fbc13a2d0 0x7f7fbc0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): picked mon.noname-c con 0x7f7fbc13a2d0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc0702b0 con 0x7f7fbc139b10 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f7fbc071bf0 con 0x7f7fbc13a2d0 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f7fbc071350 con 0x7f7fbc139ef0 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc29b0640 10 monclient(hunting): authenticate will time out at 1887.281848s 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fbc139b10 0x7f7fbc072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7fbc13a2d0 0x7f7fbc0a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fbc139b10 0x7f7fbc072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49520/0 (socket says 172.21.15.98:49520) 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7fbc13a2d0 0x7f7fbc0a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54706/0 (socket says 172.21.15.98:54706) 2024-09-11T03:05:33.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 -- 172.21.15.98:0/48290393 learned_addr learned my addr 172.21.15.98:0/48290393 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): get_auth_request con 0x7f7fbc139b10 auth_method 0 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 10 monclient(hunting): get_auth_request con 0x7f7fbc13a2d0 auth_method 0 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7fbc139ef0 0x7f7fbc0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 10 monclient(hunting): get_auth_request con 0x7f7fbc139ef0 auth_method 0 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_done global_id 94151 payload 274 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 -- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7fbc139ef0 msgr2=0x7f7fbc0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7fbc139ef0 0x7f7fbc0a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 -- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7fbc13a2d0 msgr2=0x7f7fbc0a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7fbc13a2d0 0x7f7fbc0a7f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 monclient: found mon.noname-b 2024-09-11T03:05:33.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:05:33.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbbfff640 1 -- 172.21.15.98:0/48290393 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc0a9620 con 0x7f7fbc139b10 2024-09-11T03:05:33.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fc0f4d640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7fbc13a2d0 0x7f7fbc0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.397+0000 7f7fbb7fe640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7fbc139ef0 0x7f7fbc0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fbc139b10 0x7f7fbc072600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009e80 tx=0x7f7fb401c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deaebcb449165e95 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 1 -- 172.21.15.98:0/48290393 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb40090d0 con 0x7f7fbc139b10 2024-09-11T03:05:33.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:05:33.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: dump: 2024-09-11T03:05:33.413 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.414 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 5 monclient: authenticate success, global_id 94151 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 1 -- 172.21.15.98:0/48290393 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb40a5a70 con 0x7f7fbc139b10 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 1 -- 172.21.15.98:0/48290393 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb40a5c60 con 0x7f7fbc139b10 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 4 set_mon_vals no callback set 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:33.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbaffd640 10 monclient: dump: 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.416 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient: shutdown 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fbc139b10 msgr2=0x7f7fbc072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fbc139b10 0x7f7fbc072600 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009e80 tx=0x7f7fb401c910 comp rx=0 tx=0).stop 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- 172.21.15.98:0/48290393 shutdown_connections 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f7fbc13a2d0 0x7f7fbc0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f7fbc139ef0 0x7f7fbc0a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/48290393 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f7fbc139b10 0x7f7fbc072600 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- 172.21.15.98:0/48290393 >> 172.21.15.98:0/48290393 conn(0x7f7fbc0ac790 msgr2=0x7f7fbc0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- 172.21.15.98:0/48290393 shutdown_connections 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- 172.21.15.98:0/48290393 wait complete. 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient: monmap: 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.418 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 librados: starting msgr at 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 librados: starting objecter 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 Processor -- start 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- start start 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 librados: setting wanted keys 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 librados: calling monclient init 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient: init 2024-09-11T03:05:33.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient: _add_conns ranks=[5,3,7,6,0,8,2,4,1] 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7fbc139b10 0x7f7fbc1d18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): picked mon.noname-h con 0x7f7fbc139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc139ef0 0x7f7fbc13faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): picked mon.noname-g con 0x7f7fbc139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7fbc13a2d0 0x7f7fbc13ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): picked mon.noname-i con 0x7f7fbc13a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc143460 con 0x7f7fbc139ef0 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f7fbc0a88a0 con 0x7f7fbc139b10 2024-09-11T03:05:33.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f7fbc1435e0 con 0x7f7fbc13a2d0 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc29b0640 10 monclient(hunting): authenticate will time out at 1887.287000s 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7fbc139b10 0x7f7fbc1d18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7fbc13a2d0 0x7f7fbc13ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc139ef0 0x7f7fbc13faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7fbc139b10 0x7f7fbc1d18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:45786/0 (socket says 172.21.15.98:45786) 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 1 -- 172.21.15.98:0/2541897307 learned_addr learned my addr 172.21.15.98:0/2541897307 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 10 monclient(hunting): get_auth_request con 0x7f7fbc139b10 auth_method 0 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 10 monclient(hunting): get_auth_request con 0x7f7fbc13a2d0 auth_method 0 2024-09-11T03:05:33.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 10 monclient(hunting): get_auth_request con 0x7f7fbc139ef0 auth_method 0 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fc0f4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.401+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 10 monclient(hunting): handle_auth_done global_id 74534 payload 824 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7fbc13a2d0 msgr2=0x7f7fbc13ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7fbc13a2d0 0x7f7fbc13ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc139ef0 msgr2=0x7f7fbc13faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc139ef0 0x7f7fbc13faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 1 monclient: found mon.noname-h 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 1 -- 172.21.15.98:0/2541897307 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc13c230 con 0x7f7fbc139b10 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc0f4d640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f7fbc13a2d0 0x7f7fbc13ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc139ef0 0x7f7fbc13faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbbfff640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7fbc139b10 0x7f7fbc1d18d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc13c400 tx=0x7f7fb40196b0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 client.?.objecter ms_handle_connect 0x7f7fbc139b10 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb40bc020 con 0x7f7fbc139b10 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: dump: 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb4003e10 con 0x7f7fbc139b10 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 5 monclient: authenticate success, global_id 74534 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 10 monclient: _renew_subs 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc0aa200 con 0x7f7fbc139b10 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb40044b0 con 0x7f7fbc139b10 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 10 client.74534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: dump: 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fba7fc640 4 set_mon_vals no callback set 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 10 monclient: _renew_subs 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc13e570 con 0x7f7fbc139b10 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 1 librados: init done 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13a9f0 0x7f7fbc13add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc13e1f0 con 0x7f7fbc13a9f0 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13a9f0 0x7f7fbc13add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient: get_auth_request con 0x7f7fbc13a9f0 auth_method 0 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f7fb4004650 con 0x7f7fbc139b10 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7fa803df40 0x7f7fa8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_done global_id 74487 payload 824 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 1 -- 172.21.15.98:0/2541897307 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fbc06e680 con 0x7f7fbc13a9f0 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(632..632 src has 1..632) ==== 6769+0+0 (secure 0 0 0) 0x7f7fb40eaab0 con 0x7f7fbc139b10 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 client.74534.objecter ms_dispatch 0x7f7fbc000ba0 osd_map(632..632 src has 1..632) 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 3 client.74534.objecter handle_osd_map got epochs [632,632] > 0 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 3 client.74534.objecter handle_osd_map decoding full epoch 632 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 20 client.74534.objecter dump_active .. 0 homeless 2024-09-11T03:05:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc0f4d640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7fa803df40 0x7f7fa8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc0f4d640 10 monclient: get_auth_request con 0x7f7fa803df40 auth_method 0 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 client.74534.objecter ms_handle_connect 0x7f7fbc13a9f0 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 client.74534.objecter resend_mon_ops 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fbb7fe640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13a9f0 0x7f7fbc13add0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc13c880 tx=0x7f7fb000f610 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb0019070 con 0x7f7fbc13a9f0 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 monclient: dump: 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fb8ff9640 10 client.74534.objecter ms_handle_connect 0x7f7fa803df40 2024-09-11T03:05:33.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.405+0000 7f7fc0f4d640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7fa803df40 0x7f7fa8040400 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009580 tx=0x7f7fac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.409+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7fb0004030 con 0x7f7fbc13a9f0 2024-09-11T03:05:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.409+0000 7f7fb8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.409+0000 7f7fb8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.409+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13a9f0 msgr2=0x7f7fbc13add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.409+0000 7f7fb8ff9640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13a9f0 0x7f7fbc13add0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc13c880 tx=0x7f7fb000f610 comp rx=0 tx=0).stop 2024-09-11T03:05:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.409+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13a9f0 msgr2=0x7f7fbc13add0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.409+0000 7f7fb8ff9640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13a9f0 0x7f7fbc13add0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc13c880 tx=0x7f7fb000f610 comp rx=0 tx=0).stop 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fc29b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13b310 0x7f7fbc0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fc29b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc06ca10 con 0x7f7fbc13b310 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13b310 0x7f7fbc0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient: get_auth_request con 0x7f7fbc13b310 auth_method 0 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 10 monclient(hunting): handle_auth_done global_id 74496 payload 824 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 1 -- 172.21.15.98:0/2541897307 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7fbc0aa430 con 0x7f7fbc13b310 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fbb7fe640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13b310 0x7f7fbc0a0e60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c10 tx=0x7f7fb00046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fb8ff9640 10 client.74534.objecter ms_handle_connect 0x7f7fbc13b310 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fb8ff9640 10 client.74534.objecter resend_mon_ops 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7fb0019040 con 0x7f7fbc13b310 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fb8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.413+0000 7f7fb8ff9640 10 monclient: dump: 2024-09-11T03:05:33.435 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.436 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7f7fb001e660 con 0x7f7fbc13b310 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fb8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fb8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13b310 msgr2=0x7f7fbc0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fb8ff9640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13b310 0x7f7fbc0a0e60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c10 tx=0x7f7fb00046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fb8ff9640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13b310 msgr2=0x7f7fbc0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fb8ff9640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f7fbc13b310 0x7f7fbc0a0e60 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000c10 tx=0x7f7fb00046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.437 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 290348, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:33.438 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.439 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.440 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:33.441 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.442 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.445 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.446 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:33.447 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.448 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.449 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.450 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.451 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.452 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.453 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.457 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.458 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.459 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:33.460 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 10 librados: watch_flush enter 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 10 librados: watch_flush exit 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 20 client.74534.objecter shutdown clearing up homeless session... 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 10 client.74534.objecter successfully canceled tick 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7fa803df40 msgr2=0x7f7fa8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f7fa803df40 0x7f7fa8040400 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009580 tx=0x7f7fac009190 comp rx=0 tx=0).stop 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 10 monclient: shutdown 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7fbc139b10 msgr2=0x7f7fbc1d18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 --2- 172.21.15.98:0/2541897307 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f7fbc139b10 0x7f7fbc1d18d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc13c400 tx=0x7f7fb40196b0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc0f4d640 1 -- 172.21.15.98:0/2541897307 reap_dead start 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc0f4d640 1 -- 172.21.15.98:0/2541897307 reap_dead start 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 shutdown_connections 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 >> 172.21.15.98:0/2541897307 conn(0x7f7fbc0ac790 msgr2=0x7f7fbc135a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 shutdown_connections 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 -- 172.21.15.98:0/2541897307 wait complete. 2024-09-11T03:05:33.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.417+0000 7f7fc29b0640 1 librados: shutdown 2024-09-11T03:05:33.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:33.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T03:05:33.464 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:05:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient: get_monmap_and_config 2024-09-11T03:05:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient: monmap: 2024-09-11T03:05:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:33.581746+0000 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:33.581746+0000 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:33.580 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:33.581 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 1 Processor -- start 2024-09-11T03:05:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 1 -- start start 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient: init 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient: _add_conns ranks=[9,0,10,5,3,2,6,7,1,8,4] 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f096c0a3690 0x7f096c0a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient(hunting): picked mon.noname-a con 0x7f096c0a3690 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f096c09df80 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f096c0a3fb0 0x7f096c136b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient(hunting): picked mon.noname-f con 0x7f096c0a3fb0 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f096c0a0870 con 0x7f096c09df80 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f096c073d70 con 0x7f096c0a3690 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f096c09fe90 con 0x7f096c0a3fb0 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f09747cc640 10 monclient(hunting): authenticate will time out at 1887.462415s 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f096c0a3690 0x7f096c0a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0971d67640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f096c09df80 0x7f096c136450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56832/0 (socket says 172.21.15.98:56832) 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0971d67640 1 -- 172.21.15.98:0/26319643 learned_addr learned my addr 172.21.15.98:0/26319643 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): get_auth_request con 0x7f096c0a3690 auth_method 0 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient(hunting): handle_auth_done global_id 45387 payload 274 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972d69640 1 --2- 172.21.15.98:0/26319643 >> v2:172.21.15.190:3300/0 conn(0x7f096c0a3fb0 0x7f096c136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 1 -- 172.21.15.98:0/26319643 >> v2:172.21.15.190:3300/0 conn(0x7f096c0a3fb0 msgr2=0x7f096c136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 1 --2- 172.21.15.98:0/26319643 >> v2:172.21.15.190:3300/0 conn(0x7f096c0a3fb0 0x7f096c136b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 1 -- 172.21.15.98:0/26319643 >> v1:172.21.15.98:6789/0 conn(0x7f096c09df80 legacy=0x7f096c136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 1 monclient: found mon.noname-a 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972568640 1 -- 172.21.15.98:0/26319643 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096c09fbb0 con 0x7f096c0a3690 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0971566640 1 -- 172.21.15.98:0/26319643 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1347689032 0 0) 0x7f095c001120 con 0x7f096c09df80 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972d69640 10 monclient: get_auth_request con 0x7f096c0a3fb0 auth_method 0 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0971566640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:05:33.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.577+0000 7f0972d69640 1 --2- 172.21.15.98:0/26319643 >> v2:172.21.15.190:3300/0 conn(0x7f096c0a3fb0 0x7f096c136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 1 --2- 172.21.15.98:0/26319643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3690 0x7f096c0a3a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f096c09f090 tx=0x7f096801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee12e600fd2cc5a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 1 -- 172.21.15.98:0/26319643 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09680b0070 con 0x7f096c0a3690 2024-09-11T03:05:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: dump: 2024-09-11T03:05:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 5 monclient: authenticate success, global_id 45387 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 1 -- 172.21.15.98:0/26319643 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09680aa810 con 0x7f096c0a3690 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 1 -- 172.21.15.98:0/26319643 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09680a9300 con 0x7f096c0a3690 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971566640 10 monclient: dump: 2024-09-11T03:05:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 4 set_mon_vals no callback set 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient: shutdown 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- 172.21.15.98:0/26319643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3690 msgr2=0x7f096c0a3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 --2- 172.21.15.98:0/26319643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3690 0x7f096c0a3a70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f096c09f090 tx=0x7f096801c3d0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- 172.21.15.98:0/26319643 shutdown_connections 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 --2- 172.21.15.98:0/26319643 >> v2:172.21.15.190:3300/0 conn(0x7f096c0a3fb0 0x7f096c136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 --2- 172.21.15.98:0/26319643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3690 0x7f096c0a3a70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- 172.21.15.98:0/26319643 >> 172.21.15.98:0/26319643 conn(0x7f096c0a4780 msgr2=0x7f096c0a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- 172.21.15.98:0/26319643 shutdown_connections 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- 172.21.15.98:0/26319643 wait complete. 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient: monmap: 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:33.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 librados: starting msgr at 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 librados: starting objecter 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 Processor -- start 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- start start 2024-09-11T03:05:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 librados: setting wanted keys 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 librados: calling monclient init 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient: init 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient: _add_conns ranks=[4,3,0,8,1,7,2,6,5] 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f096c09df80 0x7f096c1cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): picked mon.noname-c con 0x7f096c09df80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f096c0a3690 0x7f096c136060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): picked mon.noname-g con 0x7f096c0a3690 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3fb0 0x7f096c13b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): picked mon.noname-a con 0x7f096c0a3fb0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f096c137220 con 0x7f096c0a3fb0 2024-09-11T03:05:33.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f096c1370a0 con 0x7f096c0a3690 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f096c09ca50 con 0x7f096c09df80 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f09747cc640 10 monclient(hunting): authenticate will time out at 1887.467476s 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f096c09df80 0x7f096c1cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3fb0 0x7f096c13b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f096c09df80 0x7f096c1cced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54722/0 (socket says 172.21.15.98:54722) 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 1 -- 172.21.15.98:0/1660047878 learned_addr learned my addr 172.21.15.98:0/1660047878 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 10 monclient(hunting): get_auth_request con 0x7f096c09df80 auth_method 0 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): get_auth_request con 0x7f096c0a3fb0 auth_method 0 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971d67640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f096c0a3690 0x7f096c136060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient(hunting): handle_auth_done global_id 45405 payload 824 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f096c09df80 msgr2=0x7f096c1cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f096c09df80 0x7f096c1cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f096c0a3690 msgr2=0x7f096c136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f096c0a3690 0x7f096c136060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 1 monclient: found mon.noname-a 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972d69640 1 -- 172.21.15.98:0/1660047878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096c1cdde0 con 0x7f096c0a3fb0 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971d67640 10 monclient: get_auth_request con 0x7f096c0a3690 auth_method 0 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0972568640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f096c09df80 0x7f096c1cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.581+0000 7f0971d67640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f096c0a3690 0x7f096c136060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972d69640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3fb0 0x7f096c13b2e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f096400cc90 tx=0x7f096400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 client.?.objecter ms_handle_connect 0x7f096c0a3fb0 2024-09-11T03:05:33.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f096400f8f0 con 0x7f096c0a3fb0 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: dump: 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 5 monclient: authenticate success, global_id 45405 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f096400fad0 con 0x7f096c0a3fb0 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 10 monclient: _renew_subs 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096c11f9e0 con 0x7f096c0a3fb0 2024-09-11T03:05:33.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f096400fc80 con 0x7f096c0a3fb0 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0970d65640 4 set_mon_vals no callback set 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 10 client.45405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: dump: 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.605 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 10 monclient: _renew_subs 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:33.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f096c09d670 con 0x7f096c0a3fb0 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 1 librados: init done 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02d450 0x7f096c02d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f096c06ca10 con 0x7f096c02d450 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f096400f480 con 0x7f096c0a3fb0 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02d450 0x7f096c02d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095003df40 0x7f0950040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient: get_auth_request con 0x7f096c02d450 auth_method 0 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(633..633 src has 1..633) ==== 6769+0+0 (secure 0 0 0) 0x7f09640505f0 con 0x7f096c0a3fb0 2024-09-11T03:05:33.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 client.45405.objecter ms_dispatch 0x7f096c000ba0 osd_map(633..633 src has 1..633) 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 3 client.45405.objecter handle_osd_map got epochs [633,633] > 0 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 3 client.45405.objecter handle_osd_map decoding full epoch 633 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 20 client.45405.objecter dump_active .. 0 homeless 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0971d67640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095003df40 0x7f0950040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0971d67640 10 monclient: get_auth_request con 0x7f095003df40 auth_method 0 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 10 monclient(hunting): handle_auth_done global_id 74543 payload 824 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 1 -- 172.21.15.98:0/1660047878 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f096c068b40 con 0x7f096c02d450 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 client.45405.objecter ms_handle_connect 0x7f096c02d450 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 client.45405.objecter resend_mon_ops 2024-09-11T03:05:33.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0972568640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02d450 0x7f096c02d830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f096801a000 tx=0x7f0968019170 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 client.45405.objecter ms_handle_connect 0x7f095003df40 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0971d67640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095003df40 0x7f0950040400 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7f096c13c9c0 tx=0x7f095c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09680b0040 con 0x7f096c02d450 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.585+0000 7f0962ffd640 10 monclient: dump: 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.609 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.589+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f09680aa810 con 0x7f096c02d450 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.589+0000 7f0962ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.589+0000 7f0962ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.589+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02d450 msgr2=0x7f096c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.589+0000 7f0962ffd640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02d450 0x7f096c02d830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f096801a000 tx=0x7f0968019170 comp rx=0 tx=0).stop 2024-09-11T03:05:33.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.589+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02d450 msgr2=0x7f096c02d830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.589+0000 7f0962ffd640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02d450 0x7f096c02d830 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f09747cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:33.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f09747cc640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02dd70 0x7f096c02e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f09747cc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f096c06e870 con 0x7f096c02dd70 2024-09-11T03:05:33.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02dd70 0x7f096c02e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient: get_auth_request con 0x7f096c02dd70 auth_method 0 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 10 monclient(hunting): handle_auth_done global_id 74552 payload 824 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 1 -- 172.21.15.98:0/1660047878 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f096c09d8a0 con 0x7f096c02dd70 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0972568640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02dd70 0x7f096c02e180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f09680b9000 tx=0x7f09680a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0962ffd640 10 client.45405.objecter ms_handle_connect 0x7f096c02dd70 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0962ffd640 10 client.45405.objecter resend_mon_ops 2024-09-11T03:05:33.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09680b0030 con 0x7f096c02dd70 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0962ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0962ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.593+0000 7f0962ffd640 10 monclient: dump: 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.614 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9435 (secure 0 0 0) 0x7f09680c6070 con 0x7f096c02dd70 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0962ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0962ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02dd70 msgr2=0x7f096c02e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0962ffd640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02dd70 0x7f096c02e180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f09680b9000 tx=0x7f09680a8000 comp rx=0 tx=0).stop 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0962ffd640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02dd70 msgr2=0x7f096c02e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0962ffd640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f096c02dd70 0x7f096c02e180 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f09680b9000 tx=0x7f09680a8000 comp rx=0 tx=0).stop 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 290528, 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:33.616 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:33.619 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:33.620 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.621 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.623 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:33.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:33.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:33.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.633 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:33.635 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.637 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.638 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.639 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.639 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.639 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:33.639 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 10 librados: watch_flush enter 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 10 librados: watch_flush exit 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 20 client.45405.objecter shutdown clearing up homeless session... 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 10 client.45405.objecter successfully canceled tick 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095003df40 msgr2=0x7f0950040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095003df40 0x7f0950040400 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7f096c13c9c0 tx=0x7f095c009040 comp rx=0 tx=0).stop 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 10 monclient: shutdown 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3fb0 msgr2=0x7f096c13b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 --2- 172.21.15.98:0/1660047878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f096c0a3fb0 0x7f096c13b2e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f096400cc90 tx=0x7f096400a650 comp rx=0 tx=0).stop 2024-09-11T03:05:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0972d69640 1 -- 172.21.15.98:0/1660047878 reap_dead start 2024-09-11T03:05:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f0972d69640 1 -- 172.21.15.98:0/1660047878 reap_dead start 2024-09-11T03:05:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 shutdown_connections 2024-09-11T03:05:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 >> 172.21.15.98:0/1660047878 conn(0x7f096c0a4780 msgr2=0x7f096c13dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 shutdown_connections 2024-09-11T03:05:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 -- 172.21.15.98:0/1660047878 wait complete. 2024-09-11T03:05:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.597+0000 7f09747cc640 1 librados: shutdown 2024-09-11T03:05:33.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:33.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T03:05:33.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient: get_monmap_and_config 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient: monmap: 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:33.758456+0000 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:33.758456+0000 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.756 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:33.757 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 1 Processor -- start 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 1 -- start start 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient: init 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient: _add_conns ranks=[10,0,2,1,8,6,4,9,5,7,3] 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fa0200a46a0 0x7fa0200a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient(hunting): picked mon.noname-f con 0x7fa0200a46a0 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0200a4fc0 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02013a040 0x7fa020141c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient(hunting): picked mon.noname-b con 0x7fa02013a040 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0200a17b0 con 0x7fa0200a4fc0 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fa0200a0750 con 0x7fa0200a46a0 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa0200775d0 con 0x7fa02013a040 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa02690e640 10 monclient(hunting): authenticate will time out at 1887.639383s 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa0177fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa0200a4fc0 0x7fa020139930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56842/0 (socket says 172.21.15.98:56842) 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa0177fe640 1 -- 172.21.15.98:0/602958879 learned_addr learned my addr 172.21.15.98:0/602958879 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 1 --2- 172.21.15.98:0/602958879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02013a040 0x7fa020141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 10 monclient(hunting): get_auth_request con 0x7fa02013a040 auth_method 0 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01ffff640 1 --2- 172.21.15.98:0/602958879 >> v2:172.21.15.190:3300/0 conn(0x7fa0200a46a0 0x7fa0200a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa024eab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01ffff640 10 monclient(hunting): get_auth_request con 0x7fa0200a46a0 auth_method 0 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01f7fe640 1 -- 172.21.15.98:0/602958879 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2863357598 0 0) 0x7fa0200a17b0 con 0x7fa0200a4fc0 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01f7fe640 10 monclient(hunting): _negotiate 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01f7fe640 10 monclient(hunting): my global_id is 45414 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.753+0000 7fa01f7fe640 1 -- 172.21.15.98:0/602958879 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa00c003530 con 0x7fa0200a4fc0 2024-09-11T03:05:33.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 10 monclient(hunting): handle_auth_done global_id 94160 payload 274 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 1 -- 172.21.15.98:0/602958879 >> v2:172.21.15.190:3300/0 conn(0x7fa0200a46a0 msgr2=0x7fa0200a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 1 --2- 172.21.15.98:0/602958879 >> v2:172.21.15.190:3300/0 conn(0x7fa0200a46a0 0x7fa0200a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 1 -- 172.21.15.98:0/602958879 >> v1:172.21.15.98:6789/0 conn(0x7fa0200a4fc0 legacy=0x7fa020139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 1 monclient: found mon.noname-b 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:05:33.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 1 -- 172.21.15.98:0/602958879 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0200a0200 con 0x7fa02013a040 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa024eab640 1 --2- 172.21.15.98:0/602958879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02013a040 0x7fa020141c00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa018009d90 tx=0x7fa01801c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9891654100d9efd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 1 -- 172.21.15.98:0/602958879 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0180a6910 con 0x7fa02013a040 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: dump: 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.770 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 1 -- 172.21.15.98:0/602958879 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0180a6ad0 con 0x7fa02013a040 2024-09-11T03:05:33.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 1 -- 172.21.15.98:0/602958879 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0180a6c80 con 0x7fa02013a040 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 5 monclient: authenticate success, global_id 94160 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 4 set_mon_vals no callback set 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa01f7fe640 10 monclient: dump: 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.772 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 10 monclient: shutdown 2024-09-11T03:05:33.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 -- 172.21.15.98:0/602958879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02013a040 msgr2=0x7fa020141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 --2- 172.21.15.98:0/602958879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02013a040 0x7fa020141c00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa018009d90 tx=0x7fa01801c7d0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 -- 172.21.15.98:0/602958879 shutdown_connections 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 --2- 172.21.15.98:0/602958879 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa02013a040 0x7fa020141c00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 --2- 172.21.15.98:0/602958879 >> v2:172.21.15.190:3300/0 conn(0x7fa0200a46a0 0x7fa0200a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 -- 172.21.15.98:0/602958879 >> 172.21.15.98:0/602958879 conn(0x7fa02006ed00 msgr2=0x7fa02006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 -- 172.21.15.98:0/602958879 shutdown_connections 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 -- 172.21.15.98:0/602958879 wait complete. 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 10 monclient: monmap: 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:33.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 librados: starting msgr at 2024-09-11T03:05:33.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.757+0000 7fa02690e640 1 librados: starting objecter 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 Processor -- start 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 -- start start 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 librados: setting wanted keys 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 librados: calling monclient init 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient: init 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient: _add_conns ranks=[6,5,7,2,8,0,3,1,4] 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa0200a46a0 0x7fa0201d1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): picked mon.noname-d con 0x7fa0200a46a0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0200a4fc0 0x7fa0201d1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): picked mon.noname-h con 0x7fa0200a4fc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013a040 0x7fa0201d22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): picked mon.noname-i con 0x7fa02013a040 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa0200779d0 con 0x7fa0200a4fc0 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fa020077850 con 0x7fa0200a46a0 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa020144960 con 0x7fa02013a040 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): authenticate will time out at 1887.644629s 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa0200a46a0 0x7fa0201d1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa0200a46a0 0x7fa0201d1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:57048/0 (socket says 172.21.15.98:57048) 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 -- 172.21.15.98:0/2166010339 learned_addr learned my addr 172.21.15.98:0/2166010339 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): get_auth_request con 0x7fa0200a46a0 auth_method 0 2024-09-11T03:05:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0200a4fc0 0x7fa0201d1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa024eab640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013a040 0x7fa0201d22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): get_auth_request con 0x7fa0200a4fc0 auth_method 0 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa024eab640 10 monclient(hunting): get_auth_request con 0x7fa02013a040 auth_method 0 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa024eab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa024eab640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa024eab640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient(hunting): handle_auth_done global_id 84516 payload 824 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013a040 msgr2=0x7fa0201d22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013a040 0x7fa0201d22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0200a4fc0 msgr2=0x7fa0201d1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0200a4fc0 0x7fa0201d1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 monclient: found mon.noname-d 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 -- 172.21.15.98:0/2166010339 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa020141240 con 0x7fa0200a46a0 2024-09-11T03:05:33.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa0200a4fc0 0x7fa0201d1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01ffff640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa0200a46a0 0x7fa0201d1870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa01000e960 tx=0x7fa01000ee20 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 client.?.objecter ms_handle_connect 0x7fa0200a46a0 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa010017070 con 0x7fa0200a46a0 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: dump: 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa01000ce70 con 0x7fa0200a46a0 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0100138e0 con 0x7fa0200a46a0 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01effd640 4 set_mon_vals no callback set 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 monclient: dump: 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.782 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 5 monclient: authenticate success, global_id 84516 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient: _renew_subs 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:33.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa020141520 con 0x7fa0200a46a0 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 client.84516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient: _renew_subs 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa020141e50 con 0x7fa0200a46a0 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 librados: init done 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013d7c0 0x7fa02013dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa0200a4e20 con 0x7fa02013d7c0 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa010013470 con 0x7fa0200a46a0 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013d7c0 0x7fa02013dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa00003df40 0x7fa000040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient: get_auth_request con 0x7fa02013d7c0 auth_method 0 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(633..633 src has 1..633) ==== 6769+0+0 (secure 0 0 0) 0x7fa01000f040 con 0x7fa0200a46a0 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 10 client.84516.objecter ms_dispatch 0x7fa020000ba0 osd_map(633..633 src has 1..633) 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 3 client.84516.objecter handle_osd_map got epochs [633,633] > 0 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 3 client.84516.objecter handle_osd_map decoding full epoch 633 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa01d7fa640 20 client.84516.objecter dump_active .. 0 homeless 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa024eab640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa00003df40 0x7fa000040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa024eab640 10 monclient: get_auth_request con 0x7fa00003df40 auth_method 0 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 10 monclient(hunting): handle_auth_done global_id 20248 payload 824 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.761+0000 7fa0177fe640 1 -- 172.21.15.98:0/2166010339 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa020141a20 con 0x7fa02013d7c0 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 client.84516.objecter ms_handle_connect 0x7fa02013d7c0 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 client.84516.objecter resend_mon_ops 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa0177fe640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013d7c0 0x7fa02013dba0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fa02013f830 tx=0x7fa00800eb90 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa008005d30 con 0x7fa02013d7c0 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 monclient: dump: 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.786 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 client.84516.objecter ms_handle_connect 0x7fa00003df40 2024-09-11T03:05:33.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa024eab640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa00003df40 0x7fa000040400 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7fa0180af000 tx=0x7fa0180a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa008018070 con 0x7fa02013d7c0 2024-09-11T03:05:33.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:33.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013d7c0 msgr2=0x7fa02013dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013d7c0 0x7fa02013dba0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fa02013f830 tx=0x7fa00800eb90 comp rx=0 tx=0).stop 2024-09-11T03:05:33.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013d7c0 msgr2=0x7fa02013dba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.765+0000 7fa01d7fa640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013d7c0 0x7fa02013dba0 secure :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fa02013f830 tx=0x7fa00800eb90 comp rx=0 tx=0).stop 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa02690e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa02690e640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013e0e0 0x7fa02009d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa02690e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fa02006ca10 con 0x7fa02013e0e0 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013e0e0 0x7fa02009d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 10 monclient: get_auth_request con 0x7fa02013e0e0 auth_method 0 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.769+0000 7fa0177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa0177fe640 10 monclient(hunting): handle_auth_done global_id 20257 payload 824 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa0177fe640 1 -- 172.21.15.98:0/2166010339 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa0200a4a80 con 0x7fa02013e0e0 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 10 client.84516.objecter ms_handle_connect 0x7fa02013e0e0 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa0177fe640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013e0e0 0x7fa02009d350 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fa008017000 tx=0x7fa008005800 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 10 client.84516.objecter resend_mon_ops 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa008007540 con 0x7fa02013e0e0 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 10 monclient: dump: 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.791 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9270 (secure 0 0 0) 0x7fa0080070d0 con 0x7fa02013e0e0 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013e0e0 msgr2=0x7fa02009d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013e0e0 0x7fa02009d350 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fa008017000 tx=0x7fa008005800 comp rx=0 tx=0).stop 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013e0e0 msgr2=0x7fa02009d350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa01d7fa640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fa02013e0e0 0x7fa02009d350 secure :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fa008017000 tx=0x7fa008005800 comp rx=0 tx=0).stop 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:33.793 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 949550, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.794 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 6, 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.795 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.796 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:33.797 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:33.798 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:33.799 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.802 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.803 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.804 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.805 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:33.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.809 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:33.811 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:33.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:33.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:33.815 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:33.816 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:33.816 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T03:05:33.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:33.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:33.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:33.816 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:33.816 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 10 librados: watch_flush enter 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 10 librados: watch_flush exit 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 20 client.84516.objecter shutdown clearing up homeless session... 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 10 client.84516.objecter successfully canceled tick 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa00003df40 msgr2=0x7fa000040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa00003df40 0x7fa000040400 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7fa0180af000 tx=0x7fa0180a7000 comp rx=0 tx=0).stop 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 10 monclient: shutdown 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa0200a46a0 msgr2=0x7fa0201d1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 --2- 172.21.15.98:0/2166010339 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fa0200a46a0 0x7fa0201d1870 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa01000e960 tx=0x7fa01000ee20 comp rx=0 tx=0).stop 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa024eab640 1 -- 172.21.15.98:0/2166010339 reap_dead start 2024-09-11T03:05:33.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa024eab640 1 -- 172.21.15.98:0/2166010339 reap_dead start 2024-09-11T03:05:33.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 shutdown_connections 2024-09-11T03:05:33.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 >> 172.21.15.98:0/2166010339 conn(0x7fa02006ed00 msgr2=0x7fa0200739d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 shutdown_connections 2024-09-11T03:05:33.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 -- 172.21.15.98:0/2166010339 wait complete. 2024-09-11T03:05:33.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.773+0000 7fa02690e640 1 librados: shutdown 2024-09-11T03:05:33.819 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:33.819 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T03:05:33.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T03:05:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.933+0000 7fe335167640 10 monclient: get_monmap_and_config 2024-09-11T03:05:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.933+0000 7fe335167640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.933+0000 7fe335167640 10 monclient: monmap: 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:33.939946+0000 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:33.939946+0000 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.938 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:33.939 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:33.940 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 1 Processor -- start 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 1 -- start start 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient: init 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient: _add_conns ranks=[8,4,1,2,0,7,9,6,3,5,10] 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe33009b500 0x7fe33009b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient(hunting): picked mon.noname-e con 0x7fe33009b500 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe33009be20 0x7fe3300a3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient(hunting): picked mon.noname-c con 0x7fe33009be20 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe3300a3750 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe33009d5d0 con 0x7fe3300a3750 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe3300a4cc0 con 0x7fe33009be20 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe33009c5e0 con 0x7fe33009b500 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient(hunting): authenticate will time out at 1887.820800s 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe33009be20 0x7fe3300a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe33009b500 0x7fe33009b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe33009be20 0x7fe3300a3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54734/0 (socket says 172.21.15.98:54734) 2024-09-11T03:05:33.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 -- 172.21.15.98:0/142834982 learned_addr learned my addr 172.21.15.98:0/142834982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): get_auth_request con 0x7fe33009be20 auth_method 0 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe33009b500 0x7fe33009b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55790/0 (socket says 172.21.15.98:55790) 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 10 monclient(hunting): get_auth_request con 0x7fe33009b500 auth_method 0 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient(hunting): handle_auth_done global_id 24754 payload 274 2024-09-11T03:05:33.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 -- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe33009b500 msgr2=0x7fe33009b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:33.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 --2- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe33009b500 0x7fe33009b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 -- 172.21.15.98:0/142834982 >> v1:172.21.15.190:6789/0 conn(0x7fe3300a3750 legacy=0x7fe33013e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 monclient: found mon.noname-c 2024-09-11T03:05:33.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:33.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 -- 172.21.15.98:0/142834982 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33009c8c0 con 0x7fe33009be20 2024-09-11T03:05:33.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32ed9d640 1 --2- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe33009b500 0x7fe33009b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe327fff640 1 --2- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe33009be20 0x7fe3300a3210 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fe3180098a0 tx=0x7fe31801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7d37b05331557b53 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 1 -- 172.21.15.98:0/142834982 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3180b1070 con 0x7fe33009be20 2024-09-11T03:05:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: dump: 2024-09-11T03:05:33.949 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.950 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.951 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.952 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 5 monclient: authenticate success, global_id 24754 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 1 -- 172.21.15.98:0/142834982 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3180a8050 con 0x7fe33009be20 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe335167640 4 set_mon_vals no callback set 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 1 -- 172.21.15.98:0/142834982 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3180abcc0 con 0x7fe33009be20 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:33.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.937+0000 7fe32e59c640 10 monclient: dump: 2024-09-11T03:05:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.954 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.955 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.956 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: shutdown 2024-09-11T03:05:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe33009be20 msgr2=0x7fe3300a3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 --2- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe33009be20 0x7fe3300a3210 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fe3180098a0 tx=0x7fe31801c440 comp rx=0 tx=0).stop 2024-09-11T03:05:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- 172.21.15.98:0/142834982 shutdown_connections 2024-09-11T03:05:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 --2- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe33009be20 0x7fe3300a3210 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 --2- 172.21.15.98:0/142834982 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe33009b500 0x7fe33009b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- 172.21.15.98:0/142834982 >> 172.21.15.98:0/142834982 conn(0x7fe3300a86f0 msgr2=0x7fe3300a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- 172.21.15.98:0/142834982 shutdown_connections 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- 172.21.15.98:0/142834982 wait complete. 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: build_initial_monmap 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: monmap: 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:33.958 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:33.959 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 librados: starting msgr at 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 librados: starting objecter 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 Processor -- start 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- start start 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 librados: setting wanted keys 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 librados: calling monclient init 2024-09-11T03:05:33.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: init 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: _add_conns ranks=[0,4,2,6,8,5,1,7,3] 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33009b500 0x7fe33012d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): picked mon.noname-a con 0x7fe33009b500 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3300a3750 0x7fe33012db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): picked mon.noname-c con 0x7fe3300a3750 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe33012e060 0x7fe33009a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): picked mon.noname-b con 0x7fe33012e060 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:33.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe330068e00 con 0x7fe33009b500 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3300a1c30 con 0x7fe33012e060 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe33012fea0 con 0x7fe3300a3750 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient(hunting): authenticate will time out at 1887.826257s 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3300a3750 0x7fe33012db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3300a3750 0x7fe33012db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54746/0 (socket says 172.21.15.98:54746) 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 -- 172.21.15.98:0/2182610603 learned_addr learned my addr 172.21.15.98:0/2182610603 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): get_auth_request con 0x7fe3300a3750 auth_method 0 2024-09-11T03:05:33.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33009b500 0x7fe33012d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 10 monclient(hunting): get_auth_request con 0x7fe33009b500 auth_method 0 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe33012e060 0x7fe33009a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 10 monclient(hunting): get_auth_request con 0x7fe33012e060 auth_method 0 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient(hunting): handle_auth_done global_id 24763 payload 824 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 -- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe33012e060 msgr2=0x7fe33009a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe33012e060 0x7fe33009a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 -- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33009b500 msgr2=0x7fe33012d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33009b500 0x7fe33012d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 monclient: found mon.noname-c 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 -- 172.21.15.98:0/2182610603 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33012f8e0 con 0x7fe3300a3750 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32ed9d640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33009b500 0x7fe33012d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe327fff640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3300a3750 0x7fe33012db20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fe3180098a0 tx=0x7fe3180191e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 client.?.objecter ms_handle_connect 0x7fe3300a3750 2024-09-11T03:05:33.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 1 -- 172.21.15.98:0/2182610603 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3180ba070 con 0x7fe3300a3750 2024-09-11T03:05:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:33.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: dump: 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.973 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 1 -- 172.21.15.98:0/2182610603 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe318003c10 con 0x7fe3300a3750 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 1 -- 172.21.15.98:0/2182610603 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3180b1040 con 0x7fe3300a3750 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 5 monclient: authenticate success, global_id 24763 2024-09-11T03:05:33.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: _renew_subs 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33012fbc0 con 0x7fe3300a3750 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe32d59a640 4 set_mon_vals no callback set 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe3277fe640 10 monclient: dump: 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:33.975 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 client.24763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:33.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: _renew_subs 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33012f600 con 0x7fe3300a3750 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.941+0000 7fe335167640 1 librados: init done 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe335167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe335167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe335167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe330073c10 con 0x7fe3300a3750 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 1 -- 172.21.15.98:0/2182610603 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe3180037a0 con 0x7fe3300a3750 2024-09-11T03:05:33.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe31003df70 0x7fe310040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 1 -- 172.21.15.98:0/2182610603 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(633..633 src has 1..633) ==== 6769+0+0 (secure 0 0 0) 0x7fe3180e8aa0 con 0x7fe3300a3750 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 10 client.24763.objecter ms_dispatch 0x7fe330000ba0 osd_map(633..633 src has 1..633) 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 3 client.24763.objecter handle_osd_map got epochs [633,633] > 0 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 3 client.24763.objecter handle_osd_map decoding full epoch 633 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 20 client.24763.objecter dump_active .. 0 homeless 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe32ed9d640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe31003df70 0x7fe310040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe32ed9d640 10 monclient: get_auth_request con 0x7fe31003df70 auth_method 0 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe3277fe640 10 client.24763.objecter ms_handle_connect 0x7fe31003df70 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.945+0000 7fe32ed9d640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe31003df70 0x7fe310040430 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7fe320009580 tx=0x7fe320009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.949+0000 7fe3277fe640 1 -- 172.21.15.98:0/2182610603 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe318016150 con 0x7fe3300a3750 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.949+0000 7fe3277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:33.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:33.949+0000 7fe3277fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.185+0000 7fe335167640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T03:05:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.185+0000 7fe335167640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T03:05:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.185+0000 7fe335167640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:34.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.185+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe33009b8e0 con 0x7fe3300a3750 2024-09-11T03:05:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.189+0000 7fe3277fe640 1 -- 172.21.15.98:0/2182610603 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe3180acc10 con 0x7fe3300a3750 2024-09-11T03:05:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.189+0000 7fe3277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T03:05:34.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.189+0000 7fe3277fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:34.196 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.197 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.197 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=9,logm=60} crc {auth=652782755,config=692619927,health=3977267314,kv=1009120483,logm=281754466}) 2024-09-11T03:05:34.197 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=4195894354,config=692619927,health=4196208778,kv=3991532845,logm=3441683775}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=11,kv=9,logm=60} crc {auth=652782755,config=692619927,health=1383553553,kv=4292987402,logm=434191996}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=1792931416,config=692619927,health=2180136440,kv=3991532845,logm=1824672188}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.198+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=3648772570,config=692619927,health=4196208778,kv=3991532845,logm=1678817324}) 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.198 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.199 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=15,config=1,health=11,kv=10,logm=63} crc {auth=2280776741,config=1228700967,health=1383553553,kv=1766424844,logm=4039458825}) 2024-09-11T03:05:34.199 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.199 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.199 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=16,config=1,health=12,kv=10,logm=61} crc {auth=1715206645,config=1228700967,health=2180136440,kv=3991532845,logm=4270274478}) 2024-09-11T03:05:34.199 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.199 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1477883800,config=692619927,health=4196208778,kv=3991532845,logm=1782539979}) 2024-09-11T03:05:34.199 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.202+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,health=10,kv=10,logm=63} crc {auth=285961423,health=409166363,kv=3991532845,logm=407426542}) 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 10 librados: watch_flush enter 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 10 librados: watch_flush exit 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 20 client.24763.objecter shutdown clearing up homeless session... 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 10 client.24763.objecter successfully canceled tick 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe31003df70 msgr2=0x7fe310040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe31003df70 0x7fe310040430 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7fe320009580 tx=0x7fe320009190 comp rx=0 tx=0).stop 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 10 monclient: shutdown 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3300a3750 msgr2=0x7fe33012db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:34.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3300a3750 0x7fe33012db20 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fe3180098a0 tx=0x7fe3180191e0 comp rx=0 tx=0).stop 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 shutdown_connections 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe31003df70 0x7fe310040430 unknown :-1 s=CLOSED pgs=3672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe33012e060 0x7fe33009a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3300a3750 0x7fe33012db20 secure :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fe3180098a0 tx=0x7fe3180191e0 comp rx=0 tx=0).stop 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 --2- 172.21.15.98:0/2182610603 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe33009b500 0x7fe33012d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 >> 172.21.15.98:0/2182610603 conn(0x7fe3300a86f0 msgr2=0x7fe33013ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 shutdown_connections 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 -- 172.21.15.98:0/2182610603 wait complete. 2024-09-11T03:05:34.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:34.193+0000 7fe335167640 1 librados: shutdown 2024-09-11T03:05:34.206 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.206 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.206 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3043226383}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=362605157}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=77560378}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1218066037}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1594872540}) 2024-09-11T03:05:34.207 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2858064370}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1694739373}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3147206319}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.206+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1119565310}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622898576}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.208 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487939811}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661216045}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2257020075}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1771667550}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.209 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1365619146}) 2024-09-11T03:05:34.210 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.210 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.210 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1116639944}) 2024-09-11T03:05:34.210 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.210 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3857011244}) 2024-09-11T03:05:34.210 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.210+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2501965106}) 2024-09-11T03:05:34.212 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159654126}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590404454}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3146648092}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.214+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=724742309}) 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.213 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3426462982}) 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=835580260}) 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1512406598}) 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1984920619}) 2024-09-11T03:05:34.214 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.218+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1596305665}) 2024-09-11T03:05:34.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=277403425}) 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752800687}) 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.222+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3686669114}) 2024-09-11T03:05:34.221 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1127577227}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3903737295}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2120647653}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1443515019}) 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.222 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2736588503}) 2024-09-11T03:05:34.223 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.226+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2863989935}) 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287951429}) 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579763755}) 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.226 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929738372}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3466247303}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2972878781}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3510985191}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=603839250}) 2024-09-11T03:05:34.227 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.228 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3270689144}) 2024-09-11T03:05:34.228 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.230+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=780196549}) 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834687308}) 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703096634}) 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2301584005}) 2024-09-11T03:05:34.230 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=767011452}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1529284128}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=179653113}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3218252354}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4017361108}) 2024-09-11T03:05:34.231 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.234+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1258300672}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242954286}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856286374}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769822266}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2656583040}) 2024-09-11T03:05:34.235 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037646283}) 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3231944916}) 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3098813970}) 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1446968265}) 2024-09-11T03:05:34.236 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.238+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3352660016}) 2024-09-11T03:05:34.238 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314278766}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3890989754}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3992669453}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2283130013}) 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.239 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3205483172}) 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1969112173}) 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2566054899}) 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3511507652}) 2024-09-11T03:05:34.240 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.242+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=96314223}) 2024-09-11T03:05:34.243 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.243 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.243 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236551471}) 2024-09-11T03:05:34.243 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=754649984}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190760431}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370533755}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4215866523}) 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.244 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.245 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2309563724}) 2024-09-11T03:05:34.245 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.245 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.245 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2506399942}) 2024-09-11T03:05:34.245 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.245 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=883492621}) 2024-09-11T03:05:34.245 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.246+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1295242780}) 2024-09-11T03:05:34.248 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.250+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.248 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.250+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.248 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.250+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1997724133}) 2024-09-11T03:05:34.248 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.250+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.248 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.250+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.248 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370964567}) 2024-09-11T03:05:34.248 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2044224936}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=371024980}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2285699747}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1828131096}) 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.249 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.250 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3186472157}) 2024-09-11T03:05:34.250 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.250 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752307827}) 2024-09-11T03:05:34.250 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=221057287}) 2024-09-11T03:05:34.251 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.251 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=867905824}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223995863}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935536941}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1030500196}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.252 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1303074203}) 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1471689550}) 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1032407374}) 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4241593721}) 2024-09-11T03:05:34.253 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.254+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=413299947}) 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1437088740}) 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3361951023}) 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=526113598}) 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.255 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1117752574}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=379936650}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3818321447}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1574266401}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3349190419}) 2024-09-11T03:05:34.256 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.258+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4084254165}) 2024-09-11T03:05:34.258 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.258 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.258 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344133612}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2358065767}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2579866103}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3596046341}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1618413586}) 2024-09-11T03:05:34.259 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2664401344}) 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3640732273}) 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2922423141}) 2024-09-11T03:05:34.260 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.262+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3127045171}) 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460951911}) 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=331745956}) 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594374490}) 2024-09-11T03:05:34.262 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2001558154}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3400171747}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1455976869}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1122180522}) 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.263 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=577255802}) 2024-09-11T03:05:34.264 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.266+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=407743392}) 2024-09-11T03:05:34.265 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.265 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.265 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303279858}) 2024-09-11T03:05:34.265 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.265 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.265 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2785302479}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335399417}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=574990627}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=368792536}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=849530294}) 2024-09-11T03:05:34.266 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.267 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.267 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3740390982}) 2024-09-11T03:05:34.267 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.267 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=831596507}) 2024-09-11T03:05:34.267 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.270+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1851949230}) 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803312336}) 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=49499871}) 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134237351}) 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.269 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1033297812}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1906931149}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1063059673}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=920642337}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3120823724}) 2024-09-11T03:05:34.270 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.274+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1846220578}) 2024-09-11T03:05:34.272 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384286468}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2265284488}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2621867636}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1964741356}) 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.273 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1085538516}) 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2575856510}) 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2035682752}) 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=997832829}) 2024-09-11T03:05:34.274 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1832054655}) 2024-09-11T03:05:34.275 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.275 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.275 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988534841}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3932782899}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2123890180}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=315134691}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.276 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2238453550}) 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1288064705}) 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1958948767}) 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1525307059}) 2024-09-11T03:05:34.277 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.278+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1824026054}) 2024-09-11T03:05:34.278 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.278 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=622769393}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1239498937}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107949951}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=922251043}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.279 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3760322738}) 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=502285263}) 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=844414333}) 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1763408304}) 2024-09-11T03:05:34.280 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.282+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1912996198}) 2024-09-11T03:05:34.281 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883067948}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1065183644}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2408860063}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2867367804}) 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.282 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=764026242}) 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=701215029}) 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3877176414}) 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2465169185}) 2024-09-11T03:05:34.283 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.286+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=509242108}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160694928}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190283216}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899835486}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=409694262}) 2024-09-11T03:05:34.286 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3282050262}) 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1033863970}) 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=908396534}) 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2591650751}) 2024-09-11T03:05:34.287 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.290+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2069879886}) 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577739310}) 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=477242179}) 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.289 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037269531}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3362797913}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=454623969}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3191227983}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=950349141}) 2024-09-11T03:05:34.290 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.291 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1072677716}) 2024-09-11T03:05:34.291 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660036029}) 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485500269}) 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774950901}) 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.293 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.294+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1318170228}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=358243771}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2563152912}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2226999049}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1948196093}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1812907303}) 2024-09-11T03:05:34.294 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3828402674}) 2024-09-11T03:05:34.296 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955770747}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3988315512}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1764908607}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1686762114}) 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.297 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.298+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2291528419}) 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=452850896}) 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=98458461}) 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2409589937}) 2024-09-11T03:05:34.298 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4190362275}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2908806834}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667977040}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=302080894}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.301 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3719712164}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2992283414}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3033942958}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4206944783}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1083561742}) 2024-09-11T03:05:34.302 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.302+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1804217459}) 2024-09-11T03:05:34.306 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101344529}) 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1404381279}) 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=650826693}) 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.307 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1964583922}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=963659502}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=207950511}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1940602258}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=356353959}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.306+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4013486587}) 2024-09-11T03:05:34.308 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=328976821}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=60435581}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201624175}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2017671697}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.309 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=42164285}) 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2311502301}) 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4144753136}) 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2578273974}) 2024-09-11T03:05:34.310 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.310+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1749013401}) 2024-09-11T03:05:34.316 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.316 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.316 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349201535}) 2024-09-11T03:05:34.316 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.316 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330691955}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011714984}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=548492616}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3898622143}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.317 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3135304870}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1039945511}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=607839721}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.314+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631125391}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833235973}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052206964}) 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.318 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1439646353}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3407462663}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2659396087}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3003457733}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2047330312}) 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.319 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=907800530}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=586472369}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.318+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946421133}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3358485610}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219981669}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=960946795}) 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.320 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=491237611}) 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3176226338}) 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=42540800}) 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1194239106}) 2024-09-11T03:05:34.321 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4079553365}) 2024-09-11T03:05:34.325 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251311937}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2323522781}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083209619}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=796678514}) 2024-09-11T03:05:34.326 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2934736409}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2697703569}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1977837709}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3384620250}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.322+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=816040594}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.327 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147250927}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4041622641}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2721647651}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=348275155}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3091344950}) 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.328 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1020334378}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2123428034}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=407133400}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.326+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3105367021}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506923307}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.329 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639736616}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637596364}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3713027384}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3322731271}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3352217896}) 2024-09-11T03:05:34.330 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.331 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.331 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=617248256}) 2024-09-11T03:05:34.331 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.331 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3852930608}) 2024-09-11T03:05:34.331 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3109760672}) 2024-09-11T03:05:34.336 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.336 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.336 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443115655}) 2024-09-11T03:05:34.336 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.336 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.336 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801397469}) 2024-09-11T03:05:34.336 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376094106}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.330+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1506934011}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1029932480}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4196979963}) 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.337 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1785970761}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2788173875}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=549789361}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2913638542}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1414123819}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2104428531}) 2024-09-11T03:05:34.338 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3254796096}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1887085086}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4231116986}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=581915126}) 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.339 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2955898242}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.334+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4051951336}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3777294224}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1504631988}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1951638307}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3595746614}) 2024-09-11T03:05:34.340 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1513637899}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=545860865}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1037694688}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=588205286}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.338+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=671613858}) 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.341 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255935312}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514315150}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2715715350}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=158985152}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2740957827}) 2024-09-11T03:05:34.342 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4089967118}) 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2413019701}) 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3720719830}) 2024-09-11T03:05:34.343 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1995921100}) 2024-09-11T03:05:34.348 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.348 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731033979}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2488825209}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=454351256}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=838356717}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.349 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=825783108}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=142451517}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=330783861}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2510197246}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.342+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1697331157}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721051082}) 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.350 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481935830}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1801909808}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2077333863}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4293274630}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.351 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2339745783}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4120000787}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3780937668}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.346+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3691907870}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225951994}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4262642474}) 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.352 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=978695576}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2142307640}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=162540663}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2482358623}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.353 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1763794283}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3436846383}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3343263115}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004122373}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532133485}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398106877}) 2024-09-11T03:05:34.354 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=377489302}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=798703903}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=319098300}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=847558980}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2322824345}) 2024-09-11T03:05:34.355 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.350+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1995306431}) 2024-09-11T03:05:34.364 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.364 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057760422}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982730295}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757168159}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1085287763}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.365 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=599940505}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2768955750}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1316267213}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2202692961}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.354+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=348486544}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014528222}) 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.366 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516936152}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712045447}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4085782095}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498946404}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.367 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=181652927}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4112367454}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3131682043}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.358+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2727409436}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600346096}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995338996}) 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.368 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557905857}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1315152146}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2981885129}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4271930517}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3428483849}) 2024-09-11T03:05:34.369 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372903950}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201102331}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366116946}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011528933}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577189727}) 2024-09-11T03:05:34.370 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4082137921}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1570648220}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1221557958}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1791477021}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2941493948}) 2024-09-11T03:05:34.371 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.362+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2801462481}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3493880601}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3950807672}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2889472141}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2729120972}) 2024-09-11T03:05:34.372 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3676973030}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2274682013}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2564814689}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267422071}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4272151691}) 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.366+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.373 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=204796154}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088493259}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1310315878}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2076486606}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.374 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3876223286}) 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1696136260}) 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1460032452}) 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1304032244}) 2024-09-11T03:05:34.375 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3404578668}) 2024-09-11T03:05:34.403 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.403 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.403 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102769702}) 2024-09-11T03:05:34.403 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.403 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.403 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=108164063}) 2024-09-11T03:05:34.403 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206260963}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=901267526}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=480967149}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=421742927}) 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.404 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=564664970}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446702765}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.370+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3254410115}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982291057}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254804476}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337785974}) 2024-09-11T03:05:34.405 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2655573971}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2686845892}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3456634930}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1410462387}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3346027730}) 2024-09-11T03:05:34.406 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=542570261}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315902359}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=94065809}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2251378430}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2276362491}) 2024-09-11T03:05:34.407 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1876524134}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3016532091}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=324775490}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2536430986}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.374+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2675789535}) 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.408 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528770657}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625312080}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192948019}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1883443254}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.409 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2785410318}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1732432292}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2339170936}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550943092}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.378+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2201353240}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3509434409}) 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.410 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161337650}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=394550638}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1661756876}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=932860832}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.411 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1823933863}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=742454342}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3871666459}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4173359847}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507075722}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465199710}) 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.412 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4144013704}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1139925975}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3832058463}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=510401606}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.413 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1170806444}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=871673221}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.382+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4234131957}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695364463}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584831977}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255554768}) 2024-09-11T03:05:34.414 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2991955916}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=815094622}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1638269232}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1679158872}) 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.415 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2641687350}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1822756182}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4238610225}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.386+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272427846}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=193008260}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=187952209}) 2024-09-11T03:05:34.416 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3812185810}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2977581418}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3743705937}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=909773678}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3856541126}) 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.417 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2954998487}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050643376}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669350787}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=114510245}) 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.418 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=877897347}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=740837578}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3416753202}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3627173740}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.390+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3564786442}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200487065}) 2024-09-11T03:05:34.419 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3423548688}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=416255876}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2611360581}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3174122018}) 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.420 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2510513804}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3740709749}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=142278313}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2273771079}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2035035052}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.421 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3082164835}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=102077060}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2624702009}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=824672416}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3301485380}) 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.422 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=946108197}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3056425159}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.394+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2726841353}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985573367}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2871480321}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.423 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4198393422}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=438804630}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2871033543}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=833900909}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2666915283}) 2024-09-11T03:05:34.424 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1686715273}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2227412895}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955435161}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505798163}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4118984176}) 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.425 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=692378762}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3698606199}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=61419490}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1164367320}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1223726635}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.398+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1183418797}) 2024-09-11T03:05:34.426 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=350433375}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1730161283}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469632725}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=201245271}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.427 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2512614478}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4219708104}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1926470525}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2574865843}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2904021645}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068051370}) 2024-09-11T03:05:34.428 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502472096}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1209055134}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887327380}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3480657889}) 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.429 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3361292698}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4015358449}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1715455171}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.402+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3685883567}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2754317594}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2321034484}) 2024-09-11T03:05:34.430 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3972223947}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3012110336}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3243355479}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=101991878}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.431 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3940453399}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=547414750}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4074972630}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096149646}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3371147748}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=57864665}) 2024-09-11T03:05:34.432 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3836159094}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1426500394}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3157392484}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4080111402}) 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.433 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1632534926}) 2024-09-11T03:05:34.434 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.406+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1186581735}) 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502784441}) 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398483415}) 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3165433103}) 2024-09-11T03:05:34.565 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3002513754}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2912294890}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2631025950}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3028200366}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=633295066}) 2024-09-11T03:05:34.566 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.410+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4274506247}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262069902}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2886226127}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4165033101}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3233853054}) 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.567 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3459844864}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=677097636}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=330311919}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1905220543}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1658337131}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3618347536}) 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.568 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1289466187}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714493934}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1181290904}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2911190417}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.569 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3176362880}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3947863827}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3331386449}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.414+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1869874650}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3313804989}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255321801}) 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.570 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236720141}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2644588962}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3859119036}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2249951685}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=417398085}) 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.571 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550642413}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1324356740}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646573533}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2324661184}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699379006}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3796529775}) 2024-09-11T03:05:34.572 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1890911009}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1304686103}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=764904896}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.418+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4181515820}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=325488439}) 2024-09-11T03:05:34.573 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694672026}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2374141865}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299565982}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690186465}) 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.574 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3287590238}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1980792573}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1328413699}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4178596506}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.422+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3120476527}) 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.575 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060528476}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879116793}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009007099}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=588856149}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.576 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=898394682}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=273412405}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2949761813}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736531361}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=319246747}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1857693888}) 2024-09-11T03:05:34.577 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1838678751}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284320853}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258071907}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1070587212}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.578 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3774475903}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=817366002}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1202256664}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.426+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2162700791}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176907214}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809418331}) 2024-09-11T03:05:34.579 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2193730649}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3408691666}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3728559225}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2392110312}) 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.580 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=512626843}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2909941424}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=438711568}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1694332416}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2031462484}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3344764558}) 2024-09-11T03:05:34.581 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=404744603}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3678411093}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2647091285}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1476749337}) 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.582 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56827192}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.430+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3832867438}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341846935}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826241089}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102256353}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.583 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3560813193}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3220413579}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1466513933}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1196213888}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2468790771}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1204525940}) 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.584 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837136444}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1969366066}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2627777865}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1258625051}) 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.585 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=593836299}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=984310434}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1075233566}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4220581591}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.434+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1906518722}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373486856}) 2024-09-11T03:05:34.586 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1083181457}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599860025}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2486794816}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4199887882}) 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=768781735}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1924857098}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517738590}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2686050218}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758259921}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.588 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3588481311}) 2024-09-11T03:05:34.589 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.589 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.589 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051372717}) 2024-09-11T03:05:34.589 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.438+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.593 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.593 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=747837261}) 2024-09-11T03:05:34.593 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.593 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1481509404}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3864968540}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=784902823}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2103702882}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2814160029}) 2024-09-11T03:05:34.594 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213415567}) 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3130903823}) 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290009928}) 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.595 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4128428813}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3670152078}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2033475619}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=989503644}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=291237818}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4176832820}) 2024-09-11T03:05:34.596 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314143465}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=716536767}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319556435}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2444445285}) 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.597 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3858484916}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1808454454}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.442+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1458657101}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311183839}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3479361675}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4285003984}) 2024-09-11T03:05:34.598 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4045275018}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151103643}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1884162766}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2165362210}) 2024-09-11T03:05:34.599 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1289824445}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=130303717}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3616504656}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3792608469}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3618687874}) 2024-09-11T03:05:34.600 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3986145601}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3886104935}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1681580722}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=714393886}) 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.601 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3345748507}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3690916337}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3283525856}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.446+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3839703820}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005245669}) 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.602 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778968552}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1478945856}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1180143726}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2674521223}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.603 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1366954941}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2531173131}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4117153910}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=67458759}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199157726}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1761377356}) 2024-09-11T03:05:34.604 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810258079}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2818020753}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1172080571}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3353684150}) 2024-09-11T03:05:34.605 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=967703977}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819306686}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.450+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2618392157}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029954638}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2714984447}) 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.606 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797367038}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1421498948}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3173060696}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4159824161}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.607 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1427114087}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2563350353}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=212893642}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630301076}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1795567740}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071619252}) 2024-09-11T03:05:34.608 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941801680}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3313169137}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2659558102}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3545041590}) 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.609 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918962658}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3370602769}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.454+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318141192}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=60312568}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2642691787}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.610 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726380498}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2840123377}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2403773006}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3723299371}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=980714831}) 2024-09-11T03:05:34.611 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2002075762}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941348772}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2585338042}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4245684266}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1101108813}) 2024-09-11T03:05:34.612 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3059446474}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2063803909}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2439834264}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=321916332}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=864204463}) 2024-09-11T03:05:34.613 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210659986}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1549710787}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1791891104}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1216479601}) 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.614 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1241749063}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.458+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2666149485}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3921211254}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217635358}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4195565794}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.615 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198779607}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2933605359}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955335698}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3025602586}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.616 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1940669738}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2997867124}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=79471551}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3296428972}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2215067822}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989233379}) 2024-09-11T03:05:34.617 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317895502}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3401565541}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.462+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2602320741}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=189138955}) 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.618 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3180083853}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3941066599}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2814070937}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2479746404}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245225922}) 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.619 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572884744}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7727200}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=795999773}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4148728405}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.620 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3206136722}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1572065761}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865758541}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1634329705}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056572426}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.621 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3047764668}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.466+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3468612761}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=787804280}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2519206727}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.622 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2232570994}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3755444362}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176982300}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2032256562}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=290602466}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2912587290}) 2024-09-11T03:05:34.623 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838198417}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1065897601}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3750772104}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=459431828}) 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.624 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=431991471}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867140535}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.470+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1330692242}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1742187579}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220318104}) 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.625 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2948008325}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447513389}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2393517550}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3680786299}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.626 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4287152417}) 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3427026639}) 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=61044304}) 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824394288}) 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.627 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=828007902}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780236919}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1448695640}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2125036662}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.628 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2913777958}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=525314929}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128183762}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.474+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=389547661}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625060696}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.629 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390794410}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1863025491}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2653756252}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1575471353}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=386963503}) 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.630 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2999565348}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2312131661}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3056192787}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076952423}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1542779846}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.631 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1820456821}) 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4000282439}) 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4128691890}) 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2604941468}) 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.632 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4215110531}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2479920806}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1925898587}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215109278}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2335131830}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.633 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782545977}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.478+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4153218748}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=305805254}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2853084436}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.634 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=262096418}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4127756486}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=616381132}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279961217}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=422673824}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1870732230}) 2024-09-11T03:05:34.635 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1801049284}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4153647471}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=963995962}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1028285954}) 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.636 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2225074999}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.482+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4200483153}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=782663763}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013549729}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1952617696}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.637 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3833550226}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1874079218}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=588069866}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2411768027}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3539164009}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3075331241}) 2024-09-11T03:05:34.638 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309672158}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603667733}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894795676}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860679418}) 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.639 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2301585242}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=914714890}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3536553522}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1415759078}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.486+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3959951830}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.640 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683829623}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118623406}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2308257118}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3628760949}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=121690850}) 2024-09-11T03:05:34.641 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=778101340}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2079420107}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=384632452}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=393361689}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641271831}) 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.642 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2593907771}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501402674}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3237650156}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3197844901}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.643 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3173003748}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4124164999}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2707029922}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2009795992}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552583435}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.644 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676671656}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.490+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2589322842}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3366433554}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2006034949}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.645 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2260679184}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=830836917}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966516526}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1770032431}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918081659}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3346589647}) 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.646 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=101712354}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1624354316}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1221881793}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2234820699}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.647 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2860826307}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2182004275}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4162296530}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521616737}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=886554251}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.648 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2327401672}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.494+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2765748043}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=589453584}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2865487092}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.649 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1416071316}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1791600866}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3238387247}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2223025456}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1402179536}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3592890278}) 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.650 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3353153673}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1607026647}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=656396336}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=791176520}) 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.651 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4217872725}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.498+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=420581189}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42182036}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351396653}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772183846}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3236723522}) 2024-09-11T03:05:34.652 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2976796016}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1758804418}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=326155705}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=8860617}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4222004522}) 2024-09-11T03:05:34.653 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879573154}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=387912057}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990696483}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=649724132}) 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.654 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1046543971}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=514682096}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1392294132}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4094122005}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.502+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=697967604}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.655 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342015398}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3574220198}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019654892}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1727889205}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3446991511}) 2024-09-11T03:05:34.656 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3960751478}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2034903816}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1945117521}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3716134201}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520130066}) 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.657 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2473463509}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078941072}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4049350044}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=586594807}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.658 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2942416569}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1424693900}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1833708025}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.506+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1759222856}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384900722}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2715176087}) 2024-09-11T03:05:34.659 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020499040}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4126739503}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=603411331}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=319208775}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.660 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=887052138}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1623827810}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2398180561}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271300017}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3371776180}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070567932}) 2024-09-11T03:05:34.661 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3860340016}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3347357994}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3838042757}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3350908911}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=371017481}) 2024-09-11T03:05:34.662 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.510+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=133011292}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737166494}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=723210203}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777399696}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4016262671}) 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.663 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=527389312}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3787698964}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1480681634}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4179140885}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1913169981}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.664 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472588401}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678568141}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1918318768}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1010458283}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.665 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3042715529}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=435868966}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1642774387}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030067973}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3914596968}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997281711}) 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.666 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1774502936}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146349529}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1673522021}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2804794551}) 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.667 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1476387314}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3443077522}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3494732112}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.514+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=767653949}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3244111276}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.668 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=926204305}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132710470}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3456977254}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=992026573}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.669 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4201932158}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3049974237}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3276772641}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1304435783}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753437132}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2684561224}) 2024-09-11T03:05:34.670 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3494123557}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2066609186}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1911478797}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3315945066}) 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.671 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2085521597}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2959829298}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.518+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=854662491}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107833906}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701702588}) 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.672 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244063660}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2754944736}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3730312679}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1500857578}) 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.673 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=88780899}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1137287340}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2044302402}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878851205}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=693136329}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.674 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946024349}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3217043937}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1363701707}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2778762916}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1830520214}) 2024-09-11T03:05:34.675 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313122287}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.522+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=895335881}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835857536}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8506953}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=272809768}) 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.676 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4082137395}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3390257566}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=897923378}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1174770846}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4252243134}) 2024-09-11T03:05:34.677 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3262629118}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045252415}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1061488752}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064195156}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2506799937}) 2024-09-11T03:05:34.678 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3317872382}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2455166244}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1651647031}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2316122817}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.526+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3289492157}) 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.679 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849692791}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260786950}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783360486}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2043144328}) 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.680 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2232244708}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=335214772}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2511994103}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2925026466}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=958319157}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.681 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339924899}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1218853840}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251600363}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=14881765}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.682 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3105150303}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3468745797}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=875513320}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3026009179}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.530+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=81895227}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940525097}) 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.683 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114462608}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2307853759}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3336802870}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4271115165}) 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.684 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4274222206}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2172537322}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=878145964}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2805850148}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699352282}) 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.685 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1946963382}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1066263464}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2702991659}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3630969935}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=650781698}) 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.686 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=33193379}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2683603410}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.534+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2847452334}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2357793490}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1599289139}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.687 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010907584}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2308894310}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3654154350}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=533088818}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2784582858}) 2024-09-11T03:05:34.688 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2554425272}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2313342775}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018529383}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192077063}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=66931700}) 2024-09-11T03:05:34.689 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3067047847}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1060680126}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3346205384}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3490079839}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=754079223}) 2024-09-11T03:05:34.690 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.538+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3538486804}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=1780832423,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=946322993}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=8,mon_config_key=9} crc {logm=3857640181,mdsmap=3491474967,mgr=432785742,mgr_metadata=3513241537,mgrstat=3347770604,mon_config_key=1000899325}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=71,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=6} crc {logm=3577518031,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=826052978}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=10,monmap=2} crc {logm=2215155603,mdsmap=3491474967,mgr=839680553,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=727252594,mon_config_key=226227825,monmap=416611756}) 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.691 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=67,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11} crc {logm=1785936079,mdsmap=3491474967,mgr=534862250,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=2106052021}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=66,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=2} crc {logm=2121808756,mdsmap=3491474967,mgr=839680554,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=2285379554,monmap=416611756}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=2036379888,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=87940778}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=345829931,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=1444986053}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=4145396560,mdsmap=3491474967,mgr=839680553,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=2828877375,mon_config_key=2074392736}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.692 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=42,osdmap=44} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=985253322,osd_pg_creating=3862806985,osd_snap=4087060844,osdmap=2880357862}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=45} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=1596038039,osd_metadata=4042788089,osd_pg_creating=3862806985,osd_snap=2816957220,osdmap=2685279281}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=45} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=4187288967,osdmap=3160792526}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=5,osd_snap=39,osdmap=48} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1810866017,osd_snap=2381107466,osdmap=2101807318}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=45} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=1596038039,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2998064236,osdmap=3789210883}) 2024-09-11T03:05:34.693 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=42,osdmap=44} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=4278568289,osd_pg_creating=3862806985,osd_snap=2966343093,osdmap=934873372}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=42,osdmap=45} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=3517310312,osd_metadata=306687733,osd_pg_creating=3862806985,osd_snap=3804169404,osdmap=2169364246}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.542+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=2,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=43} crc {mon_config_key=2749256476,monmap=2495409540,nvmeofgw=598337686,osd_metadata=1477139452,osd_pg_creating=3862806985,osd_snap=2196075714,osdmap=1862367849}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=2,monmap=2,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=41,osdmap=47} crc {mon_config_key=2749256476,monmap=416611756,nvmeofgw=3140514009,osd_metadata=3273443428,osd_pg_creating=3862806985,osd_snap=575730321,osdmap=3800059229}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1694807993}) 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.694 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1973492159}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3587428359}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3609148633}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2842167775}) 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.695 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2217807225}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3697736744}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=645232863}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.546+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3598225760}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4287134611}) 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.696 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=270554353}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=585100225}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=657622480}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3779122004}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.697 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3211995331}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3281936844}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2641774119}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2417264533}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3683397319}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=233017983}) 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.698 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1333038357}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1446431361}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3676657461}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1380689419}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.699 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=322793439}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1326558171}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.550+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1181879771}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=976915286}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1452117206}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2974326107}) 2024-09-11T03:05:34.700 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=4203686506}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=491461010}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1337591698}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3325609054}) 2024-09-11T03:05:34.701 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1486705113}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2288903785}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.554+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3598914280}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2997420408}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3486171410}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.702 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=4144829267}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2129127053}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=730346176}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3936061922}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3618794443}) 2024-09-11T03:05:34.703 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2290925850}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3862244056}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=238937988}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1333952198}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=109459598}) 2024-09-11T03:05:34.704 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=528754218}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1225805967}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1357396057}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3564490562}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.558+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3396293085}) 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.705 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=59513420}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2526743708}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1775060109}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=312300709}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.706 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3933014867}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=531215077}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2915208323}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2915358562}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.562+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3423839892}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.707 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=13196635}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2950049791}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=296169332}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3819992219}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=197937025}) 2024-09-11T03:05:34.708 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=783737984}) 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3702947482}) 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=972558999}) 2024-09-11T03:05:34.709 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.566+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3279027485}) 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=2459298764}) 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=4093474991}) 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=68} crc {osdmap=1170227428}) 2024-09-11T03:05:34.712 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=65} crc {osdmap=4082202404}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=63} crc {osdmap=2717770480}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=68} crc {osdmap=3760468379}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=64} crc {osdmap=2030728346}) 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:05:34.713 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3333205903}) 2024-09-11T03:05:34.714 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:05:34.578+0000 7f87039da640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=69} crc {osdmap=154644017}) 2024-09-11T03:05:35.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:05:35.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient: get_monmap_and_config 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient: build_initial_monmap 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient: monmap: 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:35.347328+0000 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:35.347328+0000 2024-09-11T03:05:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:35.346 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:35.347 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 1 Processor -- start 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 1 -- start start 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient: init 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient: _add_conns ranks=[0,1,10,3,9,2,5,8,6,7,4] 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc038139b10 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc03806fe30 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fc0380725d0 0x7fc038142130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient(hunting): picked mon.noname-f con 0x7fc0380725d0 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc03813a600 con 0x7fc038139b10 2024-09-11T03:05:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0380a9650 con 0x7fc03806fe30 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fc038071350 con 0x7fc0380725d0 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc03e877640 10 monclient(hunting): authenticate will time out at 1889.227742s 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc037fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc038139b10 0x7fc038139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56852/0 (socket says 172.21.15.98:56852) 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.341+0000 7fc037fff640 1 -- 172.21.15.98:0/2610297524 learned_addr learned my addr 172.21.15.98:0/2610297524 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261857507 0 0) 0x7fc03813a600 con 0x7fc038139b10 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): _negotiate 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): my global_id is 45432 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc020003530 con 0x7fc038139b10 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 1 --2- 172.21.15.98:0/2610297524 >> v2:172.21.15.190:3300/0 conn(0x7fc0380725d0 0x7fc038142130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 10 monclient(hunting): get_auth_request con 0x7fc0380725d0 auth_method 0 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3409133574 0 0) 0x7fc020003530 con 0x7fc038139b10 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 >> v2:172.21.15.190:3300/0 conn(0x7fc0380725d0 msgr2=0x7fc038142130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 --2- 172.21.15.98:0/2610297524 >> v2:172.21.15.190:3300/0 conn(0x7fc0380725d0 0x7fc038142130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 >> v1:172.21.15.190:6789/0 conn(0x7fc03806fe30 legacy=0x7fc03813ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:05:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc038071630 con 0x7fc038139b10 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fc02c001970 con 0x7fc038139b10 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03e877640 5 monclient: authenticate success, global_id 45432 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03e877640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: dump: 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 >> v1:172.21.15.98:6789/0 conn(0x7fc038139b10 legacy=0x7fc038139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: _add_conns ranks=[4,2,3,5,8,1,6,7,0] 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc020004210 0x7fc020004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): picked mon.c con 0x7fc020004210 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc020004e20 0x7fc0200052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): picked mon.b con 0x7fc020004e20 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc020009ce0 0x7fc02000c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): picked mon.g con 0x7fc020009ce0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc038071350 con 0x7fc020004e20 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc02000ca50 con 0x7fc020009ce0 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fc02000cc00 con 0x7fc020004210 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:35.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03e877640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc020004210 0x7fc020004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc020004e20 0x7fc0200052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): get_auth_request con 0x7fc020004210 auth_method 0 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 10 monclient(hunting): get_auth_request con 0x7fc020004e20 auth_method 0 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:35.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc020009ce0 0x7fc02000c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient(hunting): handle_auth_done global_id 45432 payload 293 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 -- 172.21.15.98:0/2610297524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc020009ce0 msgr2=0x7fc02000c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:35.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc020009ce0 0x7fc02000c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 -- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc020004e20 msgr2=0x7fc0200052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc020004e20 0x7fc0200052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 monclient: found mon.c 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 -- 172.21.15.98:0/2610297524 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc020003530 con 0x7fc020004210 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient: _finish_auth 0 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 10 monclient: _check_auth_tickets 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 10 monclient: get_auth_request con 0x7fc020009ce0 auth_method 0 2024-09-11T03:05:35.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc020009ce0 0x7fc02000c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03ce14640 1 -- 172.21.15.98:0/2610297524 reap_dead start 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc037fff640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc020004210 0x7fc020004810 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007ee0 tx=0x7fc02c01c9a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f7b1515555e4136c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc02c007b00 con 0x7fc020004210 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: dump: 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.368 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03e877640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc02c005d50 con 0x7fc020004210 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03e877640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 1 -- 172.21.15.98:0/2610297524 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc02c0074e0 con 0x7fc020004210 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc03e877640 4 set_mon_vals no callback set 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.345+0000 7fc036ffd640 10 monclient: dump: 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.370 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.371 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: shutdown 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc020004210 msgr2=0x7fc020004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc020004210 0x7fc020004810 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007ee0 tx=0x7fc02c01c9a0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2610297524 shutdown_connections 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 --2- 172.21.15.98:0/2610297524 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fc020004210 0x7fc020004810 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2610297524 >> 172.21.15.98:0/2610297524 conn(0x7fc0380ac740 msgr2=0x7fc0380a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2610297524 shutdown_connections 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2610297524 wait complete. 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: build_initial_monmap 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: monmap: 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:35.372 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:35.373 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 librados: starting msgr at 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 librados: starting objecter 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 Processor -- start 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- start start 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 librados: setting wanted keys 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 librados: calling monclient init 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: init 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _add_conns ranks=[7,8,0,1,4,2,5,6,3] 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc020004210 0x7fc0381d1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): picked mon.noname-i con 0x7fc020004210 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc0381d1d40 0x7fc0381d2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): picked mon.noname-e con 0x7fc0381d1d40 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038123fe0 0x7fc038124440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): picked mon.noname-a con 0x7fc038123fe0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0380a85b0 con 0x7fc038123fe0 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fc0380a8430 con 0x7fc020004210 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc0380a8730 con 0x7fc0381d1d40 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:35.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient(hunting): authenticate will time out at 1889.233750s 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038123fe0 0x7fc038124440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc0381d1d40 0x7fc0381d2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038123fe0 0x7fc038124440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36052/0 (socket says 172.21.15.98:36052) 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc0381d1d40 0x7fc0381d2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55812/0 (socket says 172.21.15.98:55812) 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 -- 172.21.15.98:0/2438911868 learned_addr learned my addr 172.21.15.98:0/2438911868 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): get_auth_request con 0x7fc038123fe0 auth_method 0 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 10 monclient(hunting): get_auth_request con 0x7fc0381d1d40 auth_method 0 2024-09-11T03:05:35.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc037fff640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc020004210 0x7fc0381d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc037fff640 10 monclient(hunting): get_auth_request con 0x7fc020004210 auth_method 0 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc037fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient(hunting): handle_auth_done global_id 45441 payload 824 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 -- 172.21.15.98:0/2438911868 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc0381d1d40 msgr2=0x7fc0381d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc0381d1d40 0x7fc0381d2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 -- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc020004210 msgr2=0x7fc0381d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc020004210 0x7fc0381d1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient: _finish_hunting 0 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 monclient: found mon.noname-a 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 -- 172.21.15.98:0/2438911868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc038127940 con 0x7fc038123fe0 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03ce14640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038123fe0 0x7fc038124440 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc03806fc60 tx=0x7fc030007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 client.?.objecter ms_handle_connect 0x7fc038123fe0 2024-09-11T03:05:35.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 1 -- 172.21.15.98:0/2438911868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc030019070 con 0x7fc038123fe0 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: dump: 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.379 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: _finish_auth 0 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:35.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 1 -- 172.21.15.98:0/2438911868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc030009c70 con 0x7fc038123fe0 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 5 monclient: authenticate success, global_id 45441 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _renew_subs 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc038073390 con 0x7fc038123fe0 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 1 -- 172.21.15.98:0/2438911868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc030009e20 con 0x7fc038123fe0 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 10 monclient: dump: 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.381 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.382 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 client.45441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _renew_subs 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc0367fc640 4 set_mon_vals no callback set 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0381258c0 con 0x7fc038123fe0 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 librados: init done 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:35.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc034ff9640 1 -- 172.21.15.98:0/2438911868 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc03001d3f0 con 0x7fc038123fe0 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.349+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0381395e0 con 0x7fc038123fe0 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc034ff9640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc02403df40 0x7fc024040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc034ff9640 1 -- 172.21.15.98:0/2438911868 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(634..634 src has 1..634) ==== 6761+0+0 (secure 0 0 0) 0x7fc03004f2b0 con 0x7fc038123fe0 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc034ff9640 10 client.45441.objecter ms_dispatch 0x7fc038000ba0 osd_map(634..634 src has 1..634) 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc034ff9640 3 client.45441.objecter handle_osd_map got epochs [634,634] > 0 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc034ff9640 3 client.45441.objecter handle_osd_map decoding full epoch 634 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc034ff9640 20 client.45441.objecter dump_active .. 0 homeless 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc037fff640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc02403df40 0x7fc024040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc037fff640 10 monclient: get_auth_request con 0x7fc02403df40 auth_method 0 2024-09-11T03:05:35.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc034ff9640 10 client.45441.objecter ms_handle_connect 0x7fc02403df40 2024-09-11T03:05:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.353+0000 7fc037fff640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc02403df40 0x7fc024040400 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0ab000 tx=0x7fc02c007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.357+0000 7fc034ff9640 1 -- 172.21.15.98:0/2438911868 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fc0381395e0 con 0x7fc038123fe0 2024-09-11T03:05:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.357+0000 7fc034ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:35.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.357+0000 7fc034ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:35.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.589+0000 7fc03e877640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:05:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc03813a510 con 0x7fc038123fe0 2024-09-11T03:05:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc034ff9640 1 -- 172.21.15.98:0/2438911868 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fc0300133a0 con 0x7fc038123fe0 2024-09-11T03:05:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc034ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:05:35.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc034ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:35.596 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":106,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"i","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:05:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 10 librados: watch_flush enter 2024-09-11T03:05:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 10 librados: watch_flush exit 2024-09-11T03:05:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 20 client.45441.objecter shutdown clearing up homeless session... 2024-09-11T03:05:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 10 client.45441.objecter successfully canceled tick 2024-09-11T03:05:35.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc02403df40 msgr2=0x7fc024040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc02403df40 0x7fc024040400 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0ab000 tx=0x7fc02c007040 comp rx=0 tx=0).stop 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 10 monclient: shutdown 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038123fe0 msgr2=0x7fc038124440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038123fe0 0x7fc038124440 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc03806fc60 tx=0x7fc030007700 comp rx=0 tx=0).stop 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 shutdown_connections 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc02403df40 0x7fc024040400 unknown :-1 s=CLOSED pgs=3673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc038123fe0 0x7fc038124440 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc0381d1d40 0x7fc0381d2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 --2- 172.21.15.98:0/2438911868 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fc020004210 0x7fc0381d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 >> 172.21.15.98:0/2438911868 conn(0x7fc0380ac740 msgr2=0x7fc0381c1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 shutdown_connections 2024-09-11T03:05:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 -- 172.21.15.98:0/2438911868 wait complete. 2024-09-11T03:05:35.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.593+0000 7fc03e877640 1 librados: shutdown 2024-09-11T03:05:35.616 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:05:35.617 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-11T03:05:35.617 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T03:05:35.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:05:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient: get_monmap_and_config 2024-09-11T03:05:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient: build_initial_monmap 2024-09-11T03:05:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient: monmap: 2024-09-11T03:05:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:35.739549+0000 2024-09-11T03:05:35.737 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:35.739549+0000 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:35.738 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:35.739 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:35.739 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 1 Processor -- start 2024-09-11T03:05:35.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 1 -- start start 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient: init 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient: _add_conns ranks=[9,2,0,3,7,1,5,4,10,6,8] 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ea80a46a0 0x7f8ea80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea80a46a0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea8139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea80a4fc0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ea8139e70 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:35.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ea80a17b0 con 0x7f8ea8139e70 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f8ea80a0750 con 0x7f8ea80a46a0 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8ea80775d0 con 0x7f8ea80a4fc0 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8eacec7640 10 monclient(hunting): authenticate will time out at 1889.619570s 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f8ea80a46a0 0x7f8ea80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea8139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea6d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8ea8139e70 0x7f8ea8140bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56862/0 (socket says 172.21.15.98:56862) 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea6d9d640 1 -- 172.21.15.98:0/3931315343 learned_addr learned my addr 172.21.15.98:0/3931315343 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:05:35.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea659c640 10 monclient(hunting): get_auth_request con 0x7f8ea80a46a0 auth_method 0 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 10 monclient(hunting): get_auth_request con 0x7f8ea80a4fc0 auth_method 0 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.733+0000 7f8ea659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_done global_id 94178 payload 274 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 1 -- 172.21.15.98:0/3931315343 >> v2:172.21.15.98:3300/0 conn(0x7f8ea80a46a0 msgr2=0x7f8ea80a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 1 --2- 172.21.15.98:0/3931315343 >> v2:172.21.15.98:3300/0 conn(0x7f8ea80a46a0 0x7f8ea80a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 1 -- 172.21.15.98:0/3931315343 >> v1:172.21.15.98:6789/0 conn(0x7f8ea8139e70 legacy=0x7f8ea8140bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 1 monclient: found mon.noname-b 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 1 -- 172.21.15.98:0/3931315343 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea80a0200 con 0x7f8ea80a4fc0 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea659c640 1 --2- 172.21.15.98:0/3931315343 >> v2:172.21.15.98:3300/0 conn(0x7f8ea80a46a0 0x7f8ea80a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea5d9b640 1 --2- 172.21.15.98:0/3931315343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea8139930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009d90 tx=0x7f8e9c01c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a024d86b258e78b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 1 -- 172.21.15.98:0/3931315343 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e9c0a6970 con 0x7f8ea80a4fc0 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: dump: 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.747 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: _finish_auth 0 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: _check_auth_tickets 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 5 monclient: authenticate success, global_id 94178 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 1 -- 172.21.15.98:0/3931315343 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e9c0a6500 con 0x7f8ea80a4fc0 2024-09-11T03:05:35.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 1 -- 172.21.15.98:0/3931315343 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e9c0a6730 con 0x7f8ea80a4fc0 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 4 set_mon_vals no callback set 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8ea559a640 10 monclient: dump: 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.749 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.750 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 10 monclient: shutdown 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 -- 172.21.15.98:0/3931315343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 msgr2=0x7f8ea8139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 --2- 172.21.15.98:0/3931315343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea8139930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c009d90 tx=0x7f8e9c01c830 comp rx=0 tx=0).stop 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 -- 172.21.15.98:0/3931315343 shutdown_connections 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 --2- 172.21.15.98:0/3931315343 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea8139930 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 --2- 172.21.15.98:0/3931315343 >> v2:172.21.15.98:3300/0 conn(0x7f8ea80a46a0 0x7f8ea80a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 -- 172.21.15.98:0/3931315343 >> 172.21.15.98:0/3931315343 conn(0x7f8ea806ed00 msgr2=0x7f8ea806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 -- 172.21.15.98:0/3931315343 shutdown_connections 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.737+0000 7f8eacec7640 1 -- 172.21.15.98:0/3931315343 wait complete. 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: build_initial_monmap 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: monmap: 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:35.752 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:35.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 librados: starting msgr at 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 librados: starting objecter 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 Processor -- start 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 -- start start 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 librados: setting wanted keys 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 librados: calling monclient init 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: init 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: _add_conns ranks=[0,2,8,6,7,5,1,3,4] 2024-09-11T03:05:35.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea80a46a0 0x7f8ea813a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): picked mon.noname-a con 0x7f8ea80a46a0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea813aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): picked mon.noname-b con 0x7f8ea80a4fc0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ea8139e70 0x7f8ea813b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): picked mon.noname-e con 0x7f8ea8139e70 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea80a53a0 con 0x7f8ea80a46a0 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f8ea8077850 con 0x7f8ea80a4fc0 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f8ea813da40 con 0x7f8ea8139e70 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:35.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): authenticate will time out at 1889.625237s 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea80a46a0 0x7f8ea813a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea813aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea80a46a0 0x7f8ea813a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36066/0 (socket says 172.21.15.98:36066) 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 -- 172.21.15.98:0/1855770838 learned_addr learned my addr 172.21.15.98:0/1855770838 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): get_auth_request con 0x7f8ea80a46a0 auth_method 0 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): get_auth_request con 0x7f8ea80a4fc0 auth_method 0 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea6d9d640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ea8139e70 0x7f8ea813b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea6d9d640 10 monclient(hunting): get_auth_request con 0x7f8ea8139e70 auth_method 0 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient(hunting): handle_auth_done global_id 45459 payload 824 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ea8139e70 msgr2=0x7f8ea813b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f8ea8139e70 0x7f8ea813b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 msgr2=0x7f8ea813aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f8ea80a4fc0 0x7f8ea813aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient: _finish_hunting 0 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 monclient: found mon.noname-a 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 -- 172.21.15.98:0/1855770838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea813e000 con 0x7f8ea80a46a0 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea659c640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea80a46a0 0x7f8ea813a9b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000e980 tx=0x7f8ea000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 client.?.objecter ms_handle_connect 0x7f8ea80a46a0 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea0017070 con 0x7f8ea80a46a0 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:35.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: dump: 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.765 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea000ce70 con 0x7f8ea80a46a0 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea00138e0 con 0x7f8ea80a46a0 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 5 monclient: authenticate success, global_id 45459 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: _renew_subs 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea813e2e0 con 0x7f8ea80a46a0 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea4d99640 4 set_mon_vals no callback set 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 monclient: dump: 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.767 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 client.45459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: _renew_subs 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea813bff0 con 0x7f8ea80a46a0 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 librados: init done 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:35.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea813e7e0 0x7f8ea81d19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea80a5520 con 0x7f8ea813e7e0 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8ea0013470 con 0x7f8ea80a46a0 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea813e7e0 0x7f8ea81d19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient: get_auth_request con 0x7f8ea813e7e0 auth_method 0 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8e8803df70 0x7f8e88040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_done global_id 45468 payload 824 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8ea5d9b640 1 -- 172.21.15.98:0/1855770838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ea80a4110 con 0x7f8ea813e7e0 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(634..634 src has 1..634) ==== 6761+0+0 (secure 0 0 0) 0x7f8ea000f040 con 0x7f8ea80a46a0 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 10 client.45459.objecter ms_dispatch 0x7f8ea8000ba0 osd_map(634..634 src has 1..634) 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 3 client.45459.objecter handle_osd_map got epochs [634,634] > 0 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.741+0000 7f8e96ffd640 3 client.45459.objecter handle_osd_map decoding full epoch 634 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 20 client.45459.objecter dump_active .. 0 homeless 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 client.45459.objecter ms_handle_connect 0x7f8ea813e7e0 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 client.45459.objecter resend_mon_ops 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8ea5d9b640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea813e7e0 0x7f8ea81d19a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80a21e0 tx=0x7f8e9c0195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8ea6d9d640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8e8803df70 0x7f8e88040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8ea6d9d640 10 monclient: get_auth_request con 0x7f8e8803df70 auth_method 0 2024-09-11T03:05:35.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 client.45459.objecter ms_handle_connect 0x7f8e8803df70 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8ea6d9d640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8e8803df70 0x7f8e88040430 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f8e98007fd0 tx=0x7f8e98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e9c0be070 con 0x7f8ea813e7e0 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 monclient: dump: 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.771 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8e9c004780 con 0x7f8ea813e7e0 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:35.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea813e7e0 msgr2=0x7f8ea81d19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea813e7e0 0x7f8ea81d19a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80a21e0 tx=0x7f8e9c0195d0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea813e7e0 msgr2=0x7f8ea81d19a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.745+0000 7f8e96ffd640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea813e7e0 0x7f8ea81d19a0 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80a21e0 tx=0x7f8e9c0195d0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8eacec7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8eacec7640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea81d1ee0 0x7f8ea81d22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8eacec7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea81d2830 con 0x7f8ea81d1ee0 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea81d1ee0 0x7f8ea81d22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient: get_auth_request con 0x7f8ea81d1ee0 auth_method 0 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 10 monclient(hunting): handle_auth_done global_id 45477 payload 824 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 1 -- 172.21.15.98:0/1855770838 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ea80a4a80 con 0x7f8ea81d1ee0 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8ea5d9b640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea81d1ee0 0x7f8ea81d22f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c01aee0 tx=0x7f8e9c0a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8e96ffd640 10 client.45459.objecter ms_handle_connect 0x7f8ea81d1ee0 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8e96ffd640 10 client.45459.objecter resend_mon_ops 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e9c0cd8c0 con 0x7f8ea81d1ee0 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8e96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8e96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.753+0000 7f8e96ffd640 10 monclient: dump: 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9602 (secure 0 0 0) 0x7f8e9c0cd450 con 0x7f8ea81d1ee0 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8e96ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8e96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea81d1ee0 msgr2=0x7f8ea81d22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8e96ffd640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea81d1ee0 0x7f8ea81d22f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c01aee0 tx=0x7f8e9c0a9000 comp rx=0 tx=0).stop 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8e96ffd640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea81d1ee0 msgr2=0x7f8ea81d22f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8e96ffd640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea81d1ee0 0x7f8ea81d22f0 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c01aee0 tx=0x7f8e9c0a9000 comp rx=0 tx=0).stop 2024-09-11T03:05:35.776 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 562193, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:35.777 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:35.778 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:35.779 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:35.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:35.782 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.783 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:35.784 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:35.785 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.786 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.787 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.790 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.791 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.792 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.794 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.795 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.796 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.797 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.798 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.799 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:35.800 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.801 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:35.802 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:35.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 10 librados: watch_flush enter 2024-09-11T03:05:35.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 10 librados: watch_flush exit 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 20 client.45459.objecter shutdown clearing up homeless session... 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 10 client.45459.objecter successfully canceled tick 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8e8803df70 msgr2=0x7f8e88040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8e8803df70 0x7f8e88040430 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f8e98007fd0 tx=0x7f8e98009040 comp rx=0 tx=0).stop 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 10 monclient: shutdown 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea80a46a0 msgr2=0x7f8ea813a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 --2- 172.21.15.98:0/1855770838 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ea80a46a0 0x7f8ea813a9b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8ea000e980 tx=0x7f8ea000ee40 comp rx=0 tx=0).stop 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8ea6d9d640 1 -- 172.21.15.98:0/1855770838 reap_dead start 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8ea6d9d640 1 -- 172.21.15.98:0/1855770838 reap_dead start 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 shutdown_connections 2024-09-11T03:05:35.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 >> 172.21.15.98:0/1855770838 conn(0x7f8ea806ed00 msgr2=0x7f8ea8070610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:35.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 shutdown_connections 2024-09-11T03:05:35.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 -- 172.21.15.98:0/1855770838 wait complete. 2024-09-11T03:05:35.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.757+0000 7f8eacec7640 1 librados: shutdown 2024-09-11T03:05:35.805 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:35.805 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T03:05:35.805 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:05:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient: get_monmap_and_config 2024-09-11T03:05:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient: build_initial_monmap 2024-09-11T03:05:35.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient: monmap: 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:35.929809+0000 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:35.929809+0000 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:35.928 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:35.929 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:35.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 Processor -- start 2024-09-11T03:05:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 -- start start 2024-09-11T03:05:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient: init 2024-09-11T03:05:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient: _add_conns ranks=[6,7,4,9,1,2,10,3,5,8,0] 2024-09-11T03:05:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31900a3690 0x7f31900a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): picked mon.noname-d con 0x7f31900a3690 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31900a1eb0 0x7f31900a3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): picked mon.noname-i con 0x7f31900a1eb0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190137460 0x7f31901378c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): picked mon.noname-c con 0x7f3190137460 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f319009ec40 con 0x7f3190137460 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f31900a44f0 con 0x7f31900a3690 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f31900a1ca0 con 0x7f31900a1eb0 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3196813640 10 monclient(hunting): authenticate will time out at 1889.810078s 2024-09-11T03:05:35.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190137460 0x7f31901378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190137460 0x7f31901378c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54752/0 (socket says 172.21.15.98:54752) 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 -- 172.21.15.98:0/288472205 learned_addr learned my addr 172.21.15.98:0/288472205 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): get_auth_request con 0x7f3190137460 auth_method 0 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31900a3690 0x7f31900a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 10 monclient(hunting): get_auth_request con 0x7f31900a3690 auth_method 0 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f318f7fe640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31900a1eb0 0x7f31900a3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient(hunting): handle_auth_done global_id 24772 payload 274 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 -- 172.21.15.98:0/288472205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31900a1eb0 msgr2=0x7f31900a3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31900a1eb0 0x7f31900a3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 -- 172.21.15.98:0/288472205 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31900a3690 msgr2=0x7f31900a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31900a3690 0x7f31900a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient: _finish_hunting 0 2024-09-11T03:05:35.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 monclient: found mon.noname-c 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.925+0000 7f3194db0640 1 -- 172.21.15.98:0/288472205 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319009dd20 con 0x7f3190137460 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3194db0640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190137460 0x7f31901378c0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f31880098a0 tx=0x7f318801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=841ecb2c72e0d36e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 1 -- 172.21.15.98:0/288472205 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31880b0070 con 0x7f3190137460 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: dump: 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.937 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.938 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: _finish_auth 0 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 1 -- 172.21.15.98:0/288472205 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31880aa810 con 0x7f3190137460 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 5 monclient: authenticate success, global_id 24772 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 1 -- 172.21.15.98:0/288472205 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31880a9300 con 0x7f3190137460 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 4 set_mon_vals no callback set 2024-09-11T03:05:35.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318effd640 10 monclient: dump: 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.941 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.941 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.941 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.941 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient: shutdown 2024-09-11T03:05:35.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- 172.21.15.98:0/288472205 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190137460 msgr2=0x7f31901378c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190137460 0x7f31901378c0 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f31880098a0 tx=0x7f318801c440 comp rx=0 tx=0).stop 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- 172.21.15.98:0/288472205 shutdown_connections 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f3190137460 0x7f31901378c0 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f31900a1eb0 0x7f31900a3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 --2- 172.21.15.98:0/288472205 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f31900a3690 0x7f31900a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- 172.21.15.98:0/288472205 >> 172.21.15.98:0/288472205 conn(0x7f31900a4780 msgr2=0x7f31900a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- 172.21.15.98:0/288472205 shutdown_connections 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- 172.21.15.98:0/288472205 wait complete. 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient: build_initial_monmap 2024-09-11T03:05:35.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient: monmap: 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:35.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 librados: starting msgr at 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 librados: starting objecter 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 Processor -- start 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- start start 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 librados: setting wanted keys 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 librados: calling monclient init 2024-09-11T03:05:35.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient: init 2024-09-11T03:05:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient: _add_conns ranks=[4,0,7,8,6,1,5,2,3] 2024-09-11T03:05:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31900a1eb0 0x7f3190135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): picked mon.noname-c con 0x7f31900a1eb0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31900a3690 0x7f31901364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): picked mon.noname-a con 0x7f31900a3690 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3190137460 0x7f319011e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): picked mon.noname-i con 0x7f3190137460 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31900a1a10 con 0x7f31900a3690 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f31900a1890 con 0x7f31900a1eb0 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f31901410e0 con 0x7f3190137460 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:35.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3196813640 10 monclient(hunting): authenticate will time out at 1889.815272s 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31900a1eb0 0x7f3190135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31900a1eb0 0x7f3190135fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54766/0 (socket says 172.21.15.98:54766) 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31900a3690 0x7f31901364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 -- 172.21.15.98:0/4063309713 learned_addr learned my addr 172.21.15.98:0/4063309713 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): get_auth_request con 0x7f31900a1eb0 auth_method 0 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 10 monclient(hunting): get_auth_request con 0x7f31900a3690 auth_method 0 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient(hunting): handle_auth_done global_id 24781 payload 824 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3190137460 msgr2=0x7f319011e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f3194db0640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3190137460 0x7f319011e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f3190137460 0x7f319011e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31900a3690 msgr2=0x7f31901364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31900a3690 0x7f31901364f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 monclient: found mon.noname-c 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 -- 172.21.15.98:0/4063309713 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31901370e0 con 0x7f31900a1eb0 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318ffff640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31900a1eb0 0x7f3190135fb0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f318400e960 tx=0x7f318400ee20 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318cff9640 10 client.?.objecter ms_handle_connect 0x7f31900a1eb0 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.929+0000 7f318cff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3184017070 con 0x7f31900a1eb0 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: dump: 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.955 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.956 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: _finish_auth 0 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: _check_auth_tickets 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f318400ce70 con 0x7f31900a1eb0 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 5 monclient: authenticate success, global_id 24781 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 10 monclient: _renew_subs 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31900a6c90 con 0x7f31900a1eb0 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31840138e0 con 0x7f31900a1eb0 2024-09-11T03:05:35.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:35.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: dump: 2024-09-11T03:05:35.959 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.960 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 10 client.24781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318e7fc640 4 set_mon_vals no callback set 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 10 monclient: _renew_subs 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3190136a30 con 0x7f31900a1eb0 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 1 librados: init done 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901204e0 0x7f319011e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f31900734b0 con 0x7f31901204e0 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901204e0 0x7f319011e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient: get_auth_request con 0x7f31901204e0 auth_method 0 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f3184013470 con 0x7f31900a1eb0 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 10 monclient(hunting): handle_auth_done global_id 94196 payload 824 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f317c03df40 0x7f317c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 1 -- 172.21.15.98:0/4063309713 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3190068b40 con 0x7f31901204e0 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(635..635 src has 1..635) ==== 6388+0+0 (secure 0 0 0) 0x7f318400f040 con 0x7f31900a1eb0 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 client.24781.objecter ms_dispatch 0x7f3190000ba0 osd_map(635..635 src has 1..635) 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 3 client.24781.objecter handle_osd_map got epochs [635,635] > 0 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 3 client.24781.objecter handle_osd_map decoding full epoch 635 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318f7fe640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901204e0 0x7f319011e810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f319011fff0 tx=0x7f318000d750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 20 client.24781.objecter dump_active .. 0 homeless 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 client.24781.objecter ms_handle_connect 0x7f31901204e0 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 client.24781.objecter resend_mon_ops 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f318000a9f0 con 0x7f31901204e0 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: dump: 2024-09-11T03:05:35.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.964 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3194db0640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f317c03df40 0x7f317c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3194db0640 10 monclient: get_auth_request con 0x7f317c03df40 auth_method 0 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f318000a580 con 0x7f31901204e0 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901204e0 msgr2=0x7f319011e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901204e0 0x7f319011e810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f319011fff0 tx=0x7f318000d750 comp rx=0 tx=0).stop 2024-09-11T03:05:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901204e0 msgr2=0x7f319011e810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f31901204e0 0x7f319011e810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f318cff9640 10 client.24781.objecter ms_handle_connect 0x7f317c03df40 2024-09-11T03:05:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.933+0000 7f3194db0640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f317c03df40 0x7f317c040400 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 crypto rx=0x7f31880ad000 tx=0x7f31880a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f319011ed50 0x7f319011f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f319006ca10 con 0x7f319011ed50 2024-09-11T03:05:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f319011ed50 0x7f319011f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient: get_auth_request con 0x7f319011ed50 auth_method 0 2024-09-11T03:05:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 10 monclient(hunting): handle_auth_done global_id 94205 payload 824 2024-09-11T03:05:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 1 -- 172.21.15.98:0/4063309713 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3190121610 con 0x7f319011ed50 2024-09-11T03:05:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 10 client.24781.objecter ms_handle_connect 0x7f319011ed50 2024-09-11T03:05:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 10 client.24781.objecter resend_mon_ops 2024-09-11T03:05:35.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318f7fe640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f319011ed50 0x7f319011f160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f319011fff0 tx=0x7f3180004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f318000ae40 con 0x7f319011ed50 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 10 monclient: dump: 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:35.970 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:35.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f318001d040 con 0x7f319011ed50 2024-09-11T03:05:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f319011ed50 msgr2=0x7f319011f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f319011ed50 0x7f319011f160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f319011fff0 tx=0x7f3180004a50 comp rx=0 tx=0).stop 2024-09-11T03:05:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f319011ed50 msgr2=0x7f319011f160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:35.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f318cff9640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f319011ed50 0x7f319011f160 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f319011fff0 tx=0x7f3180004a50 comp rx=0 tx=0).stop 2024-09-11T03:05:35.972 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 34327, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:35.973 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 8, 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:35.974 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:35.975 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:35.978 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.979 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.980 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:35.984 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:35.987 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.988 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.990 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:35.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.992 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:35.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:35.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.996 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:35.997 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 10 librados: watch_flush enter 2024-09-11T03:05:36.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 10 librados: watch_flush exit 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 20 client.24781.objecter shutdown clearing up homeless session... 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 10 client.24781.objecter successfully canceled tick 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f317c03df40 msgr2=0x7f317c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f317c03df40 0x7f317c040400 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 crypto rx=0x7f31880ad000 tx=0x7f31880a6000 comp rx=0 tx=0).stop 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 10 monclient: shutdown 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31900a1eb0 msgr2=0x7f3190135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 --2- 172.21.15.98:0/4063309713 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f31900a1eb0 0x7f3190135fb0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f318400e960 tx=0x7f318400ee20 comp rx=0 tx=0).stop 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3194db0640 1 -- 172.21.15.98:0/4063309713 reap_dead start 2024-09-11T03:05:36.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3194db0640 1 -- 172.21.15.98:0/4063309713 reap_dead start 2024-09-11T03:05:36.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 shutdown_connections 2024-09-11T03:05:36.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 >> 172.21.15.98:0/4063309713 conn(0x7f31900a4780 msgr2=0x7f31900a70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 shutdown_connections 2024-09-11T03:05:36.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 -- 172.21.15.98:0/4063309713 wait complete. 2024-09-11T03:05:36.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:35.941+0000 7f3196813640 1 librados: shutdown 2024-09-11T03:05:36.002 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:36.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T03:05:36.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient: get_monmap_and_config 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient: monmap: 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:36.129409+0000 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:36.129409+0000 2024-09-11T03:05:36.127 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:36.128 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 Processor -- start 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 -- start start 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient: init 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient: _add_conns ranks=[4,2,3,0,1,5,8,9,7,6,10] 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89080a46a0 0x7f89080a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): picked mon.noname-c con 0x7f89080a46a0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:36.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a4fc0 0x7f8908139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): picked mon.noname-b con 0x7f89080a4fc0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8908139e70 0x7f890813a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): picked mon.noname-g con 0x7f8908139e70 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f89080a1540 con 0x7f89080a4fc0 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f89080a1790 con 0x7f8908139e70 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8908077140 con 0x7f89080a46a0 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f890d74e640 10 monclient(hunting): authenticate will time out at 1890.009830s 2024-09-11T03:05:36.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a4fc0 0x7f8908139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89080a46a0 0x7f89080a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89080a46a0 0x7f89080a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54768/0 (socket says 172.21.15.98:54768) 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 -- 172.21.15.98:0/3990225704 learned_addr learned my addr 172.21.15.98:0/3990225704 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): get_auth_request con 0x7f89080a46a0 auth_method 0 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a4fc0 0x7f8908139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49588/0 (socket says 172.21.15.98:49588) 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 10 monclient(hunting): get_auth_request con 0x7f89080a4fc0 auth_method 0 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89077fe640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8908139e70 0x7f890813a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89077fe640 10 monclient(hunting): get_auth_request con 0x7f8908139e70 auth_method 0 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient(hunting): handle_auth_done global_id 24790 payload 274 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 -- 172.21.15.98:0/3990225704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8908139e70 msgr2=0x7f890813a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8908139e70 0x7f890813a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 -- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a4fc0 msgr2=0x7f8908139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a4fc0 0x7f8908139930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 monclient: found mon.noname-c 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f8906ffd640 1 -- 172.21.15.98:0/3990225704 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89080740c0 con 0x7f89080a46a0 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.125+0000 7f89067fc640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a4fc0 0x7f8908139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89080a46a0 0x7f89080a4a80 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f88f80098d0 tx=0x7f88f801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=89b968c43036ed78 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 1 -- 172.21.15.98:0/3990225704 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88f80b0070 con 0x7f89080a46a0 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:05:36.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: dump: 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.138 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 5 monclient: authenticate success, global_id 24790 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 1 -- 172.21.15.98:0/3990225704 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f80aa810 con 0x7f89080a46a0 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 1 -- 172.21.15.98:0/3990225704 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88f80a9300 con 0x7f89080a46a0 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 4 set_mon_vals no callback set 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8905ffb640 10 monclient: dump: 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.140 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient: shutdown 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89080a46a0 msgr2=0x7f89080a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89080a46a0 0x7f89080a4a80 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f88f80098d0 tx=0x7f88f801c440 comp rx=0 tx=0).stop 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- 172.21.15.98:0/3990225704 shutdown_connections 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f8908139e70 0x7f890813a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f89080a4fc0 0x7f8908139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 --2- 172.21.15.98:0/3990225704 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f89080a46a0 0x7f89080a4a80 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- 172.21.15.98:0/3990225704 >> 172.21.15.98:0/3990225704 conn(0x7f890806ed00 msgr2=0x7f890806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- 172.21.15.98:0/3990225704 shutdown_connections 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- 172.21.15.98:0/3990225704 wait complete. 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient: monmap: 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:36.142 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 librados: starting msgr at 2024-09-11T03:05:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 librados: starting objecter 2024-09-11T03:05:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 Processor -- start 2024-09-11T03:05:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- start start 2024-09-11T03:05:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 librados: setting wanted keys 2024-09-11T03:05:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 librados: calling monclient init 2024-09-11T03:05:36.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient: init 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient: _add_conns ranks=[3,1,5,0,7,8,4,6,2] 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89080a46a0 0x7f890813ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): picked mon.noname-g con 0x7f89080a46a0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080a4fc0 0x7f89081d19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): picked mon.noname-f con 0x7f89080a4fc0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8908139e70 0x7f890813c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): picked mon.noname-h con 0x7f8908139e70 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8908077640 con 0x7f89080a4fc0 2024-09-11T03:05:36.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f89080774c0 con 0x7f89080a46a0 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f8908143410 con 0x7f8908139e70 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f890d74e640 10 monclient(hunting): authenticate will time out at 1890.015456s 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89080a46a0 0x7f890813ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f89077fe640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8908139e70 0x7f890813c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f89067fc640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080a4fc0 0x7f89081d19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89080a46a0 0x7f890813ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:34490/0 (socket says 172.21.15.98:34490) 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 1 -- 172.21.15.98:0/3606488326 learned_addr learned my addr 172.21.15.98:0/3606488326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 10 monclient(hunting): get_auth_request con 0x7f89080a46a0 auth_method 0 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f8906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f89077fe640 10 monclient(hunting): get_auth_request con 0x7f8908139e70 auth_method 0 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f89077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f89077fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.129+0000 7f89077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): get_auth_request con 0x7f89080a4fc0 auth_method 0 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 10 monclient(hunting): handle_auth_done global_id 74514 payload 824 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8908139e70 msgr2=0x7f890813c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f8908139e70 0x7f890813c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080a4fc0 msgr2=0x7f89081d19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89080a4fc0 0x7f89081d19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 1 monclient: found mon.noname-g 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 1 -- 172.21.15.98:0/3606488326 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89081236f0 con 0x7f89080a46a0 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f8906ffd640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89080a46a0 0x7f890813ec70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f88f80097c0 tx=0x7f88f80191e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 client.?.objecter ms_handle_connect 0x7f89080a46a0 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88f80b9070 con 0x7f89080a46a0 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T03:05:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: dump: 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f80044b0 con 0x7f89080a46a0 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 5 monclient: authenticate success, global_id 74514 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 10 monclient: _renew_subs 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8908123f50 con 0x7f89080a46a0 2024-09-11T03:05:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 10 client.74514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 10 monclient: _renew_subs 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8908123c70 con 0x7f89080a46a0 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 1 librados: init done 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88f80b0040 con 0x7f89080a46a0 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: dump: 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89057fa640 4 set_mon_vals no callback set 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec002890 0x7f88ec004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f88ec007cc0 con 0x7f88ec002890 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec002890 0x7f88ec004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient: get_auth_request con 0x7f88ec002890 auth_method 0 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 10 monclient(hunting): handle_auth_done global_id 24799 payload 824 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 1 -- 172.21.15.98:0/3606488326 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88ec007ec0 con 0x7f88ec002890 2024-09-11T03:05:36.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 client.74514.objecter ms_handle_connect 0x7f88ec002890 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 client.74514.objecter resend_mon_ops 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f89067fc640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec002890 0x7f88ec004d30 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f88f00119a0 tx=0x7f88f0011e60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88f000fc20 con 0x7f88ec002890 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: dump: 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f88f80aacd0 con 0x7f89080a46a0 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f88e803e1e0 0x7f88e80406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(635..635 src has 1..635) ==== 6388+0+0 (secure 0 0 0) 0x7f88f80e89b0 con 0x7f89080a46a0 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 client.74514.objecter ms_dispatch 0x7f8908000ba0 osd_map(635..635 src has 1..635) 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 3 client.74514.objecter handle_osd_map got epochs [635,635] > 0 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 3 client.74514.objecter handle_osd_map decoding full epoch 635 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 20 client.74514.objecter dump_active .. 0 homeless 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f88f000fdc0 con 0x7f88ec002890 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec002890 msgr2=0x7f88ec004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec002890 0x7f88ec004d30 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f88f00119a0 tx=0x7f88f0011e60 comp rx=0 tx=0).stop 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec002890 msgr2=0x7f88ec004d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.133+0000 7f88ff7fe640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec002890 0x7f88ec004d30 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.137+0000 7f89077fe640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f88e803e1e0 0x7f88e80406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.137+0000 7f89077fe640 10 monclient: get_auth_request con 0x7f88e803e1e0 auth_method 0 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.137+0000 7f88ff7fe640 10 client.74514.objecter ms_handle_connect 0x7f88e803e1e0 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.137+0000 7f89077fe640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f88e803e1e0 0x7f88e80406a0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7f8908077370 tx=0x7f8900009780 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec006cd0 0x7f88ec007160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f88ec00a2a0 con 0x7f88ec006cd0 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec006cd0 0x7f88ec007160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient: get_auth_request con 0x7f88ec006cd0 auth_method 0 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 10 monclient(hunting): handle_auth_done global_id 24808 payload 824 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 1 -- 172.21.15.98:0/3606488326 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88ec0076a0 con 0x7f88ec006cd0 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 10 client.74514.objecter ms_handle_connect 0x7f88ec006cd0 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89067fc640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec006cd0 0x7f88ec007160 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f88f00119a0 tx=0x7f88f00046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 10 client.74514.objecter resend_mon_ops 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88f0014a50 con 0x7f88ec006cd0 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 10 monclient: dump: 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.166 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9435 (secure 0 0 0) 0x7f88f002b3c0 con 0x7f88ec006cd0 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec006cd0 msgr2=0x7f88ec007160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec006cd0 0x7f88ec007160 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f88f00119a0 tx=0x7f88f00046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec006cd0 msgr2=0x7f88ec007160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f88ff7fe640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f88ec006cd0 0x7f88ec007160 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 951925, 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:36.168 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.169 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:36.170 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.171 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.172 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.173 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.174 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:36.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.179 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.183 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:36.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.188 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.189 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.191 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.192 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.193 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 10 librados: watch_flush enter 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 10 librados: watch_flush exit 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 20 client.74514.objecter shutdown clearing up homeless session... 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 10 client.74514.objecter successfully canceled tick 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f88e803e1e0 msgr2=0x7f88e80406a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f88e803e1e0 0x7f88e80406a0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7f8908077370 tx=0x7f8900009780 comp rx=0 tx=0).stop 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 10 monclient: shutdown 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89080a46a0 msgr2=0x7f890813ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89080a46a0 0x7f890813ec70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f88f80097c0 tx=0x7f88f80191e0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f89077fe640 1 -- 172.21.15.98:0/3606488326 reap_dead start 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 shutdown_connections 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 --2- 172.21.15.98:0/3606488326 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f89080a46a0 0x7f890813ec70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 >> 172.21.15.98:0/3606488326 conn(0x7f890806ed00 msgr2=0x7f890813fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 shutdown_connections 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 -- 172.21.15.98:0/3606488326 wait complete. 2024-09-11T03:05:36.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.141+0000 7f890d74e640 1 librados: shutdown 2024-09-11T03:05:36.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:36.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T03:05:36.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:05:36.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient: get_monmap_and_config 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient: monmap: 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:36.317332+0000 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:36.317332+0000 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.315 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:36.316 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 1 Processor -- start 2024-09-11T03:05:36.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 1 -- start start 2024-09-11T03:05:36.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient: init 2024-09-11T03:05:36.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient: _add_conns ranks=[8,0,9,1,6,10,5,4,2,3,7] 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09700a3de0 0x7f09700a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient(hunting): picked mon.noname-e con 0x7f09700a3de0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09700a41c0 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f097013c980 0x7f097013ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient(hunting): picked mon.noname-a con 0x7f097013c980 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097009e670 con 0x7f09700a41c0 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f097009f970 con 0x7f097013c980 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f09700a2900 con 0x7f09700a3de0 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f09777a8640 10 monclient(hunting): authenticate will time out at 1890.197446s 2024-09-11T03:05:36.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f097013c980 0x7f097013ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0974d43640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f09700a41c0 0x7f09700a45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:56874/0 (socket says 172.21.15.98:56874) 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0974d43640 1 -- 172.21.15.98:0/225439106 learned_addr learned my addr 172.21.15.98:0/225439106 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): get_auth_request con 0x7f097013c980 auth_method 0 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 1 --2- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09700a3de0 0x7f09700a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 10 monclient(hunting): get_auth_request con 0x7f09700a3de0 auth_method 0 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient(hunting): handle_auth_done global_id 45495 payload 274 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 1 -- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09700a3de0 msgr2=0x7f09700a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 1 --2- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09700a3de0 0x7f09700a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 1 -- 172.21.15.98:0/225439106 >> v1:172.21.15.98:6789/0 conn(0x7f09700a41c0 legacy=0x7f09700a45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 1 monclient: found mon.noname-a 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f096ffff640 1 -- 172.21.15.98:0/225439106 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565145539 0 0) 0x7f097009e670 con 0x7f09700a41c0 2024-09-11T03:05:36.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f0975d45640 1 -- 172.21.15.98:0/225439106 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097009fc10 con 0x7f097013c980 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.313+0000 7f096ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 --2- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f097013c980 0x7f097013ed70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f09600097f0 tx=0x7f096001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c70d9da381b908d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 1 -- 172.21.15.98:0/225439106 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09600b0070 con 0x7f097013c980 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: dump: 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.324 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 1 -- 172.21.15.98:0/225439106 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09600aa810 con 0x7f097013c980 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 5 monclient: authenticate success, global_id 45495 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 4 set_mon_vals no callback set 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 1 -- 172.21.15.98:0/225439106 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09600a9300 con 0x7f097013c980 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096ffff640 10 monclient: dump: 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.326 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: shutdown 2024-09-11T03:05:36.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f097013c980 msgr2=0x7f097013ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 --2- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f097013c980 0x7f097013ed70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f09600097f0 tx=0x7f096001c360 comp rx=0 tx=0).stop 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- 172.21.15.98:0/225439106 shutdown_connections 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 --2- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f097013c980 0x7f097013ed70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 --2- 172.21.15.98:0/225439106 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09700a3de0 0x7f09700a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- 172.21.15.98:0/225439106 >> 172.21.15.98:0/225439106 conn(0x7f09700a4ed0 msgr2=0x7f09700a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- 172.21.15.98:0/225439106 shutdown_connections 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- 172.21.15.98:0/225439106 wait complete. 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: monmap: 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:36.328 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 librados: starting msgr at 2024-09-11T03:05:36.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 librados: starting objecter 2024-09-11T03:05:36.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 Processor -- start 2024-09-11T03:05:36.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- start start 2024-09-11T03:05:36.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 librados: setting wanted keys 2024-09-11T03:05:36.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 librados: calling monclient init 2024-09-11T03:05:36.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: init 2024-09-11T03:05:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: _add_conns ranks=[1,5,8,4,0,7,3,2,6] 2024-09-11T03:05:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09700a3de0 0x7f097009a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): picked mon.noname-f con 0x7f09700a3de0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09700a41c0 0x7f097009c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): picked mon.noname-h con 0x7f09700a41c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:36.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f097013c980 0x7f097009c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): picked mon.noname-e con 0x7f097013c980 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0970141e60 con 0x7f09700a3de0 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f0970141ce0 con 0x7f09700a41c0 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0970141fe0 con 0x7f097013c980 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient(hunting): authenticate will time out at 1890.202606s 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f097013c980 0x7f097009c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f097013c980 0x7f097009c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55836/0 (socket says 172.21.15.98:55836) 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 -- 172.21.15.98:0/4108960965 learned_addr learned my addr 172.21.15.98:0/4108960965 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): get_auth_request con 0x7f097013c980 auth_method 0 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0974d43640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09700a41c0 0x7f097009c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975544640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09700a3de0 0x7f097009a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient(hunting): handle_auth_done global_id 25316 payload 824 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09700a41c0 msgr2=0x7f097009c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09700a41c0 0x7f097009c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09700a3de0 msgr2=0x7f097009a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09700a3de0 0x7f097009a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 monclient: found mon.noname-e 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 -- 172.21.15.98:0/4108960965 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097013b5a0 con 0x7f097013c980 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975544640 10 monclient: get_auth_request con 0x7f09700a3de0 auth_method 0 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0974d43640 10 monclient: get_auth_request con 0x7f09700a41c0 auth_method 0 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975544640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09700a3de0 0x7f097009a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0974d43640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09700a41c0 0x7f097009c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f0975d45640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f097013c980 0x7f097009c910 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f09600ae040 tx=0x7f0960019100 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 client.?.objecter ms_handle_connect 0x7f097013c980 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09600b0050 con 0x7f097013c980 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: dump: 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.340 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09600bc070 con 0x7f097013c980 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09600aae30 con 0x7f097013c980 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 5 monclient: authenticate success, global_id 25316 2024-09-11T03:05:36.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: _renew_subs 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097013bb20 con 0x7f097013c980 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096f7fe640 4 set_mon_vals no callback set 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f096dffb640 10 monclient: dump: 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.342 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.343 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 client.25316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: _renew_subs 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f097013a8e0 con 0x7f097013c980 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.317+0000 7f09777a8640 1 librados: init done 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f09777a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f09777a8640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009ac90 0x7f097009ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f09777a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f097006ca10 con 0x7f097009ac90 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009ac90 0x7f097009ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0960004250 con 0x7f097013c980 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient: get_auth_request con 0x7f097009ac90 auth_method 0 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095c03df40 0x7f095c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 10 monclient(hunting): handle_auth_done global_id 84534 payload 824 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 1 -- 172.21.15.98:0/4108960965 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09700a45a0 con 0x7f097009ac90 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(635..635 src has 1..635) ==== 6388+0+0 (secure 0 0 0) 0x7f09600ea090 con 0x7f097013c980 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 client.25316.objecter ms_dispatch 0x7f0970000ba0 osd_map(635..635 src has 1..635) 2024-09-11T03:05:36.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 3 client.25316.objecter handle_osd_map got epochs [635,635] > 0 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 3 client.25316.objecter handle_osd_map decoding full epoch 635 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 20 client.25316.objecter dump_active .. 0 homeless 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 client.25316.objecter ms_handle_connect 0x7f097009ac90 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 client.25316.objecter resend_mon_ops 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0975544640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009ac90 0x7f097009ce50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0968010820 tx=0x7f0968010ce0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f096800f040 con 0x7f097009ac90 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0974d43640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095c03df40 0x7f095c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 monclient: dump: 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.347 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0974d43640 10 monclient: get_auth_request con 0x7f095c03df40 auth_method 0 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f096801c400 con 0x7f097009ac90 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009ac90 msgr2=0x7f097009ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009ac90 0x7f097009ce50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0968010820 tx=0x7f0968010ce0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009ac90 msgr2=0x7f097009ce50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009ac90 0x7f097009ce50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f096dffb640 10 client.25316.objecter ms_handle_connect 0x7f095c03df40 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.321+0000 7f0974d43640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095c03df40 0x7f095c040400 secure :-1 s=READY pgs=3677 cs=0 l=1 rev1=1 crypto rx=0x7f09640012f0 tx=0x7f0964009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f09777a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f09777a8640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009d390 0x7f0970072780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f09777a8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f097006e870 con 0x7f097009d390 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009d390 0x7f0970072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient: get_auth_request con 0x7f097009d390 auth_method 0 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.325+0000 7f0975544640 10 monclient(hunting): handle_auth_done global_id 84543 payload 824 2024-09-11T03:05:36.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f0975544640 1 -- 172.21.15.98:0/4108960965 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f09700a0bb0 con 0x7f097009d390 2024-09-11T03:05:36.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 10 client.25316.objecter ms_handle_connect 0x7f097009d390 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 10 client.25316.objecter resend_mon_ops 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f0975544640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009d390 0x7f0970072780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0968011010 tx=0x7f09680046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0968005bb0 con 0x7f097009d390 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 10 monclient: dump: 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.351 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7f0968005bb0 con 0x7f097009d390 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009d390 msgr2=0x7f0970072780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009d390 0x7f0970072780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0968011010 tx=0x7f09680046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009d390 msgr2=0x7f0970072780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f096dffb640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f097009d390 0x7f0970072780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 162142, 2024-09-11T03:05:36.353 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.354 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.355 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.356 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:36.357 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.358 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.360 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.362 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.363 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.365 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.369 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.370 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.372 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:36.373 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:36.374 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:36.375 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 10 librados: watch_flush enter 2024-09-11T03:05:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 10 librados: watch_flush exit 2024-09-11T03:05:36.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 20 client.25316.objecter shutdown clearing up homeless session... 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 10 client.25316.objecter successfully canceled tick 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095c03df40 msgr2=0x7f095c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f095c03df40 0x7f095c040400 secure :-1 s=READY pgs=3677 cs=0 l=1 rev1=1 crypto rx=0x7f09640012f0 tx=0x7f0964009040 comp rx=0 tx=0).stop 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 10 monclient: shutdown 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f097013c980 msgr2=0x7f097009c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 --2- 172.21.15.98:0/4108960965 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f097013c980 0x7f097009c910 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f09600ae040 tx=0x7f0960019100 comp rx=0 tx=0).stop 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f0975d45640 1 -- 172.21.15.98:0/4108960965 reap_dead start 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f0975d45640 1 -- 172.21.15.98:0/4108960965 reap_dead start 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 shutdown_connections 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 >> 172.21.15.98:0/4108960965 conn(0x7f09700a4ed0 msgr2=0x7f097013cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 shutdown_connections 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 -- 172.21.15.98:0/4108960965 wait complete. 2024-09-11T03:05:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.329+0000 7f09777a8640 1 librados: shutdown 2024-09-11T03:05:36.379 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:36.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T03:05:36.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:05:36.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.489+0000 7f0d40abb640 10 monclient: get_monmap_and_config 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.489+0000 7f0d40abb640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient: monmap: 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:36.497213+0000 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:36.497213+0000 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:36.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:36.496 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 1 Processor -- start 2024-09-11T03:05:36.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 1 -- start start 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient: init 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient: _add_conns ranks=[2,1,9,0,4,5,7,8,3,6,10] 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d3c139b10 0x7f0d3c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3c139b10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0d3c139ef0 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d3c13a2d0 0x7f0d3c0a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c13a2d0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c0702b0 con 0x7f0d3c139ef0 2024-09-11T03:05:36.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f0d3c071bf0 con 0x7f0d3c13a2d0 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f0d3c071350 con 0x7f0d3c139b10 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d40abb640 10 monclient(hunting): authenticate will time out at 1890.377472s 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d3c13a2d0 0x7f0d3c0a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d3c139b10 0x7f0d3c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f0d3c13a2d0 0x7f0d3c0a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36118/0 (socket says 172.21.15.98:36118) 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 -- 172.21.15.98:0/64101526 learned_addr learned my addr 172.21.15.98:0/64101526 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): get_auth_request con 0x7f0d3c13a2d0 auth_method 0 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 10 monclient(hunting): get_auth_request con 0x7f0d3c139b10 auth_method 0 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient(hunting): handle_auth_done global_id 45513 payload 274 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 -- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d3c139b10 msgr2=0x7f0d3c072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 --2- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d3c139b10 0x7f0d3c072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 -- 172.21.15.98:0/64101526 >> v1:172.21.15.190:6789/0 conn(0x7f0d3c139ef0 legacy=0x7f0d3c0a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 monclient: found mon.noname-a 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3ad9d640 1 -- 172.21.15.98:0/64101526 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c0a9620 con 0x7f0d3c13a2d0 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.493+0000 7f0d3a59c640 1 --2- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d3c139b10 0x7f0d3c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3ad9d640 1 --2- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d3c13a2d0 0x7f0d3c0a8120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009e50 tx=0x7f0d2c01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff1535a196b8535 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 1 -- 172.21.15.98:0/64101526 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d2c0090d0 con 0x7f0d3c13a2d0 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:36.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: dump: 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.505 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 5 monclient: authenticate success, global_id 45513 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 1 -- 172.21.15.98:0/64101526 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d2c0a5ab0 con 0x7f0d3c13a2d0 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 4 set_mon_vals no callback set 2024-09-11T03:05:36.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 1 -- 172.21.15.98:0/64101526 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d2c0a5ca0 con 0x7f0d3c13a2d0 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3959a640 10 monclient: dump: 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.507 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: shutdown 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d3c13a2d0 msgr2=0x7f0d3c0a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 --2- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d3c13a2d0 0x7f0d3c0a8120 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009e50 tx=0x7f0d2c01c980 comp rx=0 tx=0).stop 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- 172.21.15.98:0/64101526 shutdown_connections 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 --2- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d3c13a2d0 0x7f0d3c0a8120 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 --2- 172.21.15.98:0/64101526 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f0d3c139b10 0x7f0d3c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- 172.21.15.98:0/64101526 >> 172.21.15.98:0/64101526 conn(0x7f0d3c0ac790 msgr2=0x7f0d3c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- 172.21.15.98:0/64101526 shutdown_connections 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- 172.21.15.98:0/64101526 wait complete. 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: monmap: 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:36.509 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 librados: starting msgr at 2024-09-11T03:05:36.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 librados: starting objecter 2024-09-11T03:05:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 Processor -- start 2024-09-11T03:05:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- start start 2024-09-11T03:05:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 librados: setting wanted keys 2024-09-11T03:05:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 librados: calling monclient init 2024-09-11T03:05:36.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: init 2024-09-11T03:05:36.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: _add_conns ranks=[4,8,1,2,5,6,7,3,0] 2024-09-11T03:05:36.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d3c139b10 0x7f0d3c13a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3c139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c139ef0 0x7f0d3c13ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): picked mon.noname-e con 0x7f0d3c139ef0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c13a2d0 0x7f0d3c13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): picked mon.noname-f con 0x7f0d3c13a2d0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c144470 con 0x7f0d3c13a2d0 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0d3c0a8870 con 0x7f0d3c139b10 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0d3c1445f0 con 0x7f0d3c139ef0 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient(hunting): authenticate will time out at 1890.382651s 2024-09-11T03:05:36.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d3c139b10 0x7f0d3c13a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d3c139b10 0x7f0d3c13a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:54792/0 (socket says 172.21.15.98:54792) 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c139ef0 0x7f0d3c13ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 1 -- 172.21.15.98:0/167187816 learned_addr learned my addr 172.21.15.98:0/167187816 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 10 monclient(hunting): get_auth_request con 0x7f0d3c139b10 auth_method 0 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): get_auth_request con 0x7f0d3c139ef0 auth_method 0 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient(hunting): handle_auth_done global_id 25325 payload 824 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d3c139b10 msgr2=0x7f0d3c13a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d3c139b10 0x7f0d3c13a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c13a2d0 msgr2=0x7f0d3c13b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3ad9d640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c13a2d0 0x7f0d3c13b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d3c13a2d0 0x7f0d3c13b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 monclient: found mon.noname-e 2024-09-11T03:05:36.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 -- 172.21.15.98:0/167187816 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c13c930 con 0x7f0d3c139ef0 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d3a59c640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0d3c139b10 0x7f0d3c13a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d39d9b640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c139ef0 0x7f0d3c13ace0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400c390 tx=0x7f0d2400c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 client.?.objecter ms_handle_connect 0x7f0d3c139ef0 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d24017070 con 0x7f0d3c139ef0 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: dump: 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d24003600 con 0x7f0d3c139ef0 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 5 monclient: authenticate success, global_id 25325 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d24003ca0 con 0x7f0d3c139ef0 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d32ffd640 10 monclient: dump: 2024-09-11T03:05:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: _renew_subs 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d33fff640 4 set_mon_vals no callback set 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c13cc10 con 0x7f0d3c139ef0 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 client.25325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: _renew_subs 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3c0a5bf0 con 0x7f0d3c139ef0 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.497+0000 7f0d40abb640 1 librados: init done 2024-09-11T03:05:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d40abb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d40abb640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c1337f0 0x7f0d3c133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d40abb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0d3c06ca10 con 0x7f0d3c1337f0 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c1337f0 0x7f0d3c133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient: get_auth_request con 0x7f0d3c1337f0 auth_method 0 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_done global_id 25334 payload 824 2024-09-11T03:05:36.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0d24003830 con 0x7f0d3c139ef0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 1 -- 172.21.15.98:0/167187816 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d3c06e680 con 0x7f0d3c1337f0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3a59c640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c1337f0 0x7f0d3c133bd0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009ef0 tx=0x7f0d3400f610 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d1c03df40 0x7f0d1c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 client.25325.objecter ms_handle_connect 0x7f0d3c1337f0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 client.25325.objecter resend_mon_ops 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(635..635 src has 1..635) ==== 6388+0+0 (secure 0 0 0) 0x7f0d2404e8d0 con 0x7f0d3c139ef0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 client.25325.objecter ms_dispatch 0x7f0d3c000ba0 osd_map(635..635 src has 1..635) 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 3 client.25325.objecter handle_osd_map got epochs [635,635] > 0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 3 client.25325.objecter handle_osd_map decoding full epoch 635 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 20 client.25325.objecter dump_active .. 0 homeless 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d3400fe50 con 0x7f0d3c1337f0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 monclient: dump: 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.528 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3ad9d640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d1c03df40 0x7f0d1c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d3400fe50 con 0x7f0d3c1337f0 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c1337f0 msgr2=0x7f0d3c133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c1337f0 0x7f0d3c133bd0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009ef0 tx=0x7f0d3400f610 comp rx=0 tx=0).stop 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c1337f0 msgr2=0x7f0d3c133bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c1337f0 0x7f0d3c133bd0 secure :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7f0d34009ef0 tx=0x7f0d3400f610 comp rx=0 tx=0).stop 2024-09-11T03:05:36.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3ad9d640 10 monclient: get_auth_request con 0x7f0d1c03df40 auth_method 0 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d32ffd640 10 client.25325.objecter ms_handle_connect 0x7f0d1c03df40 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.501+0000 7f0d3ad9d640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d1c03df40 0x7f0d1c040400 secure :-1 s=READY pgs=3678 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0ae000 tx=0x7f0d2c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d40abb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d40abb640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c134110 0x7f0d3c0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d40abb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0d3c06e870 con 0x7f0d3c134110 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c134110 0x7f0d3c0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 10 monclient: get_auth_request con 0x7f0d3c134110 auth_method 0 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.505+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d3a59c640 10 monclient(hunting): handle_auth_done global_id 25343 payload 824 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d3a59c640 1 -- 172.21.15.98:0/167187816 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d3c0aa0a0 con 0x7f0d3c134110 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 10 client.25325.objecter ms_handle_connect 0x7f0d3c134110 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 10 client.25325.objecter resend_mon_ops 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d3a59c640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c134110 0x7f0d3c0a0e60 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400fbb0 tx=0x7f0d340046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d34004490 con 0x7f0d3c134110 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 10 monclient: dump: 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.531 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.532 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7f0d34022630 con 0x7f0d3c134110 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c134110 msgr2=0x7f0d3c0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c134110 0x7f0d3c0a0e60 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400fbb0 tx=0x7f0d340046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c134110 msgr2=0x7f0d3c0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d32ffd640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c134110 0x7f0d3c0a0e60 secure :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400fbb0 tx=0x7f0d340046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.533 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 952301, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:36.534 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.535 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:36.537 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:36.538 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.539 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:36.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.543 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:36.544 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:36.545 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.546 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.547 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.548 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.550 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.551 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.552 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.553 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:36.554 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.555 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:36.556 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:36.557 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:36.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 10 librados: watch_flush enter 2024-09-11T03:05:36.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 10 librados: watch_flush exit 2024-09-11T03:05:36.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 20 client.25325.objecter shutdown clearing up homeless session... 2024-09-11T03:05:36.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 10 client.25325.objecter successfully canceled tick 2024-09-11T03:05:36.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d1c03df40 msgr2=0x7f0d1c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0d1c03df40 0x7f0d1c040400 secure :-1 s=READY pgs=3678 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0ae000 tx=0x7f0d2c0a7070 comp rx=0 tx=0).stop 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 10 monclient: shutdown 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c139ef0 msgr2=0x7f0d3c13ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 --2- 172.21.15.98:0/167187816 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0d3c139ef0 0x7f0d3c13ace0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400c390 tx=0x7f0d2400c850 comp rx=0 tx=0).stop 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d3ad9d640 1 -- 172.21.15.98:0/167187816 reap_dead start 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d3ad9d640 1 -- 172.21.15.98:0/167187816 reap_dead start 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 shutdown_connections 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 >> 172.21.15.98:0/167187816 conn(0x7f0d3c0ac790 msgr2=0x7f0d3c0ab1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 shutdown_connections 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 -- 172.21.15.98:0/167187816 wait complete. 2024-09-11T03:05:36.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.509+0000 7f0d40abb640 1 librados: shutdown 2024-09-11T03:05:36.561 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:36.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T03:05:36.561 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:05:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.681+0000 7f2b98188640 10 monclient: get_monmap_and_config 2024-09-11T03:05:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.681+0000 7f2b98188640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.681+0000 7f2b98188640 10 monclient: monmap: 2024-09-11T03:05:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.686 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:36.688756+0000 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:36.688756+0000 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.687 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:36.688 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:36.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 Processor -- start 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 -- start start 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient: init 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient: _add_conns ranks=[8,2,7,0,5,10,4,6,9,3,1] 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b90139b10 0x7f2b90072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): picked mon.noname-e con 0x7f2b90139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90139ef0 0x7f2b900a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): picked mon.noname-b con 0x7f2b90139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b9013a2d0 0x7f2b900a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): picked mon.noname-i con 0x7f2b9013a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:36.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f2b900702b0 con 0x7f2b90139ef0 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f2b90071bf0 con 0x7f2b9013a2d0 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f2b90071350 con 0x7f2b90139b10 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient(hunting): authenticate will time out at 1890.568974s 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b90139b10 0x7f2b90072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90139ef0 0x7f2b900a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b90139b10 0x7f2b90072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55858/0 (socket says 172.21.15.98:55858) 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 -- 172.21.15.98:0/2645219548 learned_addr learned my addr 172.21.15.98:0/2645219548 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): get_auth_request con 0x7f2b90139b10 auth_method 0 2024-09-11T03:05:36.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 10 monclient(hunting): get_auth_request con 0x7f2b90139ef0 auth_method 0 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b96725640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b9013a2d0 0x7f2b900a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient(hunting): handle_auth_done global_id 25352 payload 274 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 -- 172.21.15.98:0/2645219548 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b9013a2d0 msgr2=0x7f2b900a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b9013a2d0 0x7f2b900a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 -- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90139ef0 msgr2=0x7f2b900a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90139ef0 0x7f2b900a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 monclient: found mon.noname-e 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 -- 172.21.15.98:0/2645219548 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b900a9620 con 0x7f2b90139b10 2024-09-11T03:05:36.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95723640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90139ef0 0x7f2b900a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b95f24640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b90139b10 0x7f2b90072600 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098a0 tx=0x7f2b8c01c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5b4aff51c46cba server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 1 -- 172.21.15.98:0/2645219548 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b8c0b0070 con 0x7f2b90139b10 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: dump: 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.701 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.701 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.701 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.701 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 1 -- 172.21.15.98:0/2645219548 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b8c0aa810 con 0x7f2b90139b10 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 5 monclient: authenticate success, global_id 25352 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 1 -- 172.21.15.98:0/2645219548 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b8c0a9300 con 0x7f2b90139b10 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b98188640 4 set_mon_vals no callback set 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.685+0000 7f2b94f22640 10 monclient: dump: 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.703 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.704 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: shutdown 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b90139b10 msgr2=0x7f2b90072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b90139b10 0x7f2b90072600 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0098a0 tx=0x7f2b8c01c440 comp rx=0 tx=0).stop 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/2645219548 shutdown_connections 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f2b9013a2d0 0x7f2b900a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f2b90139ef0 0x7f2b900a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- 172.21.15.98:0/2645219548 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f2b90139b10 0x7f2b90072600 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/2645219548 >> 172.21.15.98:0/2645219548 conn(0x7f2b900ac790 msgr2=0x7f2b900a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/2645219548 shutdown_connections 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/2645219548 wait complete. 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: monmap: 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.706 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 librados: starting msgr at 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 librados: starting objecter 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 Processor -- start 2024-09-11T03:05:36.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- start start 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 librados: setting wanted keys 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 librados: calling monclient init 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: init 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: _add_conns ranks=[0,5,6,8,3,4,7,2,1] 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b90139b10 0x7f2b9013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): picked mon.noname-a con 0x7f2b90139b10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2b90139ef0 0x7f2b9013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): picked mon.noname-h con 0x7f2b90139ef0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:36.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b9013a2d0 0x7f2b9013b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): picked mon.noname-d con 0x7f2b9013a2d0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90143460 con 0x7f2b90139b10 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f2b900a88a0 con 0x7f2b90139ef0 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f2b901435e0 con 0x7f2b9013a2d0 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): authenticate will time out at 1890.574110s 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b90139b10 0x7f2b9013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b90139b10 0x7f2b9013a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36132/0 (socket says 172.21.15.98:36132) 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 -- 172.21.15.98:0/1714331016 learned_addr learned my addr 172.21.15.98:0/1714331016 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): get_auth_request con 0x7f2b90139b10 auth_method 0 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b9013a2d0 0x7f2b9013b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 10 monclient(hunting): get_auth_request con 0x7f2b9013a2d0 auth_method 0 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b96725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95723640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2b90139ef0 0x7f2b9013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient(hunting): handle_auth_done global_id 45522 payload 824 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b9013a2d0 msgr2=0x7f2b9013b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f2b9013a2d0 0x7f2b9013b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2b90139ef0 msgr2=0x7f2b9013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2b90139ef0 0x7f2b9013ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 monclient: found mon.noname-a 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 -- 172.21.15.98:0/1714331016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9013cff0 con 0x7f2b90139b10 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95723640 10 monclient: get_auth_request con 0x7f2b90139ef0 auth_method 0 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95723640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f2b90139ef0 0x7f2b9013ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b95f24640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b90139b10 0x7f2b9013a7d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0ae040 tx=0x7f2b8c0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 client.?.objecter ms_handle_connect 0x7f2b90139b10 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b8c0b9070 con 0x7f2b90139b10 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: dump: 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.713 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b8c0044b0 con 0x7f2b90139b10 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 5 monclient: authenticate success, global_id 45522 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: _renew_subs 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b901406f0 con 0x7f2b90139b10 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b8c0b0040 con 0x7f2b90139b10 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 10 monclient: dump: 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.715 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b87fff640 4 set_mon_vals no callback set 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 client.45522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: _renew_subs 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b900aa510 con 0x7f2b90139b10 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 librados: init done 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2b8c004040 con 0x7f2b90139b10 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90133760 0x7f2b90133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.689+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9013bad0 con 0x7f2b90133760 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b7403df30 0x7f2b740403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(635..635 src has 1..635) ==== 6388+0+0 (secure 0 0 0) 0x7f2b8c0e8c50 con 0x7f2b90139b10 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 client.45522.objecter ms_dispatch 0x7f2b90000ba0 osd_map(635..635 src has 1..635) 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 3 client.45522.objecter handle_osd_map got epochs [635,635] > 0 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 3 client.45522.objecter handle_osd_map decoding full epoch 635 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 20 client.45522.objecter dump_active .. 0 homeless 2024-09-11T03:05:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90133760 0x7f2b90133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient: get_auth_request con 0x7f2b90133760 auth_method 0 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b96725640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b7403df30 0x7f2b740403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b96725640 10 monclient: get_auth_request con 0x7f2b7403df30 auth_method 0 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 10 monclient(hunting): handle_auth_done global_id 26119 payload 824 2024-09-11T03:05:36.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 1 -- 172.21.15.98:0/1714331016 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b90135940 con 0x7f2b90133760 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 client.45522.objecter ms_handle_connect 0x7f2b90133760 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 client.45522.objecter resend_mon_ops 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b95723640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90133760 0x7f2b90133b40 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a4d60 tx=0x7f2b80010c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 client.45522.objecter ms_handle_connect 0x7f2b7403df30 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b96725640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b7403df30 0x7f2b740403f0 secure :-1 s=READY pgs=3679 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a9a70 tx=0x7f2b88007670 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b8000f040 con 0x7f2b90133760 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 monclient: dump: 2024-09-11T03:05:36.719 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.720 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.720 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.720 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.720 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.720 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.720 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b80010e70 con 0x7f2b90133760 2024-09-11T03:05:36.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:36.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90133760 msgr2=0x7f2b90133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90133760 0x7f2b90133b40 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a4d60 tx=0x7f2b80010c20 comp rx=0 tx=0).stop 2024-09-11T03:05:36.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90133760 msgr2=0x7f2b90133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.693+0000 7f2b867fc640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90133760 0x7f2b90133b40 secure :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a4d60 tx=0x7f2b80010c20 comp rx=0 tx=0).stop 2024-09-11T03:05:36.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90134080 0x7f2b90137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9006ca10 con 0x7f2b90134080 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90134080 0x7f2b90137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient: get_auth_request con 0x7f2b90134080 auth_method 0 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 10 monclient(hunting): handle_auth_done global_id 26128 payload 824 2024-09-11T03:05:36.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 1 -- 172.21.15.98:0/1714331016 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b9006e680 con 0x7f2b90134080 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b95723640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90134080 0x7f2b90137b60 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009fd0 tx=0x7f2b800046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 10 client.45522.objecter ms_handle_connect 0x7f2b90134080 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 10 client.45522.objecter resend_mon_ops 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b80005e10 con 0x7f2b90134080 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 10 monclient: dump: 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.725 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.726 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9435 (secure 0 0 0) 0x7f2b800059a0 con 0x7f2b90134080 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90134080 msgr2=0x7f2b90137b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90134080 0x7f2b90137b60 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009fd0 tx=0x7f2b800046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90134080 msgr2=0x7f2b90137b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b867fc640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2b90134080 0x7f2b90137b60 secure :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009fd0 tx=0x7f2b800046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.728 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:36.728 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 952478, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:36.729 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.730 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.731 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:36.732 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.733 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.734 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.735 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:36.736 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.737 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:36.738 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:36.739 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:36.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.741 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.743 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:36.745 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.746 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.747 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.749 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.750 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:36.751 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.752 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:36.753 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 10 librados: watch_flush enter 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 10 librados: watch_flush exit 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 20 client.45522.objecter shutdown clearing up homeless session... 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 10 client.45522.objecter successfully canceled tick 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b7403df30 msgr2=0x7f2b740403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f2b7403df30 0x7f2b740403f0 secure :-1 s=READY pgs=3679 cs=0 l=1 rev1=1 crypto rx=0x7f2b900a9a70 tx=0x7f2b88007670 comp rx=0 tx=0).stop 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 10 monclient: shutdown 2024-09-11T03:05:36.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b90139b10 msgr2=0x7f2b9013a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 --2- 172.21.15.98:0/1714331016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b90139b10 0x7f2b9013a7d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0ae040 tx=0x7f2b8c0191e0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b96725640 1 -- 172.21.15.98:0/1714331016 reap_dead start 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b96725640 1 -- 172.21.15.98:0/1714331016 reap_dead start 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 shutdown_connections 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 >> 172.21.15.98:0/1714331016 conn(0x7f2b900ac790 msgr2=0x7f2b900aa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 shutdown_connections 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 -- 172.21.15.98:0/1714331016 wait complete. 2024-09-11T03:05:36.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.701+0000 7f2b98188640 1 librados: shutdown 2024-09-11T03:05:36.756 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:36.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T03:05:36.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:05:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient: get_monmap_and_config 2024-09-11T03:05:36.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient: monmap: 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:36.878011+0000 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:36.878011+0000 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:36.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:36.877 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 1 Processor -- start 2024-09-11T03:05:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 1 -- start start 2024-09-11T03:05:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient: init 2024-09-11T03:05:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient: _add_conns ranks=[2,1,3,10,7,9,0,6,4,8,5] 2024-09-11T03:05:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcd00a3690 0x7efcd00a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient(hunting): picked mon.noname-b con 0x7efcd00a3690 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7efcd00a1eb0 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3fb0 0x7efcd0137b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient(hunting): picked mon.noname-g con 0x7efcd00a3fb0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcd00a1ca0 con 0x7efcd00a1eb0 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7efcd009ec40 con 0x7efcd00a3690 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efcd009dd20 con 0x7efcd00a3fb0 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcd579c640 10 monclient(hunting): authenticate will time out at 1890.758535s 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcd00a3690 0x7efcd00a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcd00a3690 0x7efcd00a3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49626/0 (socket says 172.21.15.98:49626) 2024-09-11T03:05:36.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 -- 172.21.15.98:0/3885170190 learned_addr learned my addr 172.21.15.98:0/3885170190 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): get_auth_request con 0x7efcd00a3690 auth_method 0 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efccf7fe640 1 --2- 172.21.15.98:0/3885170190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3fb0 0x7efcd0137b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efccf7fe640 10 monclient(hunting): get_auth_request con 0x7efcd00a3fb0 auth_method 0 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient(hunting): handle_auth_done global_id 94241 payload 274 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 -- 172.21.15.98:0/3885170190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3fb0 msgr2=0x7efcd0137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 --2- 172.21.15.98:0/3885170190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3fb0 0x7efcd0137b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 -- 172.21.15.98:0/3885170190 >> v1:172.21.15.190:6789/0 conn(0x7efcd00a1eb0 legacy=0x7efcd0137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 monclient: found mon.noname-b 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:05:36.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.873+0000 7efcceffd640 1 -- 172.21.15.98:0/3885170190 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd009d480 con 0x7efcd00a3690 2024-09-11T03:05:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcceffd640 1 --2- 172.21.15.98:0/3885170190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcd00a3690 0x7efcd00a3a70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efcc40098a0 tx=0x7efcc401c260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e69c56c4a9aede5 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 1 -- 172.21.15.98:0/3885170190 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc40b0070 con 0x7efcd00a3690 2024-09-11T03:05:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: dump: 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.892 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 1 -- 172.21.15.98:0/3885170190 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc40aa810 con 0x7efcd00a3690 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 5 monclient: authenticate success, global_id 94241 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 1 -- 172.21.15.98:0/3885170190 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc40a9300 con 0x7efcd00a3690 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccdffb640 10 monclient: dump: 2024-09-11T03:05:36.893 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 4 set_mon_vals no callback set 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient: shutdown 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- 172.21.15.98:0/3885170190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcd00a3690 msgr2=0x7efcd00a3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 --2- 172.21.15.98:0/3885170190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcd00a3690 0x7efcd00a3a70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efcc40098a0 tx=0x7efcc401c260 comp rx=0 tx=0).stop 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- 172.21.15.98:0/3885170190 shutdown_connections 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 --2- 172.21.15.98:0/3885170190 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3fb0 0x7efcd0137b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 --2- 172.21.15.98:0/3885170190 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7efcd00a3690 0x7efcd00a3a70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- 172.21.15.98:0/3885170190 >> 172.21.15.98:0/3885170190 conn(0x7efcd00a4780 msgr2=0x7efcd00a4b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- 172.21.15.98:0/3885170190 shutdown_connections 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- 172.21.15.98:0/3885170190 wait complete. 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient: build_initial_monmap 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient: monmap: 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:36.896 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 librados: starting msgr at 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 librados: starting objecter 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 Processor -- start 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- start start 2024-09-11T03:05:36.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 librados: setting wanted keys 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 librados: calling monclient init 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient: init 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient: _add_conns ranks=[1,3,0,7,6,8,5,2,4] 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd00a1eb0 0x7efcd01201a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): picked mon.noname-f con 0x7efcd00a1eb0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3690 0x7efcd0137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): picked mon.noname-g con 0x7efcd00a3690 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:36.898 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd00a3fb0 0x7efcd011e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): picked mon.noname-a con 0x7efcd00a3fb0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efcd00a1890 con 0x7efcd00a3fb0 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7efcd01382c0 con 0x7efcd00a1eb0 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efcd00a1a10 con 0x7efcd00a3690 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcd579c640 10 monclient(hunting): authenticate will time out at 1890.763653s 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd00a3fb0 0x7efcd011e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd00a3fb0 0x7efcd011e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36136/0 (socket says 172.21.15.98:36136) 2024-09-11T03:05:36.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 1 -- 172.21.15.98:0/4001639559 learned_addr learned my addr 172.21.15.98:0/4001639559 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 10 monclient(hunting): get_auth_request con 0x7efcd00a3fb0 auth_method 0 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.877+0000 7efcce7fc640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3690 0x7efcd0137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 10 monclient(hunting): get_auth_request con 0x7efcd00a3690 auth_method 0 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd00a1eb0 0x7efcd01201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): get_auth_request con 0x7efcd00a1eb0 auth_method 0 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 10 monclient(hunting): handle_auth_done global_id 45531 payload 824 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3690 msgr2=0x7efcd0137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3690 0x7efcd0137010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd00a1eb0 msgr2=0x7efcd01201a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efcd00a1eb0 0x7efcd01201a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 1 monclient: found mon.noname-a 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 1 -- 172.21.15.98:0/4001639559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd00a3230 con 0x7efcd00a3fb0 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd00a3690 0x7efcd0137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccf7fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd00a3fb0 0x7efcd011e2d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009fd0 tx=0x7efcc800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 client.?.objecter ms_handle_connect 0x7efcd00a3fb0 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:36.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc800edd0 con 0x7efcd00a3fb0 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: dump: 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 5 monclient: authenticate success, global_id 45531 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc8019070 con 0x7efcd00a3fb0 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 10 monclient: _renew_subs 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcd0137cb0 con 0x7efcd00a3fb0 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc8014d00 con 0x7efcd00a3fb0 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 10 client.45531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efccd7fa640 4 set_mon_vals no callback set 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: dump: 2024-09-11T03:05:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 10 monclient: _renew_subs 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcd0121100 con 0x7efcd00a3fb0 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 1 librados: init done 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd0137550 0x7efcd011e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efcd006ca10 con 0x7efcd0137550 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7efcc8014420 con 0x7efcd00a3fb0 2024-09-11T03:05:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd0137550 0x7efcd011e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient: get_auth_request con 0x7efcd0137550 auth_method 0 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efcbc03df40 0x7efcbc040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(636..636 src has 1..636) ==== 6761+0+0 (secure 0 0 0) 0x7efcc80501c0 con 0x7efcd00a3fb0 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 client.45531.objecter ms_dispatch 0x7efcd0000ba0 osd_map(636..636 src has 1..636) 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 3 client.45531.objecter handle_osd_map got epochs [636,636] > 0 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 3 client.45531.objecter handle_osd_map decoding full epoch 636 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 20 client.45531.objecter dump_active .. 0 homeless 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efcbc03df40 0x7efcbc040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 10 monclient: get_auth_request con 0x7efcbc03df40 auth_method 0 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 10 monclient(hunting): handle_auth_done global_id 74541 payload 824 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 1 -- 172.21.15.98:0/4001639559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efcd0068b40 con 0x7efcd0137550 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 client.45531.objecter ms_handle_connect 0x7efcd0137550 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 client.45531.objecter resend_mon_ops 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcceffd640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd0137550 0x7efcd011e810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efcd00a5530 tx=0x7efcc4019000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc40b9030 con 0x7efcd0137550 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:05:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: dump: 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 client.45531.objecter ms_handle_connect 0x7efcbc03df40 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcce7fc640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efcbc03df40 0x7efcbc040400 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 crypto rx=0x7efcc0003ec0 tx=0x7efcc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efcc40b1040 con 0x7efcd0137550 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd0137550 msgr2=0x7efcd011e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd0137550 0x7efcd011e810 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efcd00a5530 tx=0x7efcc4019000 comp rx=0 tx=0).stop 2024-09-11T03:05:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd0137550 msgr2=0x7efcd011e810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.881+0000 7efcb77fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd0137550 0x7efcd011e810 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efcd00a5530 tx=0x7efcc4019000 comp rx=0 tx=0).stop 2024-09-11T03:05:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcd579c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcd579c640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd011ed50 0x7efcd013a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcd579c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7efcd011f160 con 0x7efcd011ed50 2024-09-11T03:05:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd011ed50 0x7efcd013a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:36.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient: get_auth_request con 0x7efcd011ed50 auth_method 0 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 10 monclient(hunting): handle_auth_done global_id 74550 payload 824 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 1 -- 172.21.15.98:0/4001639559 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efcd00a3a70 con 0x7efcd011ed50 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcceffd640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd011ed50 0x7efcd013a690 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efcc4000c10 tx=0x7efcc40a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcb77fe640 10 client.45531.objecter ms_handle_connect 0x7efcd011ed50 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcb77fe640 10 client.45531.objecter resend_mon_ops 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc40b9040 con 0x7efcd011ed50 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:36.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcb77fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.889+0000 7efcb77fe640 10 monclient: dump: 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:36.916 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7efcc40bfae0 con 0x7efcd011ed50 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcb77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd011ed50 msgr2=0x7efcd013a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcb77fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd011ed50 0x7efcd013a690 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efcc4000c10 tx=0x7efcc40a8000 comp rx=0 tx=0).stop 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcb77fe640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd011ed50 msgr2=0x7efcd013a690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:36.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcb77fe640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7efcd011ed50 0x7efcd013a690 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efcc4000c10 tx=0x7efcc40a8000 comp rx=0 tx=0).stop 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 293824, 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:36.918 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:36.919 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.920 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.921 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.922 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:36.923 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:36.924 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.925 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.926 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:36.927 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:36.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.929 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.931 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.932 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.933 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.935 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.936 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.937 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:36.941 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:36.942 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:36.942 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 10 librados: watch_flush enter 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 10 librados: watch_flush exit 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 20 client.45531.objecter shutdown clearing up homeless session... 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 10 client.45531.objecter successfully canceled tick 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efcbc03df40 msgr2=0x7efcbc040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7efcbc03df40 0x7efcbc040400 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 crypto rx=0x7efcc0003ec0 tx=0x7efcc0009040 comp rx=0 tx=0).stop 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 10 monclient: shutdown 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd00a3fb0 msgr2=0x7efcd011e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:36.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd00a3fb0 0x7efcd011e2d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efcc8009fd0 tx=0x7efcc800e4b0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efccf7fe640 1 -- 172.21.15.98:0/4001639559 reap_dead start 2024-09-11T03:05:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 shutdown_connections 2024-09-11T03:05:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 --2- 172.21.15.98:0/4001639559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efcd00a3fb0 0x7efcd011e2d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 >> 172.21.15.98:0/4001639559 conn(0x7efcd00a4780 msgr2=0x7efcd0070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 shutdown_connections 2024-09-11T03:05:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 -- 172.21.15.98:0/4001639559 wait complete. 2024-09-11T03:05:36.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:36.893+0000 7efcd579c640 1 librados: shutdown 2024-09-11T03:05:36.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:36.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T03:05:36.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient: get_monmap_and_config 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient: monmap: 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:37.065448+0000 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:37.065448+0000 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:37.064 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 Processor -- start 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 -- start start 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient: init 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient: _add_conns ranks=[5,9,10,4,6,0,3,1,2,8,7] 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c139bd0 0x7f913c0a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): picked mon.noname-h con 0x7f913c139bd0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f913c139fb0 0x7f913c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): picked mon.noname-a con 0x7f913c139fb0 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f913c13a3b0 0x7f913c0a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): picked mon.noname-f con 0x7f913c13a3b0 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f913c0777d0 con 0x7f913c139fb0 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f913c0a0f60 con 0x7f913c13a3b0 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f913c070050 con 0x7f913c139bd0 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914306d640 10 monclient(hunting): authenticate will time out at 1890.945768s 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f913c139fb0 0x7f913c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f913c139fb0 0x7f913c0a4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36140/0 (socket says 172.21.15.98:36140) 2024-09-11T03:05:37.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 -- 172.21.15.98:0/1167466425 learned_addr learned my addr 172.21.15.98:0/1167466425 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): get_auth_request con 0x7f913c139fb0 auth_method 0 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914160a640 1 --2- 172.21.15.98:0/1167466425 >> v2:172.21.15.190:3300/0 conn(0x7f913c13a3b0 0x7f913c0a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f9140e09640 1 --2- 172.21.15.98:0/1167466425 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c139bd0 0x7f913c0a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914160a640 10 monclient(hunting): get_auth_request con 0x7f913c13a3b0 auth_method 0 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914160a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914160a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f914160a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient(hunting): handle_auth_done global_id 45540 payload 274 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 -- 172.21.15.98:0/1167466425 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c139bd0 msgr2=0x7f913c0a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 --2- 172.21.15.98:0/1167466425 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c139bd0 0x7f913c0a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 -- 172.21.15.98:0/1167466425 >> v2:172.21.15.190:3300/0 conn(0x7f913c13a3b0 msgr2=0x7f913c0a4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 --2- 172.21.15.98:0/1167466425 >> v2:172.21.15.190:3300/0 conn(0x7f913c13a3b0 0x7f913c0a4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 monclient: found mon.noname-a 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f913bfff640 1 -- 172.21.15.98:0/1167466425 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c0a1ac0 con 0x7f913c139fb0 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f9140e09640 10 monclient: get_auth_request con 0x7f913c139bd0 auth_method 0 2024-09-11T03:05:37.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.061+0000 7f9140e09640 1 --2- 172.21.15.98:0/1167466425 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c139bd0 0x7f913c0a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913bfff640 1 --2- 172.21.15.98:0/1167466425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c139fb0 0x7f913c0a4610 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f913c0772e0 tx=0x7f912401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af61f0f750a10f22 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 1 -- 172.21.15.98:0/1167466425 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f91240b0070 con 0x7f913c139fb0 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: dump: 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.077 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 1 -- 172.21.15.98:0/1167466425 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91240aa810 con 0x7f913c139fb0 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 5 monclient: authenticate success, global_id 45540 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 1 -- 172.21.15.98:0/1167466425 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f91240a9300 con 0x7f913c139fb0 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 4 set_mon_vals no callback set 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913b7fe640 10 monclient: dump: 2024-09-11T03:05:37.079 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.080 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient: shutdown 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- 172.21.15.98:0/1167466425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c139fb0 msgr2=0x7f913c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 --2- 172.21.15.98:0/1167466425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c139fb0 0x7f913c0a4610 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f913c0772e0 tx=0x7f912401c3e0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- 172.21.15.98:0/1167466425 shutdown_connections 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 --2- 172.21.15.98:0/1167466425 >> v2:172.21.15.190:3300/0 conn(0x7f913c13a3b0 0x7f913c0a4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 --2- 172.21.15.98:0/1167466425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f913c139fb0 0x7f913c0a4610 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 --2- 172.21.15.98:0/1167466425 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c139bd0 0x7f913c0a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- 172.21.15.98:0/1167466425 >> 172.21.15.98:0/1167466425 conn(0x7f913c06ed00 msgr2=0x7f913c06f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- 172.21.15.98:0/1167466425 shutdown_connections 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- 172.21.15.98:0/1167466425 wait complete. 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient: monmap: 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.082 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 librados: starting msgr at 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 librados: starting objecter 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 Processor -- start 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- start start 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 librados: setting wanted keys 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 librados: calling monclient init 2024-09-11T03:05:37.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient: init 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient: _add_conns ranks=[7,3,1,0,5,2,4,6,8] 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f913c139bd0 0x7f913c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): picked mon.noname-i con 0x7f913c139bd0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f913c139fb0 0x7f913c132380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): picked mon.noname-g con 0x7f913c139fb0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f913c13a3b0 0x7f913c1328c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): picked mon.noname-f con 0x7f913c13a3b0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f913c143440 con 0x7f913c13a3b0 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f913c0a54a0 con 0x7f913c139fb0 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f913c1435c0 con 0x7f913c139bd0 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914306d640 10 monclient(hunting): authenticate will time out at 1890.951507s 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914160a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f913c13a3b0 0x7f913c1328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913bfff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f913c139fb0 0x7f913c132380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f9140e09640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f913c139bd0 0x7f913c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914160a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f913c13a3b0 0x7f913c1328c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:53690/0 (socket says 172.21.15.98:53690) 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914160a640 1 -- 172.21.15.98:0/1889632636 learned_addr learned my addr 172.21.15.98:0/1889632636 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914160a640 10 monclient(hunting): get_auth_request con 0x7f913c13a3b0 auth_method 0 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914160a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914160a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f914160a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913bfff640 10 monclient(hunting): get_auth_request con 0x7f913c139fb0 auth_method 0 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f913bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f9140e09640 10 monclient(hunting): get_auth_request con 0x7f913c139bd0 auth_method 0 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f9140e09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f9140e09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.065+0000 7f9140e09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f913bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f913bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f913bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 10 monclient(hunting): handle_auth_done global_id 26155 payload 824 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f913c139bd0 msgr2=0x7f913c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f913c139bd0 0x7f913c13f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f913c139fb0 msgr2=0x7f913c132380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f913c139fb0 0x7f913c132380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 1 monclient: found mon.noname-f 2024-09-11T03:05:37.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 1 -- 172.21.15.98:0/1889632636 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913c13f1a0 con 0x7f913c13a3b0 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914160a640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f913c13a3b0 0x7f913c1328c0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f912c00d560 tx=0x7f912c00da20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 client.?.objecter ms_handle_connect 0x7f913c13a3b0 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f912c012a60 con 0x7f913c13a3b0 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: dump: 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.088 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.089 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 5 monclient: authenticate success, global_id 26155 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c019070 con 0x7f913c13a3b0 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 10 monclient: _renew_subs 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913c140680 con 0x7f913c13a3b0 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f912c012c60 con 0x7f913c13a3b0 2024-09-11T03:05:37.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f913affd640 4 set_mon_vals no callback set 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 10 client.26155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: dump: 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.091 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 10 monclient: _renew_subs 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913c13e900 con 0x7f913c13a3b0 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 1 librados: init done 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c132e00 0x7f913c13bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f913c06ca10 con 0x7f913c132e00 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c132e00 0x7f913c13bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient: get_auth_request con 0x7f913c132e00 auth_method 0 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f912c013070 con 0x7f913c13a3b0 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 10 monclient(hunting): handle_auth_done global_id 74579 payload 824 2024-09-11T03:05:37.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 1 -- 172.21.15.98:0/1889632636 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f913c138650 con 0x7f913c132e00 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f911c03df40 0x7f911c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(636..636 src has 1..636) ==== 6761+0+0 (secure 0 0 0) 0x7f912c00f040 con 0x7f913c13a3b0 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 client.26155.objecter ms_dispatch 0x7f913c000ba0 osd_map(636..636 src has 1..636) 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 3 client.26155.objecter handle_osd_map got epochs [636,636] > 0 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 3 client.26155.objecter handle_osd_map decoding full epoch 636 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f9140e09640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c132e00 0x7f913c13bbf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9134007fb0 tx=0x7f913400c8b0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 20 client.26155.objecter dump_active .. 0 homeless 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 client.26155.objecter ms_handle_connect 0x7f913c132e00 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 client.26155.objecter resend_mon_ops 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f913bfff640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f911c03df40 0x7f911c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f913bfff640 10 monclient: get_auth_request con 0x7f911c03df40 auth_method 0 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f913400bb50 con 0x7f913c132e00 2024-09-11T03:05:37.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.069+0000 7f91397fa640 10 monclient: dump: 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.095 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 10 client.26155.objecter ms_handle_connect 0x7f911c03df40 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f913bfff640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f911c03df40 0x7f911c040400 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7f91240ad000 tx=0x7f91240a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f913400b6e0 con 0x7f913c132e00 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c132e00 msgr2=0x7f913c13bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c132e00 0x7f913c13bbf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9134007fb0 tx=0x7f913400c8b0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c132e00 msgr2=0x7f913c13bbf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.073+0000 7f91397fa640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c132e00 0x7f913c13bbf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f914306d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:37.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f914306d640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c02d450 0x7f913c02d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f914306d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f913c06e870 con 0x7f913c02d450 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c02d450 0x7f913c02d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient: get_auth_request con 0x7f913c02d450 auth_method 0 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 10 monclient(hunting): handle_auth_done global_id 74588 payload 824 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 1 -- 172.21.15.98:0/1889632636 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f913c1331e0 con 0x7f913c02d450 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f91397fa640 10 client.26155.objecter ms_handle_connect 0x7f913c02d450 2024-09-11T03:05:37.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f9140e09640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c02d450 0x7f913c02d860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9134017000 tx=0x7f9134004a50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f91397fa640 10 client.26155.objecter resend_mon_ops 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9134017890 con 0x7f913c02d450 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f91397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f91397fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.077+0000 7f91397fa640 10 monclient: dump: 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.100 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9435 (secure 0 0 0) 0x7f9134017420 con 0x7f913c02d450 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f91397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f91397fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c02d450 msgr2=0x7f913c02d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f91397fa640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c02d450 0x7f913c02d860 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9134017000 tx=0x7f9134004a50 comp rx=0 tx=0).stop 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f91397fa640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c02d450 msgr2=0x7f913c02d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f91397fa640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f913c02d450 0x7f913c02d860 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 294012, 2024-09-11T03:05:37.102 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 9, 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:37.103 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.104 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:37.105 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:37.106 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:37.107 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:37.108 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.109 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.110 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.113 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:37.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.115 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.116 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:37.117 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.120 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.121 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.122 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.123 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:37.124 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:37.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:37.126 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.127 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:37.127 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 10 librados: watch_flush enter 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 10 librados: watch_flush exit 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 20 client.26155.objecter shutdown clearing up homeless session... 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 10 client.26155.objecter successfully canceled tick 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f911c03df40 msgr2=0x7f911c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f911c03df40 0x7f911c040400 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7f91240ad000 tx=0x7f91240a6000 comp rx=0 tx=0).stop 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 10 monclient: shutdown 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f913c13a3b0 msgr2=0x7f913c1328c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 --2- 172.21.15.98:0/1889632636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f913c13a3b0 0x7f913c1328c0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f912c00d560 tx=0x7f912c00da20 comp rx=0 tx=0).stop 2024-09-11T03:05:37.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914160a640 1 -- 172.21.15.98:0/1889632636 reap_dead start 2024-09-11T03:05:37.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914160a640 1 -- 172.21.15.98:0/1889632636 reap_dead start 2024-09-11T03:05:37.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 shutdown_connections 2024-09-11T03:05:37.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 >> 172.21.15.98:0/1889632636 conn(0x7f913c06ed00 msgr2=0x7f913c0706f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 shutdown_connections 2024-09-11T03:05:37.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 -- 172.21.15.98:0/1889632636 wait complete. 2024-09-11T03:05:37.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.081+0000 7f914306d640 1 librados: shutdown 2024-09-11T03:05:37.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:37.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T03:05:37.130 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient: get_monmap_and_config 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient: monmap: 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:37.249490+0000 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:37.249490+0000 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:37.248 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 Processor -- start 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 -- start start 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient: init 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient: _add_conns ranks=[10,3,9,4,6,8,1,7,5,2,0] 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): picked mon.noname-f con 0x7f5c60139b80 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): picked mon.noname-g con 0x7f5c6006fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): picked mon.noname-a con 0x7f5c60072640 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f5c600719c0 con 0x7f5c60072640 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5c60071c10 con 0x7f5c60139b80 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5c600702d0 con 0x7f5c6006fe50 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c6548a640 10 monclient(hunting): authenticate will time out at 1891.129781s 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59858/0 (socket says 172.21.15.98:59858) 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 -- 172.21.15.98:0/2305705504 learned_addr learned my addr 172.21.15.98:0/2305705504 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): get_auth_request con 0x7f5c60072640 auth_method 0 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5effd640 1 --2- 172.21.15.98:0/2305705504 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5e7fc640 1 --2- 172.21.15.98:0/2305705504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5effd640 10 monclient(hunting): get_auth_request con 0x7f5c60139b80 auth_method 0 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5e7fc640 10 monclient(hunting): get_auth_request con 0x7f5c6006fe50 auth_method 0 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient(hunting): handle_auth_done global_id 45549 payload 274 2024-09-11T03:05:37.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 -- 172.21.15.98:0/2305705504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c6006fe50 msgr2=0x7f5c600a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 --2- 172.21.15.98:0/2305705504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 -- 172.21.15.98:0/2305705504 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 msgr2=0x7f5c60139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 --2- 172.21.15.98:0/2305705504 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 monclient: found mon.noname-a 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.245+0000 7f5c5f7fe640 1 -- 172.21.15.98:0/2305705504 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c60071370 con 0x7f5c60072640 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5f7fe640 1 --2- 172.21.15.98:0/2305705504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c60072640 0x7f5c600a7f50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5c60070ad0 tx=0x7f5c5801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5566b6fa3f7e3ed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 1 -- 172.21.15.98:0/2305705504 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c580090d0 con 0x7f5c60072640 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: dump: 2024-09-11T03:05:37.255 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.256 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 5 monclient: authenticate success, global_id 45549 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 1 -- 172.21.15.98:0/2305705504 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c580a5ab0 con 0x7f5c60072640 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 1 -- 172.21.15.98:0/2305705504 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c580a5ca0 con 0x7f5c60072640 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 4 set_mon_vals no callback set 2024-09-11T03:05:37.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5dffb640 10 monclient: dump: 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.258 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: shutdown 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- 172.21.15.98:0/2305705504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c60072640 msgr2=0x7f5c600a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 --2- 172.21.15.98:0/2305705504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c60072640 0x7f5c600a7f50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5c60070ad0 tx=0x7f5c5801c8b0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- 172.21.15.98:0/2305705504 shutdown_connections 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 --2- 172.21.15.98:0/2305705504 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 --2- 172.21.15.98:0/2305705504 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 --2- 172.21.15.98:0/2305705504 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- 172.21.15.98:0/2305705504 >> 172.21.15.98:0/2305705504 conn(0x7f5c600ac730 msgr2=0x7f5c600a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- 172.21.15.98:0/2305705504 shutdown_connections 2024-09-11T03:05:37.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- 172.21.15.98:0/2305705504 wait complete. 2024-09-11T03:05:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: monmap: 2024-09-11T03:05:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:37.267 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 librados: starting msgr at 2024-09-11T03:05:37.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 librados: starting objecter 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 Processor -- start 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- start start 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 librados: setting wanted keys 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 librados: calling monclient init 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: init 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: _add_conns ranks=[0,2,1,4,3,8,6,5,7] 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c6006fe50 0x7f5c60134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): picked mon.noname-a con 0x7f5c6006fe50 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c60072640 0x7f5c601350a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): picked mon.noname-b con 0x7f5c60072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): picked mon.noname-f con 0x7f5c60139b80 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c601435c0 con 0x7f5c6006fe50 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5c60143440 con 0x7f5c60139b80 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5c60143740 con 0x7f5c60072640 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient(hunting): authenticate will time out at 1891.134833s 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c6006fe50 0x7f5c60134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c6006fe50 0x7f5c60134b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59872/0 (socket says 172.21.15.98:59872) 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 -- 172.21.15.98:0/3657485792 learned_addr learned my addr 172.21.15.98:0/3657485792 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): get_auth_request con 0x7f5c6006fe50 auth_method 0 2024-09-11T03:05:37.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c60072640 0x7f5c601350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 10 monclient(hunting): get_auth_request con 0x7f5c60072640 auth_method 0 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5f7fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient(hunting): handle_auth_done global_id 45558 payload 824 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c60072640 msgr2=0x7f5c601350a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5c60072640 0x7f5c601350a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60139b80 msgr2=0x7f5c6013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 monclient: found mon.noname-a 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:05:37.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 -- 172.21.15.98:0/3657485792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c600aa210 con 0x7f5c6006fe50 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5f7fe640 10 monclient: get_auth_request con 0x7f5c60139b80 auth_method 0 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5f7fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5effd640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c6006fe50 0x7f5c60134b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400d5f0 tx=0x7f5c5400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 client.?.objecter ms_handle_connect 0x7f5c6006fe50 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c54012a60 con 0x7f5c6006fe50 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: dump: 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.273 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.274 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 5 monclient: authenticate success, global_id 45558 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c54019070 con 0x7f5c6006fe50 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: _renew_subs 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c600a9e30 con 0x7f5c6006fe50 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c54012c20 con 0x7f5c6006fe50 2024-09-11T03:05:37.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c5d7fa640 4 set_mon_vals no callback set 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 client.45558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c477fe640 10 monclient: dump: 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.276 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: _renew_subs 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6013d790 con 0x7f5c6006fe50 2024-09-11T03:05:37.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.249+0000 7f5c6548a640 1 librados: init done 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c6548a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c6548a640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f5c6013e0c0 con 0x7f5c6013abf0 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5c54013070 con 0x7f5c6006fe50 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4c03df40 0x7f5c4c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient: get_auth_request con 0x7f5c6013abf0 auth_method 0 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(636..636 src has 1..636) ==== 6761+0+0 (secure 0 0 0) 0x7f5c5400f040 con 0x7f5c6006fe50 2024-09-11T03:05:37.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 client.45558.objecter ms_dispatch 0x7f5c60000ba0 osd_map(636..636 src has 1..636) 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 3 client.45558.objecter handle_osd_map got epochs [636,636] > 0 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 3 client.45558.objecter handle_osd_map decoding full epoch 636 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 20 client.45558.objecter dump_active .. 0 homeless 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5f7fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4c03df40 0x7f5c4c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5f7fe640 10 monclient: get_auth_request con 0x7f5c4c03df40 auth_method 0 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_done global_id 20284 payload 824 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 1 -- 172.21.15.98:0/3657485792 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c60139f60 con 0x7f5c6013abf0 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 client.45558.objecter ms_handle_connect 0x7f5c6013abf0 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5e7fc640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f5c600a5bb0 tx=0x7f5c50010c80 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 client.45558.objecter resend_mon_ops 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 client.45558.objecter ms_handle_connect 0x7f5c4c03df40 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c5f7fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4c03df40 0x7f5c4c040400 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7f5c580ae000 tx=0x7f5c580a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c50018020 con 0x7f5c6013abf0 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 monclient: dump: 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.280 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c50004240 con 0x7f5c6013abf0 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013abf0 msgr2=0x7f5c6013afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f5c600a5bb0 tx=0x7f5c50010c80 comp rx=0 tx=0).stop 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013abf0 msgr2=0x7f5c6013afd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.253+0000 7f5c477fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c6548a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c6548a640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013b510 0x7f5c600a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c6548a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f5c6013df20 con 0x7f5c6013b510 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013b510 0x7f5c600a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient: get_auth_request con 0x7f5c6013b510 auth_method 0 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 10 monclient(hunting): handle_auth_done global_id 20293 payload 824 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 1 -- 172.21.15.98:0/3657485792 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c60069090 con 0x7f5c6013b510 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c5e7fc640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013b510 0x7f5c600a0e60 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f5c600a5bb0 tx=0x7f5c500046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 10 client.45558.objecter ms_handle_connect 0x7f5c6013b510 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 10 client.45558.objecter resend_mon_ops 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c50018040 con 0x7f5c6013b510 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 10 monclient: dump: 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9271 (secure 0 0 0) 0x7f5c5000ee60 con 0x7f5c6013b510 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013b510 msgr2=0x7f5c600a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013b510 0x7f5c600a0e60 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f5c600a5bb0 tx=0x7f5c500046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013b510 msgr2=0x7f5c600a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.261+0000 7f5c477fe640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6013b510 0x7f5c600a0e60 secure :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f5c600a5bb0 tx=0x7f5c500046d0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:37.285 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 106, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 953038, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:05:37.286 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 10, 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:37.287 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:37.288 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:05:37.289 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:05:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.291 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:05:37.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.293 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.294 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.295 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.296 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:05:37.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.299 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.300 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.301 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.302 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.303 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.304 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:05:37.305 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:05:37.306 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:05:37.307 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:05:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:05:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T03:05:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:05:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:05:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:05:37.308 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:05:37.308 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 10 librados: watch_flush enter 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 10 librados: watch_flush exit 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 20 client.45558.objecter shutdown clearing up homeless session... 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 10 client.45558.objecter successfully canceled tick 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4c03df40 msgr2=0x7f5c4c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4c03df40 0x7f5c4c040400 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7f5c580ae000 tx=0x7f5c580a7070 comp rx=0 tx=0).stop 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 10 monclient: shutdown 2024-09-11T03:05:37.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c6006fe50 msgr2=0x7f5c60134b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 --2- 172.21.15.98:0/3657485792 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c6006fe50 0x7f5c60134b60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400d5f0 tx=0x7f5c5400dab0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c5f7fe640 1 -- 172.21.15.98:0/3657485792 reap_dead start 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c5f7fe640 1 -- 172.21.15.98:0/3657485792 reap_dead start 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 shutdown_connections 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 >> 172.21.15.98:0/3657485792 conn(0x7f5c600ac730 msgr2=0x7f5c600aa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 shutdown_connections 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 -- 172.21.15.98:0/3657485792 wait complete. 2024-09-11T03:05:37.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.265+0000 7f5c6548a640 1 librados: shutdown 2024-09-11T03:05:37.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:05:37.312 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a', 'i', 'g'] 2024-09-11T03:05:37.312 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b', 'e'] 2024-09-11T03:05:37.312 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-11T03:05:37.312 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-09-11T03:05:37.312 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-09-11T03:05:37.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.429+0000 7f9f4cf99640 10 monclient: get_monmap_and_config 2024-09-11T03:05:37.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.429+0000 7f9f4cf99640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.429+0000 7f9f4cf99640 10 monclient: monmap: 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:37.435962+0000 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:37.435962+0000 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.434 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:37.435 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:37.435 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.429+0000 7f9f4cf99640 1 Processor -- start 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.429+0000 7f9f4cf99640 1 -- start start 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.429+0000 7f9f4cf99640 10 monclient: init 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient: _add_conns ranks=[3,10,5,2,4,0,7,6,9,8,1] 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f48139b10 0x7f9f480725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): picked mon.noname-g con 0x7f9f48139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9f4806fe30 0x7f9f48139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): picked mon.noname-f con 0x7f9f4806fe30 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f480a7a10 0x7f9f480a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): picked mon.noname-h con 0x7f9f480a7a10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f9f480a9650 con 0x7f9f4806fe30 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9f48071350 con 0x7f9f48139b10 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9f48071630 con 0x7f9f480a7a10 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4cf99640 10 monclient(hunting): authenticate will time out at 1891.316252s 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f480a7a10 0x7f9f480a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f48139b10 0x7f9f480725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9f4806fe30 0x7f9f48139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f480a7a10 0x7f9f480a7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:36944/0 (socket says 172.21.15.98:36944) 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 -- 172.21.15.98:0/267280325 learned_addr learned my addr 172.21.15.98:0/267280325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): get_auth_request con 0x7f9f480a7a10 auth_method 0 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 10 monclient(hunting): get_auth_request con 0x7f9f48139b10 auth_method 0 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 10 monclient(hunting): get_auth_request con 0x7f9f4806fe30 auth_method 0 2024-09-11T03:05:37.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient(hunting): handle_auth_done global_id 74597 payload 274 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 -- 172.21.15.98:0/267280325 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f48139b10 msgr2=0x7f9f480725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 --2- 172.21.15.98:0/267280325 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f48139b10 0x7f9f480725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 -- 172.21.15.98:0/267280325 >> v2:172.21.15.190:3300/0 conn(0x7f9f4806fe30 msgr2=0x7f9f48139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 --2- 172.21.15.98:0/267280325 >> v2:172.21.15.190:3300/0 conn(0x7f9f4806fe30 0x7f9f48139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 monclient: found mon.noname-h 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f46d9d640 1 -- 172.21.15.98:0/267280325 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f480702b0 con 0x7f9f480a7a10 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.433+0000 7f9f45d9b640 1 --2- 172.21.15.98:0/267280325 >> v2:172.21.15.190:3300/0 conn(0x7f9f4806fe30 0x7f9f48139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 1 --2- 172.21.15.98:0/267280325 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f480a7a10 0x7f9f480a7ec0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009e50 tx=0x7f9f3801c8f0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=11e5b5beb8675893 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 1 -- 172.21.15.98:0/267280325 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9f380090d0 con 0x7f9f480a7a10 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: dump: 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.451 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 1 -- 172.21.15.98:0/267280325 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f380a59a0 con 0x7f9f480a7a10 2024-09-11T03:05:37.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 5 monclient: authenticate success, global_id 74597 2024-09-11T03:05:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:05:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 1 -- 172.21.15.98:0/267280325 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9f380ad3f0 con 0x7f9f480a7a10 2024-09-11T03:05:37.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:37.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 4 set_mon_vals no callback set 2024-09-11T03:05:37.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:05:37.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4559a640 10 monclient: dump: 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.457 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient: shutdown 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- 172.21.15.98:0/267280325 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f480a7a10 msgr2=0x7f9f480a7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/267280325 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f480a7a10 0x7f9f480a7ec0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9f38009e50 tx=0x7f9f3801c8f0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- 172.21.15.98:0/267280325 shutdown_connections 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/267280325 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f480a7a10 0x7f9f480a7ec0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/267280325 >> v2:172.21.15.190:3300/0 conn(0x7f9f4806fe30 0x7f9f48139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/267280325 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f9f48139b10 0x7f9f480725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- 172.21.15.98:0/267280325 >> 172.21.15.98:0/267280325 conn(0x7f9f480ac770 msgr2=0x7f9f480a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- 172.21.15.98:0/267280325 shutdown_connections 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- 172.21.15.98:0/267280325 wait complete. 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient: monmap: 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:37.459 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 librados: starting msgr at 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 librados: starting objecter 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 Processor -- start 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- start start 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 librados: setting wanted keys 2024-09-11T03:05:37.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 librados: calling monclient init 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient: init 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient: _add_conns ranks=[8,5,6,3,2,7,1,0,4] 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9f480a7a10 0x7f9f48133760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): picked mon.noname-e con 0x7f9f480a7a10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f48139b10 0x7f9f48133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): picked mon.noname-h con 0x7f9f48139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9f481341e0 0x7f9f4813a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): picked mon.noname-d con 0x7f9f481341e0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9f481435c0 con 0x7f9f48139b10 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f9f48143440 con 0x7f9f481341e0 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f9f48143740 con 0x7f9f480a7a10 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4cf99640 10 monclient(hunting): authenticate will time out at 1891.323574s 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9f480a7a10 0x7f9f48133760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9f480a7a10 0x7f9f48133760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44060/0 (socket says 172.21.15.98:44060) 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 1 -- 172.21.15.98:0/2861334865 learned_addr learned my addr 172.21.15.98:0/2861334865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 10 monclient(hunting): get_auth_request con 0x7f9f480a7a10 auth_method 0 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9f481341e0 0x7f9f4813a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 10 monclient(hunting): get_auth_request con 0x7f9f481341e0 auth_method 0 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f45d9b640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f48139b10 0x7f9f48133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f45d9b640 10 monclient(hunting): get_auth_request con 0x7f9f48139b10 auth_method 0 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.437+0000 7f9f45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 10 monclient(hunting): handle_auth_done global_id 25361 payload 824 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9f481341e0 msgr2=0x7f9f4813a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9f481341e0 0x7f9f4813a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f48139b10 msgr2=0x7f9f48133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f48139b10 0x7f9f48133ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 1 monclient: found mon.noname-e 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 1 -- 172.21.15.98:0/2861334865 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f480a58a0 con 0x7f9f480a7a10 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f46d9d640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f9f481341e0 0x7f9f4813a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:05:37.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9f48139b10 0x7f9f48133ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4659c640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9f480a7a10 0x7f9f48133760 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000e8a0 tx=0x7f9f4000eca0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 client.?.objecter ms_handle_connect 0x7f9f480a7a10 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9f40017070 con 0x7f9f480a7a10 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: dump: 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.465 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4000cdb0 con 0x7f9f480a7a10 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 5 monclient: authenticate success, global_id 25361 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 10 monclient: _renew_subs 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f48073720 con 0x7f9f480a7a10 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9f400138e0 con 0x7f9f480a7a10 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 10 client.25361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f44d99640 4 set_mon_vals no callback set 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: dump: 2024-09-11T03:05:37.467 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.468 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 10 monclient: _renew_subs 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f481345c0 con 0x7f9f480a7a10 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 1 librados: init done 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9f40013470 con 0x7f9f480a7a10 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4806fe30 0x7f9f4813acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4806ca10 con 0x7f9f4806fe30 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4806fe30 0x7f9f4813acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient: get_auth_request con 0x7f9f4806fe30 auth_method 0 2024-09-11T03:05:37.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9f2803df70 0x7f9f28040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(636..636 src has 1..636) ==== 6761+0+0 (secure 0 0 0) 0x7f9f4000f040 con 0x7f9f480a7a10 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 client.25361.objecter ms_dispatch 0x7f9f48000ba0 osd_map(636..636 src has 1..636) 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 3 client.25361.objecter handle_osd_map got epochs [636,636] > 0 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 3 client.25361.objecter handle_osd_map decoding full epoch 636 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 20 client.25361.objecter dump_active .. 0 homeless 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_done global_id 45567 payload 824 2024-09-11T03:05:37.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 1 -- 172.21.15.98:0/2861334865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f480731d0 con 0x7f9f4806fe30 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f46d9d640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9f2803df70 0x7f9f28040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 client.25361.objecter ms_handle_connect 0x7f9f4806fe30 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 client.25361.objecter resend_mon_ops 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f45d9b640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4806fe30 0x7f9f4813acf0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f480a99e0 tx=0x7f9f3c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f46d9d640 10 monclient: get_auth_request con 0x7f9f2803df70 auth_method 0 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9f3c00fb80 con 0x7f9f4806fe30 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: dump: 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.471 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f3c00f710 con 0x7f9f4806fe30 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4806fe30 msgr2=0x7f9f4813acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4806fe30 0x7f9f4813acf0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f480a99e0 tx=0x7f9f3c011a60 comp rx=0 tx=0).stop 2024-09-11T03:05:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4806fe30 msgr2=0x7f9f4813acf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4806fe30 0x7f9f4813acf0 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f480a99e0 tx=0x7f9f3c011a60 comp rx=0 tx=0).stop 2024-09-11T03:05:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f36ffd640 10 client.25361.objecter ms_handle_connect 0x7f9f2803df70 2024-09-11T03:05:37.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.441+0000 7f9f46d9d640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9f2803df70 0x7f9f28040430 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7f9f380ae000 tx=0x7f9f380a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T03:05:37.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4813b230 0x7f9f480a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4806e870 con 0x7f9f4813b230 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4813b230 0x7f9f480a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient: get_auth_request con 0x7f9f4813b230 auth_method 0 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 10 monclient(hunting): handle_auth_done global_id 45576 payload 824 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 1 -- 172.21.15.98:0/2861334865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f9f4813cf20 con 0x7f9f4813b230 2024-09-11T03:05:37.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f45d9b640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4813b230 0x7f9f480a0e60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f480a99e0 tx=0x7f9f3c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 10 client.25361.objecter ms_handle_connect 0x7f9f4813b230 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 10 client.25361.objecter resend_mon_ops 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9f3c0066c0 con 0x7f9f4813b230 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 10 monclient: dump: 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.476 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f9f3c006250 con 0x7f9f4813b230 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4813b230 msgr2=0x7f9f480a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4813b230 0x7f9f480a0e60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f480a99e0 tx=0x7f9f3c004740 comp rx=0 tx=0).stop 2024-09-11T03:05:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4813b230 msgr2=0x7f9f480a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f36ffd640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9f4813b230 0x7f9f480a0e60 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f480a99e0 tx=0x7f9f3c004740 comp rx=0 tx=0).stop 2024-09-11T03:05:37.478 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-09-11T03:05:37.478 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T03:05:37.478 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 10 librados: watch_flush enter 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 10 librados: watch_flush exit 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 20 client.25361.objecter shutdown clearing up homeless session... 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 10 client.25361.objecter successfully canceled tick 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9f2803df70 msgr2=0x7f9f28040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f9f2803df70 0x7f9f28040430 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7f9f380ae000 tx=0x7f9f380a7070 comp rx=0 tx=0).stop 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 10 monclient: shutdown 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9f480a7a10 msgr2=0x7f9f48133760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9f480a7a10 0x7f9f48133760 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000e8a0 tx=0x7f9f4000eca0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f46d9d640 1 -- 172.21.15.98:0/2861334865 reap_dead start 2024-09-11T03:05:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 shutdown_connections 2024-09-11T03:05:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 --2- 172.21.15.98:0/2861334865 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f9f480a7a10 0x7f9f48133760 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 >> 172.21.15.98:0/2861334865 conn(0x7f9f480ac770 msgr2=0x7f9f480ab810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 shutdown_connections 2024-09-11T03:05:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 -- 172.21.15.98:0/2861334865 wait complete. 2024-09-11T03:05:37.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.449+0000 7f9f4cf99640 1 librados: shutdown 2024-09-11T03:05:37.481 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-11T03:05:37.481 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T03:05:37.482 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:05:37.499 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T03:05:37.499 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-09-11T03:05:37.499 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i store 2024-09-11T03:05:37.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i sync_force --yes-i-really-mean-it 2024-09-11T03:05:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient: get_monmap_and_config 2024-09-11T03:05:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient: monmap: 2024-09-11T03:05:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.664 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:37.666579+0000 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:37.666579+0000 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.665 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:37.666 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:37.666 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 1 Processor -- start 2024-09-11T03:05:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 1 -- start start 2024-09-11T03:05:37.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient: init 2024-09-11T03:05:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient: _add_conns ranks=[0,10,1,6,7,9,3,5,2,4,8] 2024-09-11T03:05:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff30139b10 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff3006fe30 0x7eff300725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient(hunting): picked mon.noname-f con 0x7eff3006fe30 addr v2:172.21.15.190:3300/0 2024-09-11T03:05:37.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7eff3013ea00 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff3013a600 con 0x7eff30139b10 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff300a9650 con 0x7eff3013ea00 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff35eef640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7eff30139b10 legacy=0x7eff30139ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7eff30071350 con 0x7eff3006fe30 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff38153640 10 monclient(hunting): authenticate will time out at 1891.547026s 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff3006fe30 0x7eff300725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7eff3006fe30 0x7eff300725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:50814/0 (socket says 172.21.15.98:50814) 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 1 -- 172.21.15.98:0/2993446099 learned_addr learned my addr 172.21.15.98:0/2993446099 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): get_auth_request con 0x7eff3006fe30 auth_method 0 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.661+0000 7eff356ee640 10 monclient(hunting): handle_auth_done global_id 26182 payload 274 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 1 -- 172.21.15.98:0/2993446099 >> v1:172.21.15.190:6789/0 conn(0x7eff3013ea00 legacy=0x7eff30140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 1 -- 172.21.15.98:0/2993446099 >> v1:172.21.15.98:6789/0 conn(0x7eff30139b10 legacy=0x7eff30139ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 1 monclient: found mon.noname-f 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:05:37.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 1 -- 172.21.15.98:0/2993446099 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff30071630 con 0x7eff3006fe30 2024-09-11T03:05:37.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff356ee640 1 --2- 172.21.15.98:0/2993446099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3006fe30 0x7eff300725d0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009e50 tx=0x7eff2c01c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b813d5715da8c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.680 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 1 -- 172.21.15.98:0/2993446099 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff2c0b0070 con 0x7eff3006fe30 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: dump: 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.681 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 1 -- 172.21.15.98:0/2993446099 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2c0090d0 con 0x7eff3006fe30 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 5 monclient: authenticate success, global_id 26182 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 10 monclient: get_monmap_and_config success 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 4 set_mon_vals no callback set 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 1 -- 172.21.15.98:0/2993446099 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff2c007860 con 0x7eff3006fe30 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff34eed640 10 monclient: dump: 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.683 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.684 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.685 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.685 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.685 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.685 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.685 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 10 monclient: shutdown 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 1 -- 172.21.15.98:0/2993446099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3006fe30 msgr2=0x7eff300725d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 1 --2- 172.21.15.98:0/2993446099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3006fe30 0x7eff300725d0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009e50 tx=0x7eff2c01c5d0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 1 -- 172.21.15.98:0/2993446099 shutdown_connections 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 1 --2- 172.21.15.98:0/2993446099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7eff3006fe30 0x7eff300725d0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 1 -- 172.21.15.98:0/2993446099 >> 172.21.15.98:0/2993446099 conn(0x7eff300ac770 msgr2=0x7eff300a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 1 -- 172.21.15.98:0/2993446099 shutdown_connections 2024-09-11T03:05:37.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.665+0000 7eff38153640 1 -- 172.21.15.98:0/2993446099 wait complete. 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: build_initial_monmap 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: monmap: 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:37.687 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 librados: starting msgr at 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 librados: starting objecter 2024-09-11T03:05:37.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 Processor -- start 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 -- start start 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 librados: setting wanted keys 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 librados: calling monclient init 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: init 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: _add_conns ranks=[7,5,8,0,2,6,3,1,4] 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff30139b10 0x7eff301385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): picked mon.noname-i con 0x7eff30139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:05:37.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff3013ea00 0x7eff30138b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): picked mon.noname-h con 0x7eff3013ea00 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff300a22a0 0x7eff300a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): picked mon.noname-e con 0x7eff300a22a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7eff300517e0 con 0x7eff3013ea00 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7eff30145d80 con 0x7eff30139b10 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7eff30145f00 con 0x7eff300a22a0 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:37.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): authenticate will time out at 1891.553560s 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff300a22a0 0x7eff300a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff300a22a0 0x7eff300a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44066/0 (socket says 172.21.15.98:44066) 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 -- 172.21.15.98:0/1710904069 learned_addr learned my addr 172.21.15.98:0/1710904069 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): get_auth_request con 0x7eff300a22a0 auth_method 0 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff30139b10 0x7eff301385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff356ee640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff3013ea00 0x7eff30138b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient(hunting): get_auth_request con 0x7eff30139b10 auth_method 0 2024-09-11T03:05:37.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff356ee640 10 monclient(hunting): get_auth_request con 0x7eff3013ea00 auth_method 0 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff356ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff356ee640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff356ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient(hunting): handle_auth_done global_id 25370 payload 824 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff30139b10 msgr2=0x7eff301385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff30139b10 0x7eff301385f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff3013ea00 msgr2=0x7eff30138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff3013ea00 0x7eff30138b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient: _finish_hunting 0 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 monclient: found mon.noname-e 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 -- 172.21.15.98:0/1710904069 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff300a48f0 con 0x7eff300a22a0 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff30139b10 0x7eff301385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff356ee640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7eff3013ea00 0x7eff30138b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff366f0640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff300a22a0 0x7eff300a2700 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7eff2000c390 tx=0x7eff2000c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 client.?.objecter ms_handle_connect 0x7eff300a22a0 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff20017070 con 0x7eff300a22a0 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:05:37.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: dump: 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.694 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.695 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.695 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.695 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.695 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: _finish_auth 0 2024-09-11T03:05:37.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: _check_auth_tickets 2024-09-11T03:05:37.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:05:37.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 5 monclient: authenticate success, global_id 25370 2024-09-11T03:05:37.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff20003600 con 0x7eff300a22a0 2024-09-11T03:05:37.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: _renew_subs 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff30141e60 con 0x7eff300a22a0 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff20003ca0 con 0x7eff300a22a0 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff27fff640 4 set_mon_vals no callback set 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 client.25370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 monclient: dump: 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.697 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.698 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: _renew_subs 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7eff300a89b0 con 0x7eff300a22a0 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 librados: init done 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7eff20003830 con 0x7eff300a22a0 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff3006fe10 0x7eff300a2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7eff3006ca10 con 0x7eff3006fe10 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff1c03df70 0x7eff1c040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(636..636 src has 1..636) ==== 6761+0+0 (secure 0 0 0) 0x7eff2004eb20 con 0x7eff300a22a0 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 10 client.25370.objecter ms_dispatch 0x7eff30000ba0 osd_map(636..636 src has 1..636) 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 3 client.25370.objecter handle_osd_map got epochs [636,636] > 0 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 3 client.25370.objecter handle_osd_map decoding full epoch 636 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff267fc640 20 client.25370.objecter dump_active .. 0 homeless 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff3006fe10 0x7eff300a2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient: get_auth_request con 0x7eff3006fe10 auth_method 0 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.669+0000 7eff35eef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff356ee640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff1c03df70 0x7eff1c040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff356ee640 10 monclient: get_auth_request con 0x7eff1c03df70 auth_method 0 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff35eef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff35eef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff35eef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff35eef640 10 monclient(hunting): handle_auth_done global_id 20311 payload 824 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff35eef640 1 -- 172.21.15.98:0/1710904069 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff300725d0 con 0x7eff3006fe10 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 client.25370.objecter ms_handle_connect 0x7eff1c03df70 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff356ee640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff1c03df70 0x7eff1c040430 secure :-1 s=READY pgs=3684 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0ad000 tx=0x7eff2c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 client.25370.objecter ms_handle_connect 0x7eff3006fe10 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 client.25370.objecter resend_mon_ops 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff35eef640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff3006fe10 0x7eff300a2c40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7eff2800d700 tx=0x7eff2800dbc0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff280039c0 con 0x7eff3006fe10 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 monclient: dump: 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.702 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7eff28005a50 con 0x7eff3006fe10 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff3006fe10 msgr2=0x7eff300a2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff3006fe10 0x7eff300a2c40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7eff2800d700 tx=0x7eff2800dbc0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff3006fe10 msgr2=0x7eff300a2c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.673+0000 7eff267fc640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff3006fe10 0x7eff300a2c40 secure :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7eff2800d700 tx=0x7eff2800dbc0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T03:05:37.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff300a3180 0x7eff3013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7eff3006e870 con 0x7eff300a3180 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff300a3180 0x7eff3013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient: get_auth_request con 0x7eff300a3180 auth_method 0 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 10 monclient(hunting): handle_auth_done global_id 20320 payload 824 2024-09-11T03:05:37.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 1 -- 172.21.15.98:0/1710904069 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7eff3006d210 con 0x7eff300a3180 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff35eef640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff300a3180 0x7eff3013a7d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7eff28015000 tx=0x7eff280059e0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 10 client.25370.objecter ms_handle_connect 0x7eff300a3180 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 10 client.25370.objecter resend_mon_ops 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff2802a080 con 0x7eff300a3180 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 10 monclient: dump: 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:37.707 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7eff2801b760 con 0x7eff300a3180 2024-09-11T03:05:37.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-11T03:05:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:05:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff300a3180 msgr2=0x7eff3013a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff300a3180 0x7eff3013a7d0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7eff28015000 tx=0x7eff280059e0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff300a3180 msgr2=0x7eff3013a7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:05:37.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff267fc640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7eff300a3180 0x7eff3013a7d0 secure :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7eff28015000 tx=0x7eff280059e0 comp rx=0 tx=0).stop 2024-09-11T03:05:37.709 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:05:37.710 INFO:teuthology.orchestra.run.smithi098.stdout: "ret": 0, 2024-09-11T03:05:37.710 INFO:teuthology.orchestra.run.smithi098.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-11T03:05:37.710 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 10 librados: watch_flush enter 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 10 librados: watch_flush exit 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 20 client.25370.objecter shutdown clearing up homeless session... 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 10 client.25370.objecter successfully canceled tick 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff1c03df70 msgr2=0x7eff1c040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7eff1c03df70 0x7eff1c040430 secure :-1 s=READY pgs=3684 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0ad000 tx=0x7eff2c0a6000 comp rx=0 tx=0).stop 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 10 monclient: shutdown 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff300a22a0 msgr2=0x7eff300a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:37.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 --2- 172.21.15.98:0/1710904069 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7eff300a22a0 0x7eff300a2700 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7eff2000c390 tx=0x7eff2000c850 comp rx=0 tx=0).stop 2024-09-11T03:05:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff366f0640 1 -- 172.21.15.98:0/1710904069 reap_dead start 2024-09-11T03:05:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff366f0640 1 -- 172.21.15.98:0/1710904069 reap_dead start 2024-09-11T03:05:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 shutdown_connections 2024-09-11T03:05:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 >> 172.21.15.98:0/1710904069 conn(0x7eff300ac770 msgr2=0x7eff30140df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:05:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 shutdown_connections 2024-09-11T03:05:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 -- 172.21.15.98:0/1710904069 wait complete. 2024-09-11T03:05:37.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:37.681+0000 7eff38153640 1 librados: shutdown 2024-09-11T03:05:37.713 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-09-11T03:05:37.713 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-11T03:05:37.714 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:05:37.757 INFO:tasks.ceph.mon.i:Stopped 2024-09-11T03:05:37.757 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-09-11T03:05:37.757 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-09-11T03:05:37.758 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-11T03:05:37.758 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:05:37.775 INFO:tasks.ceph.mon.g:Stopped 2024-09-11T03:05:37.775 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T03:05:37.776 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T03:05:37.776 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-09-11T03:05:37.776 INFO:tasks.ceph.mon.e:Sent signal 19 2024-09-11T03:05:37.776 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T03:05:52.777 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T03:05:52.778 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T03:05:52.778 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-09-11T03:05:52.778 INFO:tasks.ceph.mon.e:Sent signal 18 2024-09-11T03:05:52.778 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 6 2024-09-11T03:05:52.779 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:05:52.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient: get_monmap_and_config 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient: build_initial_monmap 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient: monmap: 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:05:52.914290+0000 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:05:52.914290+0000 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:05:52.912 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:05:52.913 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:05:52.914 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:05:52.914 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:05:52.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 1 Processor -- start 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 1 -- start start 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient: init 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient: _reopen_session rank -1 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient: _add_conns ranks=[8,6,1,4,10,3,2,9,7,0,5] 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient(hunting): picked mon.noname-e con 0x7f79f40a4030 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient(hunting): picked mon.noname-d con 0x7f79f409ca50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f79f409d370 addr v1:172.21.15.190:6789/0 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f409ecc0 con 0x7f79f409d370 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f79f40a1470 con 0x7f79f409ca50 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f79f409e320 con 0x7f79f40a4030 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79fa543640 10 monclient(hunting): authenticate will time out at 1906.794875s 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:58838/0 (socket says 172.21.15.98:58838) 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:53428/0 (socket says 172.21.15.98:53428) 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 learned_addr learned my addr 172.21.15.98:0/1097432954 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79f409ca50 auth_method 0 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 10 monclient(hunting): get_auth_request con 0x7f79f40a4030 auth_method 0 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:52.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:52.909+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 10 monclient(hunting): get_auth_request con 0x7f79f40a4030 auth_method 0 2024-09-11T03:05:53.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79f409ca50 auth_method 0 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:53.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:53.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.113+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 10 monclient(hunting): get_auth_request con 0x7f79f40a4030 auth_method 0 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79f409ca50 auth_method 0 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:53.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:05:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:53.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:53.517+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:05:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:54.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 10 monclient(hunting): get_auth_request con 0x7f79f40a4030 auth_method 0 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79f409ca50 auth_method 0 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:54.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:54.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:54.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:05:54.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:54.321+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:05:55.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): tick 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 monclient(hunting): continuing hunt 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 msgr2=0x7f79f40a1d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79f40a4030 0x7f79f40a1d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 msgr2=0x7f79f409ce30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f79f409ca50 0x7f79f409ce30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> v1:172.21.15.190:6789/0 conn(0x7f79f409d370 legacy=0x7f79f413c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient: _add_conns ranks=[9,5,4,10,3,2,0,7] 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 0x7f79e0005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:55.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f79e0003880 addr v2:172.21.15.98:3300/0 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-h con 0x7f79e0006290 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f79e0008dd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f79e000b970 con 0x7f79e0003880 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f79e000baf0 con 0x7f79e0008dd0 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f79e000bcf0 con 0x7f79e0006290 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f1ffb640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:55.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 msgr2=0x7f79e0005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 0x7f79e0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request con 0x7f79e0008dd0 auth_method 0 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0006290 auth_method 0 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:55.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:55.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:55.909+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 msgr2=0x7f79e0005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:05:56.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 0x7f79e0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0006290 auth_method 0 2024-09-11T03:05:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:56.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:56.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.113+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request con 0x7f79e0008dd0 auth_method 0 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:56.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:56.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.117+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 msgr2=0x7f79e0005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:05:56.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 0x7f79e0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:05:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0006290 auth_method 0 2024-09-11T03:05:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:56.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:56.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.517+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:05:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request con 0x7f79e0008dd0 auth_method 0 2024-09-11T03:05:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:56.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:56.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:56.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:56.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:56.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:56.521+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:05:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 msgr2=0x7f79e0005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:05:57.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 0x7f79e0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:05:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0006290 auth_method 0 2024-09-11T03:05:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:57.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:57.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.321+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:05:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request con 0x7f79e0008dd0 auth_method 0 2024-09-11T03:05:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:57.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:57.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:57.325+0000 7f79f8ae0640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:05:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): tick 2024-09-11T03:05:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T03:05:58.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 monclient(hunting): continuing hunt 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 msgr2=0x7f79e0008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f79e0006290 0x7f79e0008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 msgr2=0x7f79e000b270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f79e0008dd0 0x7f79e000b270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 msgr2=0x7f79e0005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.98:3300/0 conn(0x7f79e0003880 0x7f79e0005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient: _add_conns ranks=[3,2,0,7,10] 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 0x7f79e000e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 reap_dead start 2024-09-11T03:05:58.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-g con 0x7f79e000c380 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 reap_dead start 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f79e0003c80 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79e00097a0 addr v1:172.21.15.98:6789/0 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f409f4a0 con 0x7f79e00097a0 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f79e000b970 con 0x7f79e0003c80 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f79e000baf0 con 0x7f79e000c380 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f1ffb640 10 monclient(hunting): _renew_subs 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> v1:172.21.15.98:6789/0 conn(0x7f79e00097a0 legacy=0x7f79e0009b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T03:05:58.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 msgr2=0x7f79e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 0x7f79e000e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0003c80 auth_method 0 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:58.913+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:05:59.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> v1:172.21.15.98:6789/0 conn(0x7f79e00097a0 legacy=0x7f79e0009b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T03:05:59.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 msgr2=0x7f79e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T03:05:59.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 0x7f79e000e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:59.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0003c80 auth_method 0 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:59.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.117+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:05:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> v1:172.21.15.98:6789/0 conn(0x7f79e00097a0 legacy=0x7f79e0009b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T03:05:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 msgr2=0x7f79e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T03:05:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:05:59.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 0x7f79e000e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0003c80 auth_method 0 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:05:59.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:05:59.521+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:06:00.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 >> v1:172.21.15.98:6789/0 conn(0x7f79e00097a0 legacy=0x7f79e0009b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T03:06:00.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 msgr2=0x7f79e000e790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T03:06:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 0x7f79e000e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:06:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79e0003c80 auth_method 0 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:06:00.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:00.325+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient(hunting): tick 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 monclient(hunting): continuing hunt 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 msgr2=0x7f79e000e790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f79e000c380 0x7f79e000e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 msgr2=0x7f79e0009260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:01.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79e0003c80 0x7f79e0009260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 >> v1:172.21.15.98:6789/0 conn(0x7f79e00097a0 legacy=0x7f79e0009b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient: _add_conns ranks=[10,7] 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.190:3300/0 conn(0x7f79e000b2a0 0x7f79e0010da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-f con 0x7f79e000b2a0 addr v2:172.21.15.190:3300/0 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f79e00112e0 0x7f79e00136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient(hunting): picked mon.noname-i con 0x7f79e00112e0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f79e000b970 con 0x7f79e000b2a0 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.913+0000 7f79f1ffb640 1 -- 172.21.15.98:0/1097432954 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f79e000baf0 con 0x7f79e00112e0 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f1ffb640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:01.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f37fe640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f79e00112e0 msgr2=0x7f79e00136d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3303/0 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f37fe640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f79e00112e0 0x7f79e00136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> v2:172.21.15.190:3300/0 conn(0x7f79e000b2a0 0x7f79e0010da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): get_auth_request con 0x7f79e000b2a0 auth_method 0 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient(hunting): handle_auth_done global_id 26200 payload 274 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f79e00112e0 msgr2=0x7f79e00136d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f79e00112e0 0x7f79e00136d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient: _finish_hunting 0 2024-09-11T03:06:01.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 1 monclient: found mon.noname-f 2024-09-11T03:06:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:06:01.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.917+0000 7f79f3fff640 1 -- 172.21.15.98:0/1097432954 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79e0008890 con 0x7f79e000b2a0 2024-09-11T03:06:01.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.933+0000 7f79f3fff640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e000b2a0 0x7f79e0010da0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000ee0 tx=0x7f79e401d4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3343a146ed14ab68 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 1 -- 172.21.15.98:0/1097432954 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79e40b3070 con 0x7f79e000b2a0 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: dump: 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: _finish_auth 0 2024-09-11T03:06:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 1 -- 172.21.15.98:0/1097432954 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e40ac780 con 0x7f79e000b2a0 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 5 monclient: authenticate success, global_id 26200 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 1 -- 172.21.15.98:0/1097432954 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79e400f040 con 0x7f79e000b2a0 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f2ffd640 10 monclient: dump: 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:01.968 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 10 monclient: get_monmap_and_config success 2024-09-11T03:06:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 4 set_mon_vals no callback set 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 10 monclient: shutdown 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 1 -- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e000b2a0 msgr2=0x7f79e0010da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e000b2a0 0x7f79e0010da0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f79e4000ee0 tx=0x7f79e401d4c0 comp rx=0 tx=0).stop 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79f8ae0640 1 -- 172.21.15.98:0/1097432954 reap_dead start 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 1 -- 172.21.15.98:0/1097432954 shutdown_connections 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 1 --2- 172.21.15.98:0/1097432954 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79e000b2a0 0x7f79e0010da0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 1 -- 172.21.15.98:0/1097432954 >> 172.21.15.98:0/1097432954 conn(0x7f79f40a4890 msgr2=0x7f79f40a4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 1 -- 172.21.15.98:0/1097432954 shutdown_connections 2024-09-11T03:06:01.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.965+0000 7f79fa543640 1 -- 172.21.15.98:0/1097432954 wait complete. 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: build_initial_monmap 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: monmap: 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 librados: starting msgr at 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 librados: starting objecter 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 Processor -- start 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 -- start start 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 librados: setting wanted keys 2024-09-11T03:06:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 librados: calling monclient init 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: init 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _add_conns ranks=[8,2,0,3,1,6,5,7,4] 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79e000b2a0 0x7f79f41272d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): picked mon.noname-e con 0x7f79e000b2a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79f4127810 0x7f79f4127bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): picked mon.noname-b con 0x7f79f4127810 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4128130 0x7f79f41d7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): picked mon.noname-a con 0x7f79f4128130 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4138570 con 0x7f79f4128130 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f79f40a4410 con 0x7f79f4127810 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f79f41386f0 con 0x7f79e000b2a0 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient(hunting): authenticate will time out at 1915.849474s 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f8ae0640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4128130 msgr2=0x7f79f41d7900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f8ae0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4128130 0x7f79f41d7900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79e000b2a0 0x7f79f41272d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79f4127810 0x7f79f4127bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79e000b2a0 0x7f79f41272d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44326/0 (socket says 172.21.15.98:44326) 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79f4127810 0x7f79f4127bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49556/0 (socket says 172.21.15.98:49556) 2024-09-11T03:06:01.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 -- 172.21.15.98:0/3136479862 learned_addr learned my addr 172.21.15.98:0/3136479862 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): get_auth_request con 0x7f79e000b2a0 auth_method 0 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 10 monclient(hunting): get_auth_request con 0x7f79f4127810 auth_method 0 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient(hunting): handle_auth_done global_id 25379 payload 824 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 -- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79f4127810 msgr2=0x7f79f4127bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79f4127810 0x7f79f4127bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 -- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4128130 msgr2=0x7f79f41d7900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4128130 0x7f79f41d7900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient: _finish_hunting 0 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 monclient: found mon.noname-e 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:06:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 -- 172.21.15.98:0/3136479862 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f412aa10 con 0x7f79e000b2a0 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f37fe640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79f4127810 0x7f79f4127bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f3fff640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79e000b2a0 0x7f79f41272d0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f79e400ea20 tx=0x7f79e401bd00 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 client.?.objecter ms_handle_connect 0x7f79e000b2a0 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 1 -- 172.21.15.98:0/3136479862 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79e40b3040 con 0x7f79e000b2a0 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: dump: 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: _finish_auth 0 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 5 monclient: authenticate success, global_id 25379 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 1 -- 172.21.15.98:0/3136479862 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e40acda0 con 0x7f79e000b2a0 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _renew_subs 2024-09-11T03:06:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f412acf0 con 0x7f79e000b2a0 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 1 -- 172.21.15.98:0/3136479862 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79e40ae880 con 0x7f79e000b2a0 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f0ff9640 10 monclient: dump: 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79f27fc640 4 set_mon_vals no callback set 2024-09-11T03:06:01.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 client.25379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _renew_subs 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f4072f60 con 0x7f79e000b2a0 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 librados: init done 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.969+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f412a450 con 0x7f79e000b2a0 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 1 -- 172.21.15.98:0/3136479862 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f79e4015040 con 0x7f79e000b2a0 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f79e0040360 0x7f79e0042820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 1 -- 172.21.15.98:0/3136479862 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(637..637 src has 1..637) ==== 6774+0+0 (secure 0 0 0) 0x7f79e40ab070 con 0x7f79e000b2a0 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 10 client.25379.objecter ms_dispatch 0x7f79f4000ba0 osd_map(637..637 src has 1..637) 2024-09-11T03:06:01.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 3 client.25379.objecter handle_osd_map got epochs [637,637] > 0 2024-09-11T03:06:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 3 client.25379.objecter handle_osd_map decoding full epoch 637 2024-09-11T03:06:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 20 client.25379.objecter dump_active .. 0 homeless 2024-09-11T03:06:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f37fe640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f79e0040360 0x7f79e0042820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f37fe640 10 monclient: get_auth_request con 0x7f79e0040360 auth_method 0 2024-09-11T03:06:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f37fe640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f79e0040360 0x7f79e0042820 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7f79f4126fe0 tx=0x7f79e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.973+0000 7f79f0ff9640 10 client.25379.objecter ms_handle_connect 0x7f79e0040360 2024-09-11T03:06:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.977+0000 7f79f0ff9640 1 -- 172.21.15.98:0/3136479862 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f79e4017140 con 0x7f79e000b2a0 2024-09-11T03:06:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.977+0000 7f79f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:01.977+0000 7f79f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:06:02.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:06:02.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:06:02.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:06:02.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f79f4072aa0 con 0x7f79e000b2a0 2024-09-11T03:06:02.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79f0ff9640 1 -- 172.21.15.98:0/3136479862 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3175 (secure 0 0 0) 0x7f79e40aea20 con 0x7f79e000b2a0 2024-09-11T03:06:02.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:06:02.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:06:02.220 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":112,"quorum":[1,2,4,5,6,8],"quorum_names":["f","b","c","h","d","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:06:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 10 librados: watch_flush enter 2024-09-11T03:06:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 10 librados: watch_flush exit 2024-09-11T03:06:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 20 client.25379.objecter shutdown clearing up homeless session... 2024-09-11T03:06:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 10 client.25379.objecter successfully canceled tick 2024-09-11T03:06:02.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f79e0040360 msgr2=0x7f79e0042820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f79e0040360 0x7f79e0042820 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7f79f4126fe0 tx=0x7f79e8008040 comp rx=0 tx=0).stop 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 10 monclient: shutdown 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79e000b2a0 msgr2=0x7f79f41272d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79e000b2a0 0x7f79f41272d0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f79e400ea20 tx=0x7f79e401bd00 comp rx=0 tx=0).stop 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 shutdown_connections 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79f4128130 0x7f79f41d7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f79e0040360 0x7f79e0042820 unknown :-1 s=CLOSED pgs=3685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f79f4127810 0x7f79f4127bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 --2- 172.21.15.98:0/3136479862 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f79e000b2a0 0x7f79f41272d0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 >> 172.21.15.98:0/3136479862 conn(0x7f79f40a4890 msgr2=0x7f79f4135ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 shutdown_connections 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 -- 172.21.15.98:0/3136479862 wait complete. 2024-09-11T03:06:02.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.221+0000 7f79fa543640 1 librados: shutdown 2024-09-11T03:06:02.239 INFO:tasks.mon_thrash.ceph_manager:quorum is size 6 2024-09-11T03:06:02.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon b 2024-09-11T03:06:02.240 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:06:02.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.357+0000 7fa482c4b640 10 monclient: get_monmap_and_config 2024-09-11T03:06:02.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.357+0000 7fa482c4b640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.357+0000 7fa482c4b640 10 monclient: monmap: 2024-09-11T03:06:02.358 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:06:02.360905+0000 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:06:02.360905+0000 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:02.359 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:02.360 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:02.360 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:06:02.360 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:06:02.360 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 Processor -- start 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 -- start start 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient: init 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient: _add_conns ranks=[5,2,9,7,4,3,10,0,8,1,6] 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c139b10 0x7fa47c072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): picked mon.noname-h con 0x7fa47c139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c139ef0 0x7fa47c0a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): picked mon.noname-b con 0x7fa47c139ef0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa47c13a2d0 0x7fa47c0a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa47c13a2d0 addr v2:172.21.15.98:3300/0 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fa47c0702b0 con 0x7fa47c13a2d0 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c071bf0 con 0x7fa47c139ef0 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa47c071350 con 0x7fa47c139b10 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa482c4b640 10 monclient(hunting): authenticate will time out at 1916.241320s 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa4811e8640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7fa47c13a2d0 msgr2=0x7fa47c0a7f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa4811e8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fa47c13a2d0 0x7fa47c0a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c139ef0 0x7fa47c0a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c139ef0 0x7fa47c0a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49564/0 (socket says 172.21.15.98:49564) 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 -- 172.21.15.98:0/2028974142 learned_addr learned my addr 172.21.15.98:0/2028974142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): get_auth_request con 0x7fa47c139ef0 auth_method 0 2024-09-11T03:06:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa4809e7640 1 --2- 172.21.15.98:0/2028974142 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c139b10 0x7fa47c072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa4809e7640 10 monclient(hunting): get_auth_request con 0x7fa47c139b10 auth_method 0 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa4809e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa4809e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa4809e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient(hunting): handle_auth_done global_id 94268 payload 274 2024-09-11T03:06:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 -- 172.21.15.98:0/2028974142 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c139b10 msgr2=0x7fa47c072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 --2- 172.21.15.98:0/2028974142 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c139b10 0x7fa47c072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 -- 172.21.15.98:0/2028974142 >> v2:172.21.15.98:3300/0 conn(0x7fa47c13a2d0 msgr2=0x7fa47c0a7f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 --2- 172.21.15.98:0/2028974142 >> v2:172.21.15.98:3300/0 conn(0x7fa47c13a2d0 0x7fa47c0a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient: _finish_hunting 0 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 monclient: found mon.noname-b 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 -- 172.21.15.98:0/2028974142 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c0a9620 con 0x7fa47c139ef0 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.361+0000 7fa47bfff640 1 --2- 172.21.15.98:0/2028974142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c139ef0 0x7fa47c0a7a10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0098a0 tx=0x7fa46c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7242550978b22eb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 1 -- 172.21.15.98:0/2028974142 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa46c0b0070 con 0x7fa47c139ef0 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:06:02.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: dump: 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.374 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: _finish_auth 0 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 1 -- 172.21.15.98:0/2028974142 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa46c0aa810 con 0x7fa47c139ef0 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 5 monclient: authenticate success, global_id 94268 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient: get_monmap_and_config success 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 1 -- 172.21.15.98:0/2028974142 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa46c0a9300 con 0x7fa47c139ef0 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 4 set_mon_vals no callback set 2024-09-11T03:06:02.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47b7fe640 10 monclient: dump: 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.376 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient: shutdown 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- 172.21.15.98:0/2028974142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c139ef0 msgr2=0x7fa47c0a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 --2- 172.21.15.98:0/2028974142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c139ef0 0x7fa47c0a7a10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0098a0 tx=0x7fa46c01c420 comp rx=0 tx=0).stop 2024-09-11T03:06:02.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- 172.21.15.98:0/2028974142 shutdown_connections 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 --2- 172.21.15.98:0/2028974142 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c139ef0 0x7fa47c0a7a10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 --2- 172.21.15.98:0/2028974142 >> v2:172.21.15.98:3300/0 conn(0x7fa47c13a2d0 0x7fa47c0a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 --2- 172.21.15.98:0/2028974142 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c139b10 0x7fa47c072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- 172.21.15.98:0/2028974142 >> 172.21.15.98:0/2028974142 conn(0x7fa47c0ac790 msgr2=0x7fa47c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- 172.21.15.98:0/2028974142 shutdown_connections 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- 172.21.15.98:0/2028974142 wait complete. 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient: monmap: 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.378 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:02.379 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 librados: starting msgr at 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 librados: starting objecter 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 Processor -- start 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- start start 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 librados: setting wanted keys 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 librados: calling monclient init 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient: init 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient: _add_conns ranks=[3,0,5,6,8,2,7,4,1] 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c139b10 0x7fa47c13a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): picked mon.noname-g con 0x7fa47c139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c139ef0 0x7fa47c13ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): picked mon.noname-a con 0x7fa47c139ef0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c13a2d0 0x7fa47c13b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): picked mon.noname-h con 0x7fa47c13a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c143460 con 0x7fa47c139ef0 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c0a88a0 con 0x7fa47c139b10 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fa47c1435e0 con 0x7fa47c13a2d0 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa482c4b640 10 monclient(hunting): authenticate will time out at 1916.246403s 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47bfff640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c139ef0 msgr2=0x7fa47c13ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa47bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c139ef0 0x7fa47c13ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:02.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4809e7640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c139b10 msgr2=0x7fa47c13a6d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4809e7640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c139b10 0x7fa47c13a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c13a2d0 0x7fa47c13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c13a2d0 0x7fa47c13b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:49792/0 (socket says 172.21.15.98:49792) 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 -- 172.21.15.98:0/4203100674 learned_addr learned my addr 172.21.15.98:0/4203100674 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): get_auth_request con 0x7fa47c13a2d0 auth_method 0 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient(hunting): handle_auth_done global_id 74633 payload 824 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c139b10 msgr2=0x7fa47c13a6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:06:02.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fa47c139b10 0x7fa47c13a6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c139ef0 msgr2=0x7fa47c13ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa47c139ef0 0x7fa47c13ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient: _finish_hunting 0 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 monclient: found mon.noname-h 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 -- 172.21.15.98:0/4203100674 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c13cf40 con 0x7fa47c13a2d0 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4811e8640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c13a2d0 0x7fa47c13b150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa47400be90 tx=0x7fa47400bf80 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4797fa640 10 client.?.objecter ms_handle_connect 0x7fa47c13a2d0 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.365+0000 7fa4797fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa47400cd20 con 0x7fa47c13a2d0 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: dump: 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.384 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: _finish_auth 0 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: _check_auth_tickets 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa474010030 con 0x7fa47c13a2d0 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 5 monclient: authenticate success, global_id 74633 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 10 monclient: _renew_subs 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47c13c6e0 con 0x7fa47c13a2d0 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa474015850 con 0x7fa47c13a2d0 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa47affd640 4 set_mon_vals no callback set 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: dump: 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 10 client.74633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 10 monclient: _renew_subs 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47c0aa120 con 0x7fa47c13a2d0 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 1 librados: init done 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:02.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c133760 0x7fa47c133b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c06ca10 con 0x7fa47c133760 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c133760 0x7fa47c133b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient: get_auth_request con 0x7fa47c133760 auth_method 0 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 10 monclient(hunting): handle_auth_done global_id 94277 payload 824 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 1 -- 172.21.15.98:0/4203100674 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa47c135a80 con 0x7fa47c133760 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4809e7640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c133760 0x7fa47c133b40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa464000ee0 tx=0x7fa46400eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 client.74633.objecter ms_handle_connect 0x7fa47c133760 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 client.74633.objecter resend_mon_ops 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa464017020 con 0x7fa47c133760 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: dump: 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.389 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fa474008070 con 0x7fa47c13a2d0 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa45c03e1e0 0x7fa45c0406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(637..637 src has 1..637) ==== 6774+0+0 (secure 0 0 0) 0x7fa47404fc10 con 0x7fa47c13a2d0 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 client.74633.objecter ms_dispatch 0x7fa47c000ba0 osd_map(637..637 src has 1..637) 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 3 client.74633.objecter handle_osd_map got epochs [637,637] > 0 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 3 client.74633.objecter handle_osd_map decoding full epoch 637 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 20 client.74633.objecter dump_active .. 0 homeless 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa46400ecd0 con 0x7fa47c133760 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c133760 msgr2=0x7fa47c133b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c133760 0x7fa47c133b40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa464000ee0 tx=0x7fa46400eba0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c133760 msgr2=0x7fa47c133b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c133760 0x7fa47c133b40 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa464000ee0 tx=0x7fa46400eba0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa47bfff640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa45c03e1e0 0x7fa45c0406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa47bfff640 10 monclient: get_auth_request con 0x7fa45c03e1e0 auth_method 0 2024-09-11T03:06:02.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa4797fa640 10 client.74633.objecter ms_handle_connect 0x7fa45c03e1e0 2024-09-11T03:06:02.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.369+0000 7fa47bfff640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa45c03e1e0 0x7fa45c0406a0 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0ad000 tx=0x7fa46c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c134080 0x7fa47c0a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c06e870 con 0x7fa47c134080 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c134080 0x7fa47c0a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient: get_auth_request con 0x7fa47c134080 auth_method 0 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 10 monclient(hunting): handle_auth_done global_id 94286 payload 824 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 1 -- 172.21.15.98:0/4203100674 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa47c06e680 con 0x7fa47c134080 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 10 client.74633.objecter ms_handle_connect 0x7fa47c134080 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4809e7640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c134080 0x7fa47c0a0e60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa464000ee0 tx=0x7fa464004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 10 client.74633.objecter resend_mon_ops 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa46400f040 con 0x7fa47c134080 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 10 monclient: dump: 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.395 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9568 (secure 0 0 0) 0x7fa46400cb40 con 0x7fa47c134080 2024-09-11T03:06:02.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c134080 msgr2=0x7fa47c0a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c134080 0x7fa47c0a0e60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa464000ee0 tx=0x7fa464004740 comp rx=0 tx=0).stop 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c134080 msgr2=0x7fa47c0a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4797fa640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fa47c134080 0x7fa47c0a0e60 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa464000ee0 tx=0x7fa464004740 comp rx=0 tx=0).stop 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:06:02.397 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 112, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 60763, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:06:02.398 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:06:02.399 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:06:02.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.402 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.403 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.403 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:06:02.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.403 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:06:02.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.403 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:06:02.403 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.404 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:06:02.406 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:06:02.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:06:02.411 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:06:02.412 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.413 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:06:02.414 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.415 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.416 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:06:02.417 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.418 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:06:02.419 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:06:02.420 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:06:02.421 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 10 librados: watch_flush enter 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 10 librados: watch_flush exit 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 20 client.74633.objecter shutdown clearing up homeless session... 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 10 client.74633.objecter successfully canceled tick 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa45c03e1e0 msgr2=0x7fa45c0406a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fa45c03e1e0 0x7fa45c0406a0 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0ad000 tx=0x7fa46c008040 comp rx=0 tx=0).stop 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 10 monclient: shutdown 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c13a2d0 msgr2=0x7fa47c13b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 --2- 172.21.15.98:0/4203100674 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fa47c13a2d0 0x7fa47c13b150 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa47400be90 tx=0x7fa47400bf80 comp rx=0 tx=0).stop 2024-09-11T03:06:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4811e8640 1 -- 172.21.15.98:0/4203100674 reap_dead start 2024-09-11T03:06:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa4811e8640 1 -- 172.21.15.98:0/4203100674 reap_dead start 2024-09-11T03:06:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 shutdown_connections 2024-09-11T03:06:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 >> 172.21.15.98:0/4203100674 conn(0x7fa47c0ac790 msgr2=0x7fa47c0aa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 shutdown_connections 2024-09-11T03:06:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 -- 172.21.15.98:0/4203100674 wait complete. 2024-09-11T03:06:02.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.377+0000 7fa482c4b640 1 librados: shutdown 2024-09-11T03:06:02.424 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-11T03:06:02.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon c 2024-09-11T03:06:02.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient: get_monmap_and_config 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient: monmap: 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:06:02.545463+0000 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:06:02.545463+0000 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.543 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:06:02.544 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 Processor -- start 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 -- start start 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient: init 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient: _add_conns ranks=[9,10,7,2,1,8,5,0,4,3,6] 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f207c0a39f0 0x7f207c0a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): picked mon.noname-a con 0x7f207c0a39f0 addr v2:172.21.15.98:3300/0 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f207c0a3dd0 0x7f207c0a41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): picked mon.noname-f con 0x7f207c0a3dd0 addr v2:172.21.15.190:3300/0 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f207c1377c0 0x7f207c137c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): picked mon.noname-i con 0x7f207c1377c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f207c09db20 con 0x7f207c0a39f0 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f207c0a1be0 con 0x7f207c0a3dd0 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f207c09ea20 con 0x7f207c1377c0 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2083ea7640 10 monclient(hunting): authenticate will time out at 1916.425701s 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081c43640 1 -- >> v2:172.21.15.98:3300/0 conn(0x7f207c0a39f0 msgr2=0x7f207c0a1e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081c43640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f207c0a39f0 0x7f207c0a1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:02.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2082444640 1 -- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f207c1377c0 msgr2=0x7f207c137c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3303/0 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2082444640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f207c1377c0 0x7f207c137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f207c0a3dd0 0x7f207c0a41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f207c0a3dd0 0x7f207c0a41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:48322/0 (socket says 172.21.15.98:48322) 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 -- 172.21.15.98:0/823025618 learned_addr learned my addr 172.21.15.98:0/823025618 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): get_auth_request con 0x7f207c0a3dd0 auth_method 0 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient(hunting): handle_auth_done global_id 26209 payload 274 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 -- 172.21.15.98:0/823025618 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f207c1377c0 msgr2=0x7f207c137c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 --2- 172.21.15.98:0/823025618 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f207c1377c0 0x7f207c137c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 -- 172.21.15.98:0/823025618 >> v2:172.21.15.98:3300/0 conn(0x7f207c0a39f0 msgr2=0x7f207c0a1e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 --2- 172.21.15.98:0/823025618 >> v2:172.21.15.98:3300/0 conn(0x7f207c0a39f0 0x7f207c0a1e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient: _finish_hunting 0 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 monclient: found mon.noname-f 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:06:02.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.545+0000 7f2081442640 1 -- 172.21.15.98:0/823025618 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c09d280 con 0x7f207c0a3dd0 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2081442640 1 --2- 172.21.15.98:0/823025618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f207c0a3dd0 0x7f207c0a41d0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f207c074b80 tx=0x7f207801c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cca762f6d00d76a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 1 -- 172.21.15.98:0/823025618 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f20780b0070 con 0x7f207c0a3dd0 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: dump: 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.557 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: _finish_auth 0 2024-09-11T03:06:02.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: _check_auth_tickets 2024-09-11T03:06:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 1 -- 172.21.15.98:0/823025618 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20780aa810 con 0x7f207c0a3dd0 2024-09-11T03:06:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 1 -- 172.21.15.98:0/823025618 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f20780a9300 con 0x7f207c0a3dd0 2024-09-11T03:06:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 5 monclient: authenticate success, global_id 26209 2024-09-11T03:06:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 10 monclient: get_monmap_and_config success 2024-09-11T03:06:02.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 4 set_mon_vals no callback set 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2080c41640 10 monclient: dump: 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.560 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 10 monclient: shutdown 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 -- 172.21.15.98:0/823025618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f207c0a3dd0 msgr2=0x7f207c0a41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 --2- 172.21.15.98:0/823025618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f207c0a3dd0 0x7f207c0a41d0 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f207c074b80 tx=0x7f207801c420 comp rx=0 tx=0).stop 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 -- 172.21.15.98:0/823025618 shutdown_connections 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 --2- 172.21.15.98:0/823025618 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f207c1377c0 0x7f207c137c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 --2- 172.21.15.98:0/823025618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f207c0a3dd0 0x7f207c0a41d0 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 --2- 172.21.15.98:0/823025618 >> v2:172.21.15.98:3300/0 conn(0x7f207c0a39f0 0x7f207c0a1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 -- 172.21.15.98:0/823025618 >> 172.21.15.98:0/823025618 conn(0x7f207c0a8670 msgr2=0x7f207c0a4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 -- 172.21.15.98:0/823025618 shutdown_connections 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.549+0000 7f2083ea7640 1 -- 172.21.15.98:0/823025618 wait complete. 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: monmap: 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:06:02.562 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 librados: starting msgr at 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 librados: starting objecter 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 Processor -- start 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 -- start start 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 librados: setting wanted keys 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 librados: calling monclient init 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: init 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:02.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: _add_conns ranks=[2,4,8,6,1,7,5,0,3] 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f207c0a39f0 0x7f207c1d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): picked mon.noname-b con 0x7f207c0a39f0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c0a3dd0 0x7f207c136c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): picked mon.noname-c con 0x7f207c0a3dd0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f207c1377c0 0x7f207c09f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): picked mon.noname-e con 0x7f207c1377c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f207c0a5630 con 0x7f207c0a39f0 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f207c0a54b0 con 0x7f207c0a3dd0 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f207c141230 con 0x7f207c1377c0 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): authenticate will time out at 1916.433483s 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f207c0a39f0 0x7f207c1d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f207c0a39f0 0x7f207c1d15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:49598/0 (socket says 172.21.15.98:49598) 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f207c1377c0 0x7f207c09f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 1 -- 172.21.15.98:0/1091031251 learned_addr learned my addr 172.21.15.98:0/1091031251 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): get_auth_request con 0x7f207c0a39f0 auth_method 0 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): get_auth_request con 0x7f207c1377c0 auth_method 0 2024-09-11T03:06:02.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c0a3dd0 0x7f207c136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 10 monclient(hunting): get_auth_request con 0x7f207c0a3dd0 auth_method 0 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081442640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient(hunting): handle_auth_done global_id 25388 payload 824 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c0a3dd0 msgr2=0x7f207c136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c0a3dd0 0x7f207c136c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f207c0a39f0 msgr2=0x7f207c1d15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f207c0a39f0 0x7f207c1d15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient: _finish_hunting 0 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 monclient: found mon.noname-e 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:06:02.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 -- 172.21.15.98:0/1091031251 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c124050 con 0x7f207c1377c0 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f207c0a39f0 0x7f207c1d15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2082444640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f207c1377c0 0x7f207c09f4e0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f207400bdc0 tx=0x7f2074009eb0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 client.?.objecter ms_handle_connect 0x7f207c1377c0 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2074006750 con 0x7f207c1377c0 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: dump: 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.569 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: _finish_auth 0 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: _check_auth_tickets 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:02.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2074006950 con 0x7f207c1377c0 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 5 monclient: authenticate success, global_id 25388 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: _renew_subs 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207c124330 con 0x7f207c1377c0 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2074006b00 con 0x7f207c1377c0 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:06:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 monclient: dump: 2024-09-11T03:06:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.572 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.573 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2073fff640 4 set_mon_vals no callback set 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 client.25388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: _renew_subs 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f207c1234f0 con 0x7f207c1377c0 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 librados: init done 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c137190 0x7f207c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f207c13dd50 con 0x7f207c137190 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c137190 0x7f207c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient: get_auth_request con 0x7f207c137190 auth_method 0 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f207401d030 con 0x7f207c1377c0 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 10 monclient(hunting): handle_auth_done global_id 24835 payload 824 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 1 -- 172.21.15.98:0/1091031251 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f207c0a41d0 con 0x7f207c137190 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f206003df40 0x7f2060040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(637..637 src has 1..637) ==== 6774+0+0 (secure 0 0 0) 0x7f20740504c0 con 0x7f207c1377c0 2024-09-11T03:06:02.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 10 client.25388.objecter ms_dispatch 0x7f207c000ba0 osd_map(637..637 src has 1..637) 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f2081c43640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c137190 0x7f207c02d450 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f206c005340 tx=0x7f206c005800 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 3 client.25388.objecter handle_osd_map got epochs [637,637] > 0 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.553+0000 7f20727fc640 3 client.25388.objecter handle_osd_map decoding full epoch 637 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 20 client.25388.objecter dump_active .. 0 homeless 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 client.25388.objecter ms_handle_connect 0x7f207c137190 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 client.25388.objecter resend_mon_ops 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f206c016070 con 0x7f207c137190 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 monclient: dump: 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.577 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f2081442640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f206003df40 0x7f2060040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f2081442640 10 monclient: get_auth_request con 0x7f206003df40 auth_method 0 2024-09-11T03:06:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f206c00eaf0 con 0x7f207c137190 2024-09-11T03:06:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:06:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c137190 msgr2=0x7f207c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c137190 0x7f207c02d450 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f206c005340 tx=0x7f206c005800 comp rx=0 tx=0).stop 2024-09-11T03:06:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c137190 msgr2=0x7f207c02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c137190 0x7f207c02d450 secure :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f206c005340 tx=0x7f206c005800 comp rx=0 tx=0).stop 2024-09-11T03:06:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f20727fc640 10 client.25388.objecter ms_handle_connect 0x7f206003df40 2024-09-11T03:06:02.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.557+0000 7f2081442640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f206003df40 0x7f2060040400 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f20780ad000 tx=0x7f20780a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:06:02.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c02d990 0x7f207c02dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f207c06ca10 con 0x7f207c02d990 2024-09-11T03:06:02.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c02d990 0x7f207c02dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient: get_auth_request con 0x7f207c02d990 auth_method 0 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 10 monclient(hunting): handle_auth_done global_id 24844 payload 824 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 1 -- 172.21.15.98:0/1091031251 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f207c122e10 con 0x7f207c02d990 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 10 client.25388.objecter ms_handle_connect 0x7f207c02d990 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 10 client.25388.objecter resend_mon_ops 2024-09-11T03:06:02.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2081c43640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c02d990 0x7f207c02dda0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f206c005340 tx=0x7f206c0059e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f206c016040 con 0x7f207c02d990 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 10 monclient: dump: 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.583 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9235 (secure 0 0 0) 0x7f206c0123f0 con 0x7f207c02d990 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c02d990 msgr2=0x7f207c02dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c02d990 0x7f207c02dda0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f206c005340 tx=0x7f206c0059e0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c02d990 msgr2=0x7f207c02dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:02.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f20727fc640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f207c02d990 0x7f207c02dda0 secure :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f206c005340 tx=0x7f206c0059e0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 112, 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 978345, 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:06:02.585 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:06:02.586 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:06:02.586 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:06:02.586 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:06:02.586 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:06:02.586 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.586 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:06:02.586 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:02.587 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:06:02.587 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:06:02.587 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.587 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.587 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.588 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.588 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.588 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.588 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.588 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.588 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.589 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.589 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.589 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.589 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:06:02.589 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:06:02.589 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.590 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.590 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.590 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.590 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.590 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.590 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.591 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.591 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.591 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.591 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.591 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.592 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:06:02.592 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:06:02.592 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:06:02.592 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:06:02.592 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:06:02.592 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:06:02.593 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:02.593 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:02.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:06:02.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:06:02.593 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:06:02.594 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:06:02.594 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:06:02.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:06:02.594 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:06:02.594 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:02.594 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:06:02.595 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.595 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.595 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.595 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.595 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.595 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.596 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.596 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.596 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.596 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.596 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.597 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.597 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:06:02.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.597 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:06:02.597 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.598 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:06:02.598 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:06:02.598 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.598 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.598 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.598 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.599 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:06:02.599 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.599 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.599 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:06:02.600 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.600 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.600 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.600 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:06:02.600 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:06:02.601 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.601 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.601 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.601 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.601 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:06:02.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:06:02.602 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.602 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.602 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:06:02.603 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.603 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.603 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:06:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:06:02.604 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:06:02.605 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.605 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.605 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.605 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.605 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:06:02.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:06:02.606 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.606 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:06:02.607 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.607 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:06:02.608 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.608 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.608 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.608 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:06:02.608 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:06:02.609 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.609 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.609 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.609 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.609 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:06:02.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:06:02.610 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.610 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.610 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:06:02.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.611 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.611 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:06:02.611 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.612 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.612 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.612 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:06:02.612 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:06:02.612 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.613 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.613 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.613 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.613 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:06:02.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:06:02.614 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.614 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.614 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:06:02.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.615 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.615 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:06:02.615 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.615 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.616 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.616 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:06:02.616 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:06:02.616 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.616 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.617 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.618 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:06:02.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.624 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:06:02.625 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 10 librados: watch_flush enter 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 10 librados: watch_flush exit 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 20 client.25388.objecter shutdown clearing up homeless session... 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 10 client.25388.objecter successfully canceled tick 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f206003df40 msgr2=0x7f2060040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f206003df40 0x7f2060040400 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f20780ad000 tx=0x7f20780a6000 comp rx=0 tx=0).stop 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 10 monclient: shutdown 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f207c1377c0 msgr2=0x7f207c09f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 --2- 172.21.15.98:0/1091031251 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f207c1377c0 0x7f207c09f4e0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f207400bdc0 tx=0x7f2074009eb0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2082444640 1 -- 172.21.15.98:0/1091031251 reap_dead start 2024-09-11T03:06:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2082444640 1 -- 172.21.15.98:0/1091031251 reap_dead start 2024-09-11T03:06:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 shutdown_connections 2024-09-11T03:06:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 >> 172.21.15.98:0/1091031251 conn(0x7f207c0a8670 msgr2=0x7f207c071fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.561+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 shutdown_connections 2024-09-11T03:06:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.565+0000 7f2083ea7640 1 -- 172.21.15.98:0/1091031251 wait complete. 2024-09-11T03:06:02.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.565+0000 7f2083ea7640 1 librados: shutdown 2024-09-11T03:06:02.629 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-11T03:06:02.630 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon d 2024-09-11T03:06:02.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient: get_monmap_and_config 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient: monmap: 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:06:02.753434+0000 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:06:02.753434+0000 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.751 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:06:02.752 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 Processor -- start 2024-09-11T03:06:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 -- start start 2024-09-11T03:06:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient: init 2024-09-11T03:06:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient: _add_conns ranks=[5,3,7,6,9,0,4,1,8,2,10] 2024-09-11T03:06:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb174139b10 0x7fb174072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): picked mon.noname-h con 0x7fb174139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb174139ef0 0x7fb1740a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): picked mon.noname-g con 0x7fb174139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb17413a2d0 0x7fb1740a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): picked mon.noname-i con 0x7fb17413a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fb1740702b0 con 0x7fb174139ef0 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb174071bf0 con 0x7fb174139b10 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fb174071350 con 0x7fb17413a2d0 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb17aec5640 10 monclient(hunting): authenticate will time out at 1916.633749s 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb173fff640 1 -- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb174139ef0 msgr2=0x7fb1740a7a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb179462640 1 -- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb17413a2d0 msgr2=0x7fb1740a7f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3303/0 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb173fff640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb174139ef0 0x7fb1740a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb179462640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb17413a2d0 0x7fb1740a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb174139b10 0x7fb174072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb174139b10 0x7fb174072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:49806/0 (socket says 172.21.15.98:49806) 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 -- 172.21.15.98:0/3211866957 learned_addr learned my addr 172.21.15.98:0/3211866957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): get_auth_request con 0x7fb174139b10 auth_method 0 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient(hunting): handle_auth_done global_id 74642 payload 274 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 -- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb17413a2d0 msgr2=0x7fb1740a7f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 --2- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb17413a2d0 0x7fb1740a7f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 -- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb174139ef0 msgr2=0x7fb1740a7a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 --2- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb174139ef0 0x7fb1740a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient: _finish_hunting 0 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 monclient: found mon.noname-h 2024-09-11T03:06:02.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:02.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T03:06:02.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.753+0000 7fb178c61640 1 -- 172.21.15.98:0/3211866957 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1740a9620 con 0x7fb174139b10 2024-09-11T03:06:02.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb178c61640 1 --2- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb174139b10 0x7fb174072600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1640098a0 tx=0x7fb16401c480 comp rx=0 tx=0).ready entity=mon.5 client_cookie=3418b51f33d43258 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 1 -- 172.21.15.98:0/3211866957 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1640b0070 con 0x7fb174139b10 2024-09-11T03:06:02.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: dump: 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.773 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: _finish_auth 0 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: _check_auth_tickets 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 1 -- 172.21.15.98:0/3211866957 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1640aa810 con 0x7fb174139b10 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 1 -- 172.21.15.98:0/3211866957 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1640a9300 con 0x7fb174139b10 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb17aec5640 5 monclient: authenticate success, global_id 74642 2024-09-11T03:06:02.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb17aec5640 10 monclient: get_monmap_and_config success 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb17aec5640 4 set_mon_vals no callback set 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.757+0000 7fb1737fe640 10 monclient: dump: 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.775 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: shutdown 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb174139b10 msgr2=0x7fb174072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb174139b10 0x7fb174072600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb1640098a0 tx=0x7fb16401c480 comp rx=0 tx=0).stop 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3211866957 shutdown_connections 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fb17413a2d0 0x7fb1740a7f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fb174139ef0 0x7fb1740a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3211866957 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb174139b10 0x7fb174072600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3211866957 >> 172.21.15.98:0/3211866957 conn(0x7fb1740ac790 msgr2=0x7fb1740a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3211866957 shutdown_connections 2024-09-11T03:06:02.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3211866957 wait complete. 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: monmap: 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:02.778 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 librados: starting msgr at 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 librados: starting objecter 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 Processor -- start 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- start start 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 librados: setting wanted keys 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 librados: calling monclient init 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: init 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:02.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: _add_conns ranks=[1,4,5,0,3,2,8,6,7] 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb174139b10 0x7fb17413f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): picked mon.noname-f con 0x7fb174139b10 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb174139ef0 0x7fb1740a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): picked mon.noname-c con 0x7fb174139ef0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb17413a2d0 0x7fb1740a41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): picked mon.noname-h con 0x7fb17413a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb174143460 con 0x7fb174139b10 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fb1740a88a0 con 0x7fb174139ef0 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fb1741435e0 con 0x7fb17413a2d0 2024-09-11T03:06:02.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:02.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): authenticate will time out at 1916.641918s 2024-09-11T03:06:02.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb174139ef0 0x7fb1740a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb174139ef0 0x7fb1740a3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:59644/0 (socket says 172.21.15.98:59644) 2024-09-11T03:06:02.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 -- 172.21.15.98:0/3770725518 learned_addr learned my addr 172.21.15.98:0/3770725518 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:02.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): get_auth_request con 0x7fb174139ef0 auth_method 0 2024-09-11T03:06:02.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb174139b10 0x7fb17413f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb17413a2d0 0x7fb1740a41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): get_auth_request con 0x7fb174139b10 auth_method 0 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 10 monclient(hunting): get_auth_request con 0x7fb17413a2d0 auth_method 0 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb179462640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient(hunting): handle_auth_done global_id 24853 payload 824 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb17413a2d0 msgr2=0x7fb1740a41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fb17413a2d0 0x7fb1740a41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb174139b10 msgr2=0x7fb17413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb174139b10 0x7fb17413f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient: _finish_hunting 0 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 monclient: found mon.noname-c 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 -- 172.21.15.98:0/3770725518 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17413bdf0 con 0x7fb174139ef0 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb173fff640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb174139ef0 0x7fb1740a3c70 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fb16800b530 tx=0x7fb16800b9f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 client.?.objecter ms_handle_connect 0x7fb174139ef0 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb168013020 con 0x7fb174139ef0 2024-09-11T03:06:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: dump: 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.785 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: _finish_auth 0 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: _check_auth_tickets 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb168003bd0 con 0x7fb174139ef0 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 5 monclient: authenticate success, global_id 24853 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: _renew_subs 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:06:02.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17413c0d0 con 0x7fb174139ef0 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb168003d80 con 0x7fb174139ef0 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 client.24853.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1727fc640 4 set_mon_vals no callback set 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 10 monclient: dump: 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.788 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: _renew_subs 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1740aa100 con 0x7fb174139ef0 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 librados: init done 2024-09-11T03:06:02.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb17402e5d0 0x7fb1740a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb17406ca10 con 0x7fb17402e5d0 2024-09-11T03:06:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb17402e5d0 0x7fb1740a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient: get_auth_request con 0x7fb17402e5d0 auth_method 0 2024-09-11T03:06:02.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fb168005070 con 0x7fb174139ef0 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.761+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb178c61640 10 monclient(hunting): handle_auth_done global_id 84570 payload 824 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb16003df40 0x7fb160040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb178c61640 1 -- 172.21.15.98:0/3770725518 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1741359c0 con 0x7fb17402e5d0 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(637..637 src has 1..637) ==== 6774+0+0 (secure 0 0 0) 0x7fb16804f790 con 0x7fb174139ef0 2024-09-11T03:06:02.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 client.24853.objecter ms_dispatch 0x7fb174000ba0 osd_map(637..637 src has 1..637) 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 3 client.24853.objecter handle_osd_map got epochs [637,637] > 0 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 3 client.24853.objecter handle_osd_map decoding full epoch 637 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 20 client.24853.objecter dump_active .. 0 homeless 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 client.24853.objecter ms_handle_connect 0x7fb17402e5d0 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 client.24853.objecter resend_mon_ops 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb178c61640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb17402e5d0 0x7fb1740a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb164009870 tx=0x7fb164019220 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1640ba030 con 0x7fb17402e5d0 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb179462640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb16003df40 0x7fb160040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 monclient: dump: 2024-09-11T03:06:02.792 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.793 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.793 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.793 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.793 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.793 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.793 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.793 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.794 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb179462640 10 monclient: get_auth_request con 0x7fb16003df40 auth_method 0 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb164003910 con 0x7fb17402e5d0 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb17402e5d0 msgr2=0x7fb1740a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb17402e5d0 0x7fb1740a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb164009870 tx=0x7fb164019220 comp rx=0 tx=0).stop 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb17402e5d0 msgr2=0x7fb1740a46f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb17402e5d0 0x7fb1740a46f0 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb164009870 tx=0x7fb164019220 comp rx=0 tx=0).stop 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb1717fa640 10 client.24853.objecter ms_handle_connect 0x7fb16003df40 2024-09-11T03:06:02.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.765+0000 7fb179462640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb16003df40 0x7fb160040400 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7fb17413ba10 tx=0x7fb16c00d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb17aec5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb174063e00 0x7fb174064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb17aec5640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fb17406e870 con 0x7fb174063e00 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb174063e00 0x7fb174064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient: get_auth_request con 0x7fb174063e00 auth_method 0 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 10 monclient(hunting): handle_auth_done global_id 84579 payload 824 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 1 -- 172.21.15.98:0/3770725518 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb17402e9b0 con 0x7fb174063e00 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb178c61640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb174063e00 0x7fb174064210 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb164000c10 tx=0x7fb1640a8000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb1717fa640 10 client.24853.objecter ms_handle_connect 0x7fb174063e00 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb1717fa640 10 client.24853.objecter resend_mon_ops 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1640ba040 con 0x7fb174063e00 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb1717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb1717fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.769+0000 7fb1717fa640 10 monclient: dump: 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9235 (secure 0 0 0) 0x7fb1640c1e20 con 0x7fb174063e00 2024-09-11T03:06:02.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb1717fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb1717fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb174063e00 msgr2=0x7fb174064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb1717fa640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb174063e00 0x7fb174064210 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb164000c10 tx=0x7fb1640a8000 comp rx=0 tx=0).stop 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb1717fa640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb174063e00 msgr2=0x7fb174064210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb1717fa640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fb174063e00 0x7fb174064210 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb164000c10 tx=0x7fb1640a8000 comp rx=0 tx=0).stop 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:06:02.799 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 112, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 188581, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 0, 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:02.800 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.801 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.802 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:06:02.803 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:02.805 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:02.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:02.805 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:02.805 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:02.805 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:02.805 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:02.806 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:02.806 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:02.806 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.806 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:06:02.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.806 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:06:02.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.807 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:06:02.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:06:02.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:06:02.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:06:02.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.809 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.810 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:06:02.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:06:02.810 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:06:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:06:02.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:06:02.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.814 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.816 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:06:02.818 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.821 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.822 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:06:02.823 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.824 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:06:02.825 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:02.826 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.827 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:02.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:02.829 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:06:02.829 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:06:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 10 librados: watch_flush enter 2024-09-11T03:06:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 10 librados: watch_flush exit 2024-09-11T03:06:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 20 client.24853.objecter shutdown clearing up homeless session... 2024-09-11T03:06:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 10 client.24853.objecter successfully canceled tick 2024-09-11T03:06:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb16003df40 msgr2=0x7fb160040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fb16003df40 0x7fb160040400 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7fb17413ba10 tx=0x7fb16c00d040 comp rx=0 tx=0).stop 2024-09-11T03:06:02.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 10 monclient: shutdown 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb174139ef0 msgr2=0x7fb1740a3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 --2- 172.21.15.98:0/3770725518 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fb174139ef0 0x7fb1740a3c70 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fb16800b530 tx=0x7fb16800b9f0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb179462640 1 -- 172.21.15.98:0/3770725518 reap_dead start 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb179462640 1 -- 172.21.15.98:0/3770725518 reap_dead start 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 shutdown_connections 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 >> 172.21.15.98:0/3770725518 conn(0x7fb1740ac790 msgr2=0x7fb1740aa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 shutdown_connections 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 -- 172.21.15.98:0/3770725518 wait complete. 2024-09-11T03:06:02.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.773+0000 7fb17aec5640 1 librados: shutdown 2024-09-11T03:06:02.833 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-11T03:06:02.833 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon e 2024-09-11T03:06:02.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient: get_monmap_and_config 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient: monmap: 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:06:02.985681+0000 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:06:02.985681+0000 2024-09-11T03:06:02.983 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:06:02.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 1 Processor -- start 2024-09-11T03:06:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 1 -- start start 2024-09-11T03:06:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient: init 2024-09-11T03:06:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient: _add_conns ranks=[1,6,2,5,10,3,4,0,9,8,7] 2024-09-11T03:06:02.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc0880a59c0 addr v1:172.21.15.190:6789/0 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc08809f0e0 0x7fc088142c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient(hunting): picked mon.noname-d con 0x7fc08809f0e0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08813de00 0x7fc0881451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc08813de00 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc08809c0b0 con 0x7fc0880a59c0 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc0880a5da0 con 0x7fc08813de00 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc08809baf0 con 0x7fc08809f0e0 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:02.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc08e1b3640 10 monclient(hunting): authenticate will time out at 1916.866586s 2024-09-11T03:06:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc0877fe640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fc0880a59c0 0x7fc08809e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:41210/0 (socket says 172.21.15.98:41210) 2024-09-11T03:06:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc0877fe640 1 -- 172.21.15.98:0/1807818006 learned_addr learned my addr 172.21.15.98:0/1807818006 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:06:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 1 --2- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc08809f0e0 0x7fc088142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): get_auth_request con 0x7fc08809f0e0 auth_method 0 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient(hunting): handle_auth_done global_id 84588 payload 274 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 1 -- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08813de00 msgr2=0x7fc0881451e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 1 --2- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08813de00 0x7fc0881451e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 1 -- 172.21.15.98:0/1807818006 >> v1:172.21.15.190:6789/0 conn(0x7fc0880a59c0 legacy=0x7fc08809e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 1 monclient: found mon.noname-d 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:06:02.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.985+0000 7fc086ffd640 1 -- 172.21.15.98:0/1807818006 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08809bdd0 con 0x7fc08809f0e0 2024-09-11T03:06:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc086ffd640 1 --2- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc08809f0e0 0x7fc088142c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc08000b0b0 tx=0x7fc08001cb60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a86e335914241caf server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:02.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 1 -- 172.21.15.98:0/1807818006 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0800b2070 con 0x7fc08809f0e0 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 10 monclient: dump: 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.991 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 10 monclient: _finish_auth 0 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 10 monclient: _check_auth_tickets 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 1 -- 172.21.15.98:0/1807818006 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0800a5d70 con 0x7fc08809f0e0 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc08e1b3640 5 monclient: authenticate success, global_id 84588 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc08e1b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:06:02.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc08e1b3640 10 monclient: get_monmap_and_config success 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc08e1b3640 4 set_mon_vals no callback set 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 1 -- 172.21.15.98:0/1807818006 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0800ab7b0 con 0x7fc08809f0e0 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.989+0000 7fc0867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc0867fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc0867fc640 10 monclient: dump: 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:02.993 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:02.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 10 monclient: shutdown 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 -- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc08809f0e0 msgr2=0x7fc088142c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc08809f0e0 0x7fc088142c90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc08000b0b0 tx=0x7fc08001cb60 comp rx=0 tx=0).stop 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 -- 172.21.15.98:0/1807818006 shutdown_connections 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc08813de00 0x7fc0881451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/1807818006 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc08809f0e0 0x7fc088142c90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 -- 172.21.15.98:0/1807818006 >> 172.21.15.98:0/1807818006 conn(0x7fc0880a4ed0 msgr2=0x7fc0880a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 -- 172.21.15.98:0/1807818006 shutdown_connections 2024-09-11T03:06:02.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 -- 172.21.15.98:0/1807818006 wait complete. 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 10 monclient: build_initial_monmap 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 10 monclient: monmap: 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:06:02.999 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 librados: starting msgr at 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 librados: starting objecter 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.993+0000 7fc08e1b3640 1 Processor -- start 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 -- start start 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 librados: setting wanted keys 2024-09-11T03:06:03.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 librados: calling monclient init 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient: init 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient: _add_conns ranks=[3,0,5,1,8,7,2,6,4] 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc08809f0e0 0x7fc0880916c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): picked mon.noname-g con 0x7fc08809f0e0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0880a59c0 0x7fc08808dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc0880a59c0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08813de00 0x7fc08808e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): picked mon.noname-h con 0x7fc08813de00 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc0881482d0 con 0x7fc0880a59c0 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fc088148150 con 0x7fc08809f0e0 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fc088148450 con 0x7fc08813de00 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient(hunting): authenticate will time out at 1916.877087s 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc086ffd640 1 -- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0880a59c0 msgr2=0x7fc08808dfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:3300/0 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc086ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0880a59c0 0x7fc08808dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08813de00 0x7fc08808e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08813de00 0x7fc08808e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:49834/0 (socket says 172.21.15.98:49834) 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 -- 172.21.15.98:0/3444548608 learned_addr learned my addr 172.21.15.98:0/3444548608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): get_auth_request con 0x7fc08813de00 auth_method 0 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient(hunting): handle_auth_done global_id 74660 payload 824 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc08809f0e0 msgr2=0x7fc0880916c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fc08809f0e0 0x7fc0880916c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0880a59c0 msgr2=0x7fc08808dfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc0880a59c0 0x7fc08808dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient: _finish_hunting 0 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 monclient: found mon.noname-h 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.190:3302/0 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 -- 172.21.15.98:0/3444548608 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc088145130 con 0x7fc08813de00 2024-09-11T03:06:03.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc087fff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08813de00 0x7fc08808e520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00b410 tx=0x7fc07c00b8d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 client.?.objecter ms_handle_connect 0x7fc08813de00 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc07c015020 con 0x7fc08813de00 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: dump: 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.004 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: _finish_auth 0 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: _check_auth_tickets 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:03.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.5 v2:172.21.15.190:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc07c011aa0 con 0x7fc08813de00 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 5 monclient: authenticate success, global_id 74660 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.5 v2:172.21.15.190:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc07c011c90 con 0x7fc08813de00 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient: _renew_subs 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc088074260 con 0x7fc08813de00 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc06ffff640 10 monclient: dump: 2024-09-11T03:06:03.006 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.007 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 client.74660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient: _renew_subs 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.190:3302/0 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc088090790 con 0x7fc08813de00 2024-09-11T03:06:03.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc08e1b3640 1 librados: init done 2024-09-11T03:06:03.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:02.997+0000 7fc085ffb640 4 set_mon_vals no callback set 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.5 v2:172.21.15.190:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc07c011630 con 0x7fc08813de00 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07403df40 0x7fc074040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.5 v2:172.21.15.190:3302/0 5 ==== osd_map(638..638 src has 1..638) ==== 6769+0+0 (secure 0 0 0) 0x7fc07c04fa20 con 0x7fc08813de00 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 10 client.74660.objecter ms_dispatch 0x7fc088000ba0 osd_map(638..638 src has 1..638) 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 3 client.74660.objecter handle_osd_map got epochs [638,638] > 0 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 3 client.74660.objecter handle_osd_map decoding full epoch 638 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 20 client.74660.objecter dump_active .. 0 homeless 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc0877fe640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07403df40 0x7fc074040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc0877fe640 10 monclient: get_auth_request con 0x7fc07403df40 auth_method 0 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc06ffff640 10 client.74660.objecter ms_handle_connect 0x7fc07403df40 2024-09-11T03:06:03.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.001+0000 7fc0877fe640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07403df40 0x7fc074040400 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7fc078001570 tx=0x7fc078007a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc08e1b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068002890 0x7fc068004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc08e1b3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc068007cc0 con 0x7fc068002890 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068002890 0x7fc068004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient: get_auth_request con 0x7fc068002890 auth_method 0 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 10 monclient(hunting): handle_auth_done global_id 25397 payload 824 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 1 -- 172.21.15.98:0/3444548608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc068007ec0 con 0x7fc068002890 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 10 client.74660.objecter ms_handle_connect 0x7fc068002890 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 10 client.74660.objecter resend_mon_ops 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc086ffd640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068002890 0x7fc068004d30 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fc0880a2a90 tx=0x7fc080019900 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0800bc030 con 0x7fc068002890 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 10 monclient: dump: 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.015 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc0800039a0 con 0x7fc068002890 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:06:03.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068002890 msgr2=0x7fc068004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068002890 0x7fc068004d30 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fc0880a2a90 tx=0x7fc080019900 comp rx=0 tx=0).stop 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068002890 msgr2=0x7fc068004d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.005+0000 7fc06ffff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068002890 0x7fc068004d30 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068006ce0 0x7fc068007170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc06800a200 con 0x7fc068006ce0 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068006ce0 0x7fc068007170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient: get_auth_request con 0x7fc068006ce0 auth_method 0 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 10 monclient(hunting): handle_auth_done global_id 25406 payload 824 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 1 -- 172.21.15.98:0/3444548608 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0680076b0 con 0x7fc068006ce0 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 10 client.74660.objecter ms_handle_connect 0x7fc068006ce0 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 10 client.74660.objecter resend_mon_ops 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc086ffd640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068006ce0 0x7fc068007170 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc080009f70 tx=0x7fc080009750 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc0800bc070 con 0x7fc068006ce0 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 10 monclient: dump: 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.018 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.019 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9402 (secure 0 0 0) 0x7fc080004040 con 0x7fc068006ce0 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068006ce0 msgr2=0x7fc068007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068006ce0 0x7fc068007170 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc080009f70 tx=0x7fc080009750 comp rx=0 tx=0).stop 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068006ce0 msgr2=0x7fc068007170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc06ffff640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc068006ce0 0x7fc068007170 secure :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fc080009f70 tx=0x7fc080009750 comp rx=0 tx=0).stop 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:06:03.020 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 112, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 978805, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.021 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.022 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.023 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:06:03.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.025 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.026 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:06:03.027 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:06:03.030 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.031 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.032 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:06:03.033 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.035 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:06:03.036 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.038 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.039 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.040 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.041 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.042 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:06:03.043 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:06:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 10 librados: watch_flush enter 2024-09-11T03:06:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 10 librados: watch_flush exit 2024-09-11T03:06:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 20 client.74660.objecter shutdown clearing up homeless session... 2024-09-11T03:06:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 10 client.74660.objecter successfully canceled tick 2024-09-11T03:06:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07403df40 msgr2=0x7fc074040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc07403df40 0x7fc074040400 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7fc078001570 tx=0x7fc078007a30 comp rx=0 tx=0).stop 2024-09-11T03:06:03.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc087fff640 1 -- 172.21.15.98:0/3444548608 reap_dead start 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 10 monclient: shutdown 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08813de00 msgr2=0x7fc08808e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08813de00 0x7fc08808e520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00b410 tx=0x7fc07c00b8d0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 shutdown_connections 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 --2- 172.21.15.98:0/3444548608 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fc08813de00 0x7fc08808e520 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 >> 172.21.15.98:0/3444548608 conn(0x7fc0880a4ed0 msgr2=0x7fc0880a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 shutdown_connections 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 -- 172.21.15.98:0/3444548608 wait complete. 2024-09-11T03:06:03.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.013+0000 7fc08e1b3640 1 librados: shutdown 2024-09-11T03:06:03.047 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-11T03:06:03.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon f 2024-09-11T03:06:03.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:06:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.157+0000 7fe3c6179640 10 monclient: get_monmap_and_config 2024-09-11T03:06:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.157+0000 7fe3c6179640 10 monclient: build_initial_monmap 2024-09-11T03:06:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.157+0000 7fe3c6179640 10 monclient: monmap: 2024-09-11T03:06:03.158 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:06:03.161032+0000 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:06:03.161032+0000 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:03.159 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:06:03.160 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:06:03.160 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 1 Processor -- start 2024-09-11T03:06:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 1 -- start start 2024-09-11T03:06:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient: init 2024-09-11T03:06:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:03.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient: _add_conns ranks=[7,0,1,9,10,2,4,3,6,5,8] 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3c00a46a0 0x7fe3c00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient(hunting): picked mon.noname-i con 0x7fe3c00a46a0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3c00a4fc0 addr v1:172.21.15.98:6789/0 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe3c013a040 addr v1:172.21.15.190:6789/0 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c00775b0 con 0x7fe3c00a4fc0 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c00a11d0 con 0x7fe3c013a040 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe3c00a0750 con 0x7fe3c00a46a0 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 10 monclient(hunting): authenticate will time out at 1917.041074s 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fe3c00a4fc0 legacy=0x7fe3c0139930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 -- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3c00a46a0 msgr2=0x7fe3c00a4a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3303/0 2024-09-11T03:06:03.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3c00a46a0 0x7fe3c00a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bffff640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fe3c013a040 0x7fe3c0141c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:41226/0 (socket says 172.21.15.98:41226) 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bffff640 1 -- 172.21.15.98:0/3383734820 learned_addr learned my addr 172.21.15.98:0/3383734820 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1834144356 0 0) 0x7fe3c00a11d0 con 0x7fe3c013a040 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): _negotiate 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): my global_id is 26227 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3a8003530 con 0x7fe3c013a040 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2010168055 0 0) 0x7fe3a8003530 con 0x7fe3c013a040 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3c00a46a0 msgr2=0x7fe3c00a4a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3c00a46a0 0x7fe3c00a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 >> v1:172.21.15.98:6789/0 conn(0x7fe3c00a4fc0 legacy=0x7fe3c0139930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: _finish_hunting 0 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 monclient: found mon.noname-f-legacy 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.190:6789/0 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c0077160 con 0x7fe3c013a040 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: _finish_auth 0 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 5 monclient: authenticate success, global_id 26227 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fe3b0001970 con 0x7fe3c013a040 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-09-11T03:06:03.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: dump: 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.174 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] but i'm connected to v1:172.21.15.190:6789/0, reconnecting 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 >> v1:172.21.15.190:6789/0 conn(0x7fe3c013a040 legacy=0x7fe3c0141c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient: _add_conns ranks=[4,2,3,6,7,5,8,0,1] 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3a8004210 0x7fe3a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): picked mon.c con 0x7fe3a8004210 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a8004e70 0x7fe3a80099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): picked mon.b con 0x7fe3a8004e70 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3a8009f00 0x7fe3a800c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): picked mon.g con 0x7fe3a8009f00 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c00a0750 con 0x7fe3a8004e70 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a800c9f0 con 0x7fe3a8009f00 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fe3a800cba0 con 0x7fe3a8004210 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3be7fc640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3c6179640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bffff640 1 -- 172.21.15.98:0/3383734820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3a8009f00 msgr2=0x7fe3a800c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3301/0 2024-09-11T03:06:03.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3a8004210 0x7fe3a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bffff640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3a8009f00 0x7fe3a800c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): get_auth_request con 0x7fe3a8004210 auth_method 0 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a8004e70 0x7fe3a80099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 10 monclient(hunting): get_auth_request con 0x7fe3a8004e70 auth_method 0 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_done global_id 26227 payload 293 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 -- 172.21.15.98:0/3383734820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3a8009f00 msgr2=0x7fe3a800c2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe3a8009f00 0x7fe3a800c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 -- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a8004e70 msgr2=0x7fe3a80099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a8004e70 0x7fe3a80099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 monclient: found mon.c 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bffff640 1 -- 172.21.15.98:0/3383734820 reap_dead start 2024-09-11T03:06:03.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 1 -- 172.21.15.98:0/3383734820 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a8003530 con 0x7fe3a8004210 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient: _finish_auth 0 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.161+0000 7fe3beffd640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe3a8004e70 0x7fe3a80099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3a8004210 0x7fe3a8004810 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009e30 tx=0x7fe3b801ca30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3ad729de1c9fb579 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b8009a30 con 0x7fe3a8004210 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 10 monclient: dump: 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.179 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:06:03.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b80a9070 con 0x7fe3a8004210 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient: get_monmap_and_config success 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 1 -- 172.21.15.98:0/3383734820 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b80ae470 con 0x7fe3a8004210 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 4 set_mon_vals no callback set 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3be7fc640 10 monclient: dump: 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.181 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient: shutdown 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3a8004210 msgr2=0x7fe3a8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3a8004210 0x7fe3a8004810 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009e30 tx=0x7fe3b801ca30 comp rx=0 tx=0).stop 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- 172.21.15.98:0/3383734820 shutdown_connections 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 --2- 172.21.15.98:0/3383734820 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fe3a8004210 0x7fe3a8004810 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- 172.21.15.98:0/3383734820 >> 172.21.15.98:0/3383734820 conn(0x7fe3c006ed00 msgr2=0x7fe3c006f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- 172.21.15.98:0/3383734820 shutdown_connections 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- 172.21.15.98:0/3383734820 wait complete. 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient: build_initial_monmap 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient: monmap: 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:06:03.183 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 librados: starting msgr at 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 librados: starting objecter 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 Processor -- start 2024-09-11T03:06:03.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- start start 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 librados: setting wanted keys 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 librados: calling monclient init 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient: init 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient: _add_conns ranks=[7,6,8,2,5,3,4,1,0] 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3a8004210 0x7fe3c0138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): picked mon.noname-i con 0x7fe3a8004210 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3c0138f40 0x7fe3c01236f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): picked mon.noname-d con 0x7fe3c0138f40 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3c0123c30 0x7fe3c01240b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): picked mon.noname-e con 0x7fe3c0123c30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe3c00a0ef0 con 0x7fe3c0138f40 2024-09-11T03:06:03.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fe3c00a0200 con 0x7fe3a8004210 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fe3c00a14b0 con 0x7fe3c0123c30 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3c6179640 10 monclient(hunting): authenticate will time out at 1917.047448s 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bf7fe640 1 -- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3a8004210 msgr2=0x7fe3c0138a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.190:3303/0 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bf7fe640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3a8004210 0x7fe3c0138a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3c0138f40 0x7fe3c01236f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3c0123c30 0x7fe3c01240b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3c0138f40 0x7fe3c01236f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:53106/0 (socket says 172.21.15.98:53106) 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 -- 172.21.15.98:0/1763919236 learned_addr learned my addr 172.21.15.98:0/1763919236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): get_auth_request con 0x7fe3c0138f40 auth_method 0 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 10 monclient(hunting): get_auth_request con 0x7fe3c0123c30 auth_method 0 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient(hunting): handle_auth_done global_id 84597 payload 824 2024-09-11T03:06:03.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3c0123c30 msgr2=0x7fe3c01240b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3c0123c30 0x7fe3c01240b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3a8004210 msgr2=0x7fe3c0138a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fe3a8004210 0x7fe3c0138a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient: _finish_hunting 0 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 monclient: found mon.noname-d 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.165+0000 7fe3beffd640 1 -- 172.21.15.98:0/1763919236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c01395b0 con 0x7fe3c0138f40 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bffff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fe3c0123c30 0x7fe3c01240b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3beffd640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3c0138f40 0x7fe3c01236f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009fb0 tx=0x7fe3b40054e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 client.?.objecter ms_handle_connect 0x7fe3c0138f40 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:06:03.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b4016070 con 0x7fe3c0138f40 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: dump: 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.189 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: _finish_auth 0 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: _check_auth_tickets 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b400ec20 con 0x7fe3c0138f40 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 5 monclient: authenticate success, global_id 84597 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 10 monclient: _renew_subs 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:06:03.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c0069e10 con 0x7fe3c0138f40 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b4009500 con 0x7fe3c0138f40 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: dump: 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bdffb640 4 set_mon_vals no callback set 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 10 client.84597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 10 monclient: _renew_subs 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c0126670 con 0x7fe3c0138f40 2024-09-11T03:06:03.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 1 librados: init done 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0070a30 0x7fe3c0070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c00a8260 con 0x7fe3c0070a30 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe3b4023060 con 0x7fe3c0138f40 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0070a30 0x7fe3c0070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe3ac03df40 0x7fe3ac040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient: get_auth_request con 0x7fe3c0070a30 auth_method 0 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(638..638 src has 1..638) ==== 6769+0+0 (secure 0 0 0) 0x7fe3b400f040 con 0x7fe3c0138f40 2024-09-11T03:06:03.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 client.84597.objecter ms_dispatch 0x7fe3c0000ba0 osd_map(638..638 src has 1..638) 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 3 client.84597.objecter handle_osd_map got epochs [638,638] > 0 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 3 client.84597.objecter handle_osd_map decoding full epoch 638 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 20 client.84597.objecter dump_active .. 0 homeless 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bffff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe3ac03df40 0x7fe3ac040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bffff640 10 monclient: get_auth_request con 0x7fe3ac03df40 auth_method 0 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_done global_id 26236 payload 824 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 1 -- 172.21.15.98:0/1763919236 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3c0139380 con 0x7fe3c0070a30 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 client.84597.objecter ms_handle_connect 0x7fe3c0070a30 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0070a30 0x7fe3c0070e10 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8003cc0 tx=0x7fe3a8005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 client.84597.objecter resend_mon_ops 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 client.84597.objecter ms_handle_connect 0x7fe3ac03df40 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3bffff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe3ac03df40 0x7fe3ac040400 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fe3c006ffd0 tx=0x7fe3b000ac00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b80b3070 con 0x7fe3c0070a30 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: dump: 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.195 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe3b80a9070 con 0x7fe3c0070a30 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0070a30 msgr2=0x7fe3c0070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0070a30 0x7fe3c0070e10 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8003cc0 tx=0x7fe3a8005840 comp rx=0 tx=0).stop 2024-09-11T03:06:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0070a30 msgr2=0x7fe3c0070e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:03.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.169+0000 7fe3a7fff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0070a30 0x7fe3c0070e10 secure :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8003cc0 tx=0x7fe3a8005840 comp rx=0 tx=0).stop 2024-09-11T03:06:03.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3c6179640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:06:03.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3c6179640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0071350 0x7fe3c01d9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3c6179640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c006ca10 con 0x7fe3c0071350 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0071350 0x7fe3c01d9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient: get_auth_request con 0x7fe3c0071350 auth_method 0 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 10 monclient(hunting): handle_auth_done global_id 26245 payload 824 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 1 -- 172.21.15.98:0/1763919236 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe3c0070090 con 0x7fe3c0071350 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3bf7fe640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0071350 0x7fe3c01d9470 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009e30 tx=0x7fe3b8009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3a7fff640 10 client.84597.objecter ms_handle_connect 0x7fe3c0071350 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3a7fff640 10 client.84597.objecter resend_mon_ops 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3b80ae970 con 0x7fe3c0071350 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3a7fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.177+0000 7fe3a7fff640 10 monclient: dump: 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.200 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9404 (secure 0 0 0) 0x7fe3b80ca440 con 0x7fe3c0071350 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3a7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0071350 msgr2=0x7fe3c01d9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3a7fff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0071350 0x7fe3c01d9470 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009e30 tx=0x7fe3b8009520 comp rx=0 tx=0).stop 2024-09-11T03:06:03.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3a7fff640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0071350 msgr2=0x7fe3c01d9470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3a7fff640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3c0071350 0x7fe3c01d9470 secure :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8009e30 tx=0x7fe3b8009520 comp rx=0 tx=0).stop 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 112, 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 978949, 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:06:03.202 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.203 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.204 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:06:03.205 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:03.206 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.207 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.208 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.209 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.210 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.211 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:06:03.212 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.213 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.214 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.215 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.217 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:06:03.219 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.223 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.224 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:03.225 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.225 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.225 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:06:03.225 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:06:03.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 10 librados: watch_flush enter 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 10 librados: watch_flush exit 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 20 client.84597.objecter shutdown clearing up homeless session... 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 10 client.84597.objecter successfully canceled tick 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe3ac03df40 msgr2=0x7fe3ac040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe3ac03df40 0x7fe3ac040400 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fe3c006ffd0 tx=0x7fe3b000ac00 comp rx=0 tx=0).stop 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 10 monclient: shutdown 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3c0138f40 msgr2=0x7fe3c01236f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 --2- 172.21.15.98:0/1763919236 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe3c0138f40 0x7fe3c01236f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009fb0 tx=0x7fe3b40054e0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3bffff640 1 -- 172.21.15.98:0/1763919236 reap_dead start 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3bffff640 1 -- 172.21.15.98:0/1763919236 reap_dead start 2024-09-11T03:06:03.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 shutdown_connections 2024-09-11T03:06:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 >> 172.21.15.98:0/1763919236 conn(0x7fe3c006ed00 msgr2=0x7fe3c00a7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 shutdown_connections 2024-09-11T03:06:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 -- 172.21.15.98:0/1763919236 wait complete. 2024-09-11T03:06:03.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.181+0000 7fe3c6179640 1 librados: shutdown 2024-09-11T03:06:03.228 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-11T03:06:03.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 6 for mon h 2024-09-11T03:06:03.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient: get_monmap_and_config 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient: build_initial_monmap 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient: monmap: 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:06:03.349582+0000 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:06:03.349582+0000 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.347 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:06:03.348 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 Processor -- start 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 -- start start 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient: init 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient: _add_conns ranks=[10,7,8,5,1,9,3,0,4,2,6] 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): picked mon.noname-f con 0x7f5c60139b80 addr v2:172.21.15.190:3300/0 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): picked mon.noname-i con 0x7f5c6006fe50 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): picked mon.noname-e con 0x7f5c60072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f5c600719c0 con 0x7f5c60139b80 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f5c60071c10 con 0x7f5c6006fe50 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f5c600702d0 con 0x7f5c60072640 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c684ff640 10 monclient(hunting): authenticate will time out at 1917.229772s 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c65a9a640 1 -- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6006fe50 msgr2=0x7f5c600a7a10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.190:3303/0 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c65a9a640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:44370/0 (socket says 172.21.15.98:44370) 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 -- 172.21.15.98:0/4028745459 learned_addr learned my addr 172.21.15.98:0/4028745459 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:03.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): get_auth_request con 0x7f5c60072640 auth_method 0 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c6629b640 1 --2- 172.21.15.98:0/4028745459 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c6629b640 10 monclient(hunting): get_auth_request con 0x7f5c60139b80 auth_method 0 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c6629b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c6629b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c6629b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_done global_id 25424 payload 274 2024-09-11T03:06:03.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 -- 172.21.15.98:0/4028745459 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6006fe50 msgr2=0x7f5c600a7a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 --2- 172.21.15.98:0/4028745459 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 -- 172.21.15.98:0/4028745459 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 msgr2=0x7f5c60139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 --2- 172.21.15.98:0/4028745459 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient: _finish_hunting 0 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 monclient: found mon.noname-e 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c66a9c640 1 -- 172.21.15.98:0/4028745459 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c60071370 con 0x7f5c60072640 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.349+0000 7f5c6629b640 1 --2- 172.21.15.98:0/4028745459 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:06:03.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 --2- 172.21.15.98:0/4028745459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5c60072640 0x7f5c600a7f50 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009e50 tx=0x7f5c5801c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a84b792bc4d502e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 1 -- 172.21.15.98:0/4028745459 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c580090d0 con 0x7f5c60072640 2024-09-11T03:06:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: dump: 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.363 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: _finish_auth 0 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: _check_auth_tickets 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 1 -- 172.21.15.98:0/4028745459 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c580a5ad0 con 0x7f5c60072640 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 1 -- 172.21.15.98:0/4028745459 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c580a5cc0 con 0x7f5c60072640 2024-09-11T03:06:03.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 5 monclient: authenticate success, global_id 25424 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient: get_monmap_and_config success 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 4 set_mon_vals no callback set 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65299640 10 monclient: dump: 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.365 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient: shutdown 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- 172.21.15.98:0/4028745459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5c60072640 msgr2=0x7f5c600a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 --2- 172.21.15.98:0/4028745459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5c60072640 0x7f5c600a7f50 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009e50 tx=0x7f5c5801c910 comp rx=0 tx=0).stop 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- 172.21.15.98:0/4028745459 shutdown_connections 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 --2- 172.21.15.98:0/4028745459 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5c60072640 0x7f5c600a7f50 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 --2- 172.21.15.98:0/4028745459 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f5c6006fe50 0x7f5c600a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 --2- 172.21.15.98:0/4028745459 >> v2:172.21.15.190:3300/0 conn(0x7f5c60139b80 0x7f5c60139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- 172.21.15.98:0/4028745459 >> 172.21.15.98:0/4028745459 conn(0x7f5c600ac730 msgr2=0x7f5c600a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- 172.21.15.98:0/4028745459 shutdown_connections 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- 172.21.15.98:0/4028745459 wait complete. 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient: build_initial_monmap 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient: monmap: 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:06:03.367 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 librados: starting msgr at 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 librados: starting objecter 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 Processor -- start 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- start start 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 librados: setting wanted keys 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 librados: calling monclient init 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient: init 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient: _reopen_session rank -1 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient: _add_conns ranks=[5,1,6,3,2,8,7,0,4] 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6006fe50 0x7f5c601349d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): picked mon.noname-h con 0x7f5c6006fe50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60072640 0x7f5c60134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): picked mon.noname-f con 0x7f5c60072640 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): picked mon.noname-d con 0x7f5c60139b80 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5c601435c0 con 0x7f5c60072640 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5c60143440 con 0x7f5c6006fe50 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f5c60143740 con 0x7f5c60139b80 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): _renew_subs 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c684ff640 10 monclient(hunting): authenticate will time out at 1917.235244s 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:53118/0 (socket says 172.21.15.98:53118) 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 -- 172.21.15.98:0/1217732200 learned_addr learned my addr 172.21.15.98:0/1217732200 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): get_auth_request con 0x7f5c60139b80 auth_method 0 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65a9a640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60072640 0x7f5c60134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c6629b640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6006fe50 0x7f5c601349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c6629b640 10 monclient(hunting): get_auth_request con 0x7f5c6006fe50 auth_method 0 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c6629b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c6629b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c6629b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65a9a640 10 monclient(hunting): get_auth_request con 0x7f5c60072640 auth_method 0 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65a9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65a9a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c65a9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient(hunting): handle_auth_done global_id 84606 payload 824 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6006fe50 msgr2=0x7f5c601349d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6006fe50 0x7f5c601349d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60072640 msgr2=0x7f5c60134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c60072640 0x7f5c60134f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient: _finish_hunting 0 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 monclient: found mon.noname-d 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 -- 172.21.15.98:0/1217732200 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6013c410 con 0x7f5c60139b80 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c66a9c640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5c580af040 tx=0x7f5c580196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c56ffd640 10 client.?.objecter ms_handle_connect 0x7f5c60139b80 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.353+0000 7f5c56ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c580bb020 con 0x7f5c60139b80 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: dump: 2024-09-11T03:06:03.373 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.374 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: _finish_auth 0 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c58003bb0 con 0x7f5c60139b80 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 5 monclient: authenticate success, global_id 84606 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 10 monclient: _renew_subs 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6013e090 con 0x7f5c60139b80 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c58003d60 con 0x7f5c60139b80 2024-09-11T03:06:03.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: dump: 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.376 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 10 client.84606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 10 monclient: _renew_subs 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6013ddb0 con 0x7f5c60139b80 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c64a98640 4 set_mon_vals no callback set 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 1 librados: init done 2024-09-11T03:06:03.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5c6006ca10 con 0x7f5c6013abf0 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5c58004040 con 0x7f5c60139b80 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient: get_auth_request con 0x7f5c6013abf0 auth_method 0 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4403df40 0x7f5c44040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(638..638 src has 1..638) ==== 6769+0+0 (secure 0 0 0) 0x7f5c580eab10 con 0x7f5c60139b80 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 client.84606.objecter ms_dispatch 0x7f5c60000ba0 osd_map(638..638 src has 1..638) 2024-09-11T03:06:03.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 3 client.84606.objecter handle_osd_map got epochs [638,638] > 0 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 3 client.84606.objecter handle_osd_map decoding full epoch 638 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 20 client.84606.objecter dump_active .. 0 homeless 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c65a9a640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4403df40 0x7f5c44040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c65a9a640 10 monclient: get_auth_request con 0x7f5c4403df40 auth_method 0 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 10 monclient(hunting): handle_auth_done global_id 74678 payload 824 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 1 -- 172.21.15.98:0/1217732200 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c6013d820 con 0x7f5c6013abf0 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 client.84606.objecter ms_handle_connect 0x7f5c6013abf0 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 client.84606.objecter resend_mon_ops 2024-09-11T03:06:03.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c6629b640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000f440 tx=0x7f5c5000f900 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c50019030 con 0x7f5c6013abf0 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c65a9a640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4403df40 0x7f5c44040400 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c006fd0 tx=0x7f5c5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: dump: 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.380 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 client.84606.objecter ms_handle_connect 0x7f5c4403df40 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c50014670 con 0x7f5c6013abf0 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013abf0 msgr2=0x7f5c6013afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000f440 tx=0x7f5c5000f900 comp rx=0 tx=0).stop 2024-09-11T03:06:03.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013abf0 msgr2=0x7f5c6013afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:03.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.357+0000 7f5c56ffd640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013abf0 0x7f5c6013afd0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5c5000f440 tx=0x7f5c5000f900 comp rx=0 tx=0).stop 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013b510 0x7f5c600a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5c6006e870 con 0x7f5c6013b510 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013b510 0x7f5c600a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient: get_auth_request con 0x7f5c6013b510 auth_method 0 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:06:03.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 10 monclient(hunting): handle_auth_done global_id 74687 payload 824 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 1 -- 172.21.15.98:0/1217732200 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c601357a0 con 0x7f5c6013b510 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c6629b640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013b510 0x7f5c600a0e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c50000c10 tx=0x7f5c500046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 10 client.84606.objecter ms_handle_connect 0x7f5c6013b510 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 10 client.84606.objecter resend_mon_ops 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5c50004490 con 0x7f5c6013b510 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 10 monclient: dump: 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:06:03.385 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:06:03.386 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9235 (secure 0 0 0) 0x7f5c50014e00 con 0x7f5c6013b510 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013b510 msgr2=0x7f5c600a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013b510 0x7f5c600a0e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c50000c10 tx=0x7f5c500046d0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013b510 msgr2=0x7f5c600a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:06:03.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c56ffd640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5c6013b510 0x7f5c600a0e60 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c50000c10 tx=0x7f5c500046d0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 112, 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 320295, 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:06:03.388 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 1, 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.389 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.390 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.391 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:06:03.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:06:03.393 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:06:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:06:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:06:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:06:03.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:06:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.395 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.396 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.398 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.399 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:06:03.401 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.403 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.404 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.405 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.406 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:06:03.408 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:06:03.409 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:06:03.410 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:06:03.411 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:06:03.412 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:06:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 10 librados: watch_flush enter 2024-09-11T03:06:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 10 librados: watch_flush exit 2024-09-11T03:06:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 20 client.84606.objecter shutdown clearing up homeless session... 2024-09-11T03:06:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 10 client.84606.objecter successfully canceled tick 2024-09-11T03:06:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4403df40 msgr2=0x7f5c44040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5c4403df40 0x7f5c44040400 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c006fd0 tx=0x7f5c5c008040 comp rx=0 tx=0).stop 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 10 monclient: shutdown 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.365+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5c60139b80 msgr2=0x7f5c6013a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c66a9c640 1 -- 172.21.15.98:0/1217732200 reap_dead start 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c684ff640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5c580af040 tx=0x7f5c580196b0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 shutdown_connections 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c684ff640 1 --2- 172.21.15.98:0/1217732200 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f5c60139b80 0x7f5c6013a6b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 >> 172.21.15.98:0/1217732200 conn(0x7f5c600ac730 msgr2=0x7f5c601359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 shutdown_connections 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c684ff640 1 -- 172.21.15.98:0/1217732200 wait complete. 2024-09-11T03:06:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:06:03.369+0000 7f5c684ff640 1 librados: shutdown 2024-09-11T03:06:03.416 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 6 2024-09-11T03:06:03.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-11T03:07:32.958 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7950 ms) 2024-09-11T03:07:32.958 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-11T03:07:32.958 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13189 ms) 2024-09-11T03:07:32.958 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7870 ms) 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13064 ms) 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7888 ms) 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (12164 ms) 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8074 ms) 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-11T03:07:32.959 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first is 1726023968 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,1726023973,1726023974,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,1726023973,1726023974,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,1726023973,1726023974,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,1726023973,1726023974,1726023976,1726023977,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,1726023973,1726023974,1726023976,1726023977,0 2024-09-11T03:07:32.960 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023968,1726023970,1726023971,1726023973,1726023974,1726023976,1726023977,0 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: got ls 1726023971,1726023973,1726023974,1726023976,1726023977,1726023979,1726023980,0 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: first now 1726023971, trimmed 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (45178 ms) 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: foo0 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13981 ms) 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18036 ms) 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (23016 ms) 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5154 ms) 2024-09-11T03:07:32.961 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (2797 ms) 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3038 ms) 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3031 ms) 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (285478 ms total) 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (991851 ms total) 2024-09-11T03:07:32.962 INFO:tasks.workunit.client.0.smithi190.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28188 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28188 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28380 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28380 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28720 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28720 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.964 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28519 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28519 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28429 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28429 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28231 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28231 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28461 2024-09-11T03:07:32.965 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28461 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28890 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28890 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28918 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28918 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28259 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28259 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28743 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28743 2024-09-11T03:07:32.966 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28172 2024-09-11T03:07:32.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28172 2024-09-11T03:07:32.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28204 2024-09-11T03:07:32.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28204 2024-09-11T03:07:32.970 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28692 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28692 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28164 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28164 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28180 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28180 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29021 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29021 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28398 2024-09-11T03:07:32.971 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28398 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29118 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29118 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28449 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28449 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28859 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28859 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29158 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29158 2024-09-11T03:07:32.972 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28416 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28416 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28710 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28710 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28957 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28957 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28799 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28799 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29087 2024-09-11T03:07:32.973 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29087 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28663 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28663 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28343 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28343 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29137 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29137 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28574 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28574 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.974 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28156 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28156 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28196 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28196 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28482 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28482 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28219 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28219 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28608 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28608 2024-09-11T03:07:32.975 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28631 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28631 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=28930 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 28930 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ for t in "${!pids[@]}" 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ pid=29190 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ wait 29190 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ exit 0 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ cleanup 2024-09-11T03:07:32.976 INFO:tasks.workunit.client.0.smithi190.stderr:+ pkill -P 28150 2024-09-11T03:07:32.977 INFO:tasks.workunit.client.0.smithi190.stderr:+ true 2024-09-11T03:07:32.978 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T03:07:32.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T03:07:32.992 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-11T03:07:32.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T03:07:33.356 DEBUG:teuthology.parallel:result is None 2024-09-11T03:07:33.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T03:07:33.368 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T03:07:33.368 DEBUG:teuthology.orchestra.run.smithi190:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T03:07:33.417 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-11T03:07:33.417 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T03:07:33.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T03:07:33.420 INFO:tasks.ceph.mon.a:Started 2024-09-11T03:07:33.420 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-09-11T03:07:33.420 INFO:tasks.ceph.mon.i:Restarting daemon 2024-09-11T03:07:33.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-09-11T03:07:33.422 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T03:07:33.422 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-11T03:07:33.433 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-11T03:07:33.467 INFO:tasks.ceph.mon.i:Started 2024-09-11T03:07:33.467 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-09-11T03:07:33.467 INFO:tasks.ceph.mon.g:Restarting daemon 2024-09-11T03:07:33.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-09-11T03:07:33.469 INFO:tasks.ceph.mon.g:Started 2024-09-11T03:07:33.469 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-11T03:07:33.470 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-11T03:07:33.470 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-09-11T03:07:33.470 INFO:tasks.ceph.mon.e:Sent signal 19 2024-09-11T03:07:33.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-11T03:07:33.548 INFO:tasks.ceph.mon.a.smithi098.stderr:2024-09-11T03:07:33.548+0000 7f1137c32e00 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 651 -> 0 (sync_force? mon rebuild?) 2024-09-11T03:07:33.587 INFO:tasks.ceph.mon.i.smithi190.stderr:2024-09-11T03:07:33.592+0000 7f7dda119e00 -1 mon.i@-1(???).log v0 log_external_backlog rewinding external_log_to from 651 -> 0 (sync_force? mon rebuild?) 2024-09-11T03:07:48.471 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-11T03:07:48.471 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-11T03:07:48.471 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-09-11T03:07:48.471 INFO:tasks.ceph.mon.e:Sent signal 18 2024-09-11T03:07:48.472 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:07:48.472 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:07:48.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient: get_monmap_and_config 2024-09-11T03:07:48.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient: build_initial_monmap 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient: monmap: 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:07:48.646431+0000 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:07:48.646431+0000 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:07:48.647 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 Processor -- start 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 -- start start 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient: init 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient: _reopen_session rank -1 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient: _add_conns ranks=[5,8,7,6,1,9,10,2,4,3,0] 2024-09-11T03:07:48.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): picked mon.noname-h con 0x7fcc1006f6c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): picked mon.noname-e con 0x7fcc1006faa0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): picked mon.noname-i con 0x7fcc100703c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcc10071ff0 con 0x7fcc1006f6c0 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fcc100a8750 con 0x7fcc100703c0 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcc100716c0 con 0x7fcc1006faa0 2024-09-11T03:07:48.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): _renew_subs 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc14c5d640 10 monclient(hunting): authenticate will time out at 2022.526661s 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:55564/0 (socket says 172.21.15.98:55564) 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 learned_addr learned my addr 172.21.15.98:0/469094402 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcc1006faa0 auth_method 0 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:48.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request con 0x7fcc1006f6c0 auth_method 0 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcc100703c0 auth_method 0 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:48.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:48.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.644+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:07:48.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:48.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcc1006faa0 auth_method 0 2024-09-11T03:07:48.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request con 0x7fcc1006f6c0 auth_method 0 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:48.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcc100703c0 auth_method 0 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:48.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:48.848+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request con 0x7fcc1006f6c0 auth_method 0 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcc100703c0 auth_method 0 2024-09-11T03:07:49.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcc1006faa0 auth_method 0 2024-09-11T03:07:49.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:49.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:49.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:49.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:07:49.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:49.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:49.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:49.252+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:07:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request con 0x7fcc1006f6c0 auth_method 0 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcc100703c0 auth_method 0 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:50.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcc1006faa0 auth_method 0 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:07:50.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:50.056+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:07:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): tick 2024-09-11T03:07:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): _check_auth_tickets 2024-09-11T03:07:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 monclient(hunting): continuing hunt 2024-09-11T03:07:51.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): _reopen_session rank -1 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 msgr2=0x7fcc1006fe80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcc1006faa0 0x7fcc1006fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 msgr2=0x7fcc101397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcc100703c0 0x7fcc101397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 msgr2=0x7fcc10072940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcc1006f6c0 0x7fcc10072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient: _add_conns ranks=[1,3,6,0,10,2,4,9] 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcbfc003830 addr v1:172.21.15.190:6789/0 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): picked mon.noname-g con 0x7fcbfc007400 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:51.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): picked mon.noname-d con 0x7fcbfc009ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 -- 172.21.15.98:0/469094402 --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc100a64a0 con 0x7fcbfc003830 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 -- 172.21.15.98:0/469094402 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcc10071ff0 con 0x7fcbfc007400 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 1 -- 172.21.15.98:0/469094402 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcc100a8750 con 0x7fcbfc009ef0 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0cd99640 10 monclient(hunting): _renew_subs 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcbfc009ef0 auth_method 0 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:51.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcbfc007400 auth_method 0 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.644+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:51.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.648+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:07:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcbfc009ef0 auth_method 0 2024-09-11T03:07:51.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:51.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.848+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcbfc007400 auth_method 0 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:51.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:51.852+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcbfc009ef0 auth_method 0 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.252+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:07:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:52.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcbfc007400 auth_method 0 2024-09-11T03:07:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:52.256+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-11T03:07:53.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:53.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcbfc009ef0 auth_method 0 2024-09-11T03:07:53.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:53.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.056+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:07:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcbfc007400 auth_method 0 2024-09-11T03:07:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.060+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-11T03:07:53.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.979+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890879993 0 0) 0x7fcc100a64a0 con 0x7fcbfc003830 2024-09-11T03:07:53.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.979+0000 7fcc0dd9b640 10 monclient(hunting): _negotiate 2024-09-11T03:07:53.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.979+0000 7fcc0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:53.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.979+0000 7fcc0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:53.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.979+0000 7fcc0dd9b640 10 monclient(hunting): my global_id is 26272 2024-09-11T03:07:53.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.979+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf0003530 con 0x7fcbfc003830 2024-09-11T03:07:53.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 <== mon.1 v1:172.21.15.190:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1419287821 0 0) 0x7fcbf0003530 con 0x7fcbfc003830 2024-09-11T03:07:53.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 msgr2=0x7fcbfc00c3b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcbfc009ef0 0x7fcbfc00c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 msgr2=0x7fcbfc009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbfc007400 0x7fcbfc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: _finish_hunting 0 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 monclient: found mon.noname-f-legacy 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.190:6789/0 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 --> v1:172.21.15.190:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc00dbf0 con 0x7fcbfc003830 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: _finish_auth 0 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: _check_auth_tickets 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:07:53.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/469094402 reap_dead start 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 <== mon.1 v1:172.21.15.190:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7fcc0000f040 con 0x7fcbfc003830 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: dump: 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:53.993 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] but i'm connected to v1:172.21.15.190:6789/0, reconnecting 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: _reopen_session rank -1 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 >> v1:172.21.15.190:6789/0 conn(0x7fcbfc003830 legacy=0x7fcbfc005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient: _add_conns ranks=[1,2,8,5,7,4,0,6,3] 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf0004260 0x7fcbf0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:53.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): picked mon.f con 0x7fcbf0004260 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf0004e70 0x7fcbf00099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): picked mon.b con 0x7fcbf0004e70 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf0009f00 0x7fcbf000c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): picked mon.e con 0x7fcbf0009f00 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc10071ff0 con 0x7fcbf0004260 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fcc100a8750 con 0x7fcbf0004e70 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcbf000c9f0 con 0x7fcbf0009f00 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): _renew_subs 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0dd9b640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf0009f00 0x7fcbf000c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:53.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf0004260 0x7fcbf0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request con 0x7fcbf0004260 auth_method 0 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcbf0009f00 auth_method 0 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:53.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf0004e70 0x7fcbf00099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcbf0004e70 auth_method 0 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc05d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:07:53.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient(hunting): handle_auth_done global_id 26272 payload 293 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf0009f00 msgr2=0x7fcbf000c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf0009f00 0x7fcbf000c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf0004e70 msgr2=0x7fcbf00099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf0004e70 0x7fcbf00099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient: _finish_hunting 0 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 1 monclient: found mon.f 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.190:3300/0 2024-09-11T03:07:53.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 1 -- 172.21.15.98:0/469094402 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf0003530 con 0x7fcbf0004260 2024-09-11T03:07:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient: _finish_auth 0 2024-09-11T03:07:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 10 monclient: _check_auth_tickets 2024-09-11T03:07:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc0e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:07:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc14c5d640 5 monclient: authenticate success, global_id 26272 2024-09-11T03:07:53.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.987+0000 7fcc14c5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:07:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.991+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf0004260 0x7fcbf0004810 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000ee0 tx=0x7fcc0001d610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39266d912f735e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:07:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc0000ea50 con 0x7fcbf0004260 2024-09-11T03:07:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:07:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 10 monclient: dump: 2024-09-11T03:07:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:54.006 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:54.007 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc14c5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc000b2070 con 0x7fcbf0004260 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 1 -- 172.21.15.98:0/469094402 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc000ad880 con 0x7fcbf0004260 2024-09-11T03:07:54.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:07:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc0dd9b640 10 monclient: dump: 2024-09-11T03:07:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.009 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:54.011 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc14c5d640 10 monclient: get_monmap_and_config success 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.995+0000 7fcc14c5d640 4 set_mon_vals no callback set 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient: shutdown 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf0004260 msgr2=0x7fcbf0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:54.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf0004260 0x7fcbf0004810 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000ee0 tx=0x7fcc0001d610 comp rx=0 tx=0).stop 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- 172.21.15.98:0/469094402 shutdown_connections 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fcbf0004e70 0x7fcbf00099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcbf0009f00 0x7fcbf000c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/469094402 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcbf0004260 0x7fcbf0004810 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- 172.21.15.98:0/469094402 >> 172.21.15.98:0/469094402 conn(0x7fcc100acac0 msgr2=0x7fcc100a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- 172.21.15.98:0/469094402 shutdown_connections 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- 172.21.15.98:0/469094402 wait complete. 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient: build_initial_monmap 2024-09-11T03:07:54.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient: monmap: 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:07:54.014 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 librados: starting msgr at 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 librados: starting objecter 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 Processor -- start 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- start start 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 librados: setting wanted keys 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 librados: calling monclient init 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient: init 2024-09-11T03:07:54.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient: _reopen_session rank -1 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient: _add_conns ranks=[5,3,0,4,6,7,8,1,2] 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcbf0004260 0x7fcc101d10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): picked mon.noname-h con 0x7fcbf0004260 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbf0004e70 0x7fcc10138800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): picked mon.noname-g con 0x7fcbf0004e70 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 0x7fcc10138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): picked mon.noname-a con 0x7fcbf0009f00 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcc1013a360 con 0x7fcbf0009f00 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fcc1013a1e0 con 0x7fcbf0004e70 2024-09-11T03:07:54.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcc1013a4e0 con 0x7fcbf0004260 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): _renew_subs 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:53.999+0000 7fcc14c5d640 10 monclient(hunting): authenticate will time out at 2027.883790s 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 0x7fcc10138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 0x7fcc10138d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:35684/0 (socket says 172.21.15.98:35684) 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/2163570712 learned_addr learned my addr 172.21.15.98:0/2163570712 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcbf0009f00 auth_method 0 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcbf0004260 0x7fcc101d10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbf0004e70 0x7fcc10138800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:54.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request con 0x7fcbf0004260 auth_method 0 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request con 0x7fcbf0004e70 auth_method 0 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/2163570712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 msgr2=0x7fcc10138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 1 -- 172.21.15.98:0/2163570712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 msgr2=0x7fcc10138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 0x7fcc10138d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0ed9d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 0x7fcc10138d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:07:54.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient(hunting): handle_auth_done global_id 104106 payload 824 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 -- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcbf0004260 msgr2=0x7fcc101d10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcbf0004260 0x7fcc101d10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 -- 172.21.15.98:0/2163570712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 msgr2=0x7fcc10138d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 0x7fcc10138d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 monclient: found mon.noname-g 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.190:3301/0 2024-09-11T03:07:54.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 -- 172.21.15.98:0/2163570712 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc101261c0 con 0x7fcbf0004e70 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc05d9b640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbf0004e70 0x7fcc10138800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8017870 tx=0x7fcbf8017d30 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 client.?.objecter ms_handle_connect 0x7fcbf0004e70 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 1 -- 172.21.15.98:0/2163570712 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbf8028070 con 0x7fcbf0004e70 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: dump: 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:54.020 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: _finish_auth 0 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: _check_auth_tickets 2024-09-11T03:07:54.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 5 monclient: authenticate success, global_id 104106 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 monclient: _renew_subs 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc10124bf0 con 0x7fcbf0004e70 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 1 -- 172.21.15.98:0/2163570712 <== mon.3 v2:172.21.15.190:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf8016bc0 con 0x7fcbf0004e70 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 1 -- 172.21.15.98:0/2163570712 <== mon.3 v2:172.21.15.190:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcbf800f040 con 0x7fcbf0004e70 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0d59a640 4 set_mon_vals no callback set 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 client.104106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:07:54.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 monclient: dump: 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:54.023 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 monclient: _renew_subs 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc10123e60 con 0x7fcbf0004e70 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 1 librados: init done 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc1013cc70 con 0x7fcbf0004e70 2024-09-11T03:07:54.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 1 -- 172.21.15.98:0/2163570712 <== mon.3 v2:172.21.15.190:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fcbf8015040 con 0x7fcbf0004e70 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbfc03ef80 0x7fcbfc041440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 1 -- 172.21.15.98:0/2163570712 <== mon.3 v2:172.21.15.190:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fcbf8050110 con 0x7fcbf0004e70 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 10 client.104106.objecter ms_dispatch 0x7fcc10000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 3 client.104106.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 3 client.104106.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc077fe640 20 client.104106.objecter dump_active .. 0 homeless 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbfc03ef80 0x7fcbfc041440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:54.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.003+0000 7fcc0e59c640 10 monclient: get_auth_request con 0x7fcbfc03ef80 auth_method 0 2024-09-11T03:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.007+0000 7fcc077fe640 10 client.104106.objecter ms_handle_connect 0x7fcbfc03ef80 2024-09-11T03:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.007+0000 7fcc0e59c640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbfc03ef80 0x7fcbfc041440 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 crypto rx=0x7fcc000b1000 tx=0x7fcc00004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.019+0000 7fcc077fe640 1 -- 172.21.15.98:0/2163570712 <== mon.3 v2:172.21.15.190:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fcbf8020070 con 0x7fcbf0004e70 2024-09-11T03:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.019+0000 7fcc077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:07:54.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.019+0000 7fcc077fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:07:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.251+0000 7fcc14c5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:07:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.251+0000 7fcc14c5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:07:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.251+0000 7fcc14c5d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.190:3301/0 2024-09-11T03:07:54.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.251+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcc100a5840 con 0x7fcbf0004e70 2024-09-11T03:07:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.251+0000 7fcc077fe640 1 -- 172.21.15.98:0/2163570712 <== mon.3 v2:172.21.15.190:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3187 (secure 0 0 0) 0x7fcbf8015360 con 0x7fcbf0004e70 2024-09-11T03:07:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.251+0000 7fcc077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:07:54.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.251+0000 7fcc077fe640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:07:54.254 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":120,"quorum":[1,2,3,4,5,6,7,8],"quorum_names":["f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:07:54.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 10 librados: watch_flush enter 2024-09-11T03:07:54.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 10 librados: watch_flush exit 2024-09-11T03:07:54.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 20 client.104106.objecter shutdown clearing up homeless session... 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 10 client.104106.objecter successfully canceled tick 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbfc03ef80 msgr2=0x7fcbfc041440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbfc03ef80 0x7fcbfc041440 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 crypto rx=0x7fcc000b1000 tx=0x7fcc00004740 comp rx=0 tx=0).stop 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 10 monclient: shutdown 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbf0004e70 msgr2=0x7fcc10138800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbf0004e70 0x7fcc10138800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8017870 tx=0x7fcbf8017d30 comp rx=0 tx=0).stop 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 shutdown_connections 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcbfc03ef80 0x7fcbfc041440 unknown :-1 s=CLOSED pgs=3696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcbf0009f00 0x7fcc10138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fcbf0004e70 0x7fcc10138800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 --2- 172.21.15.98:0/2163570712 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcbf0004260 0x7fcc101d10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 >> 172.21.15.98:0/2163570712 conn(0x7fcc100acac0 msgr2=0x7fcc100aa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:07:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 shutdown_connections 2024-09-11T03:07:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 -- 172.21.15.98:0/2163570712 wait complete. 2024-09-11T03:07:54.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:54.255+0000 7fcc14c5d640 1 librados: shutdown 2024-09-11T03:07:57.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:07:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient: get_monmap_and_config 2024-09-11T03:07:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient: build_initial_monmap 2024-09-11T03:07:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient: monmap: 2024-09-11T03:07:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:07:57.404 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:07:57.406988+0000 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:07:57.406988+0000 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:07:57.405 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:07:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:07:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:07:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:07:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:07:57.406 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:07:57.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 Processor -- start 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 -- start start 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient: init 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient: _reopen_session rank -1 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient: _add_conns ranks=[9,8,10,5,7,4,6,0,1,3,2] 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 0x7f21a00725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): picked mon.noname-a con 0x7f21a0139b10 addr v2:172.21.15.98:3300/0 2024-09-11T03:07:57.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f21a006fe30 0x7f21a0139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): picked mon.noname-e con 0x7f21a006fe30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f21a00a7a10 0x7f21a00a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): picked mon.noname-f con 0x7f21a00a7a10 addr v2:172.21.15.190:3300/0 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f21a00a9650 con 0x7f21a0139b10 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f21a0071350 con 0x7f21a00a7a10 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f21a0071630 con 0x7f21a006fe30 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): _renew_subs 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f21a4ee7640 10 monclient(hunting): authenticate will time out at 2031.287359s 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 0x7f21a00725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:57.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 0x7f21a00725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53944/0 (socket says 172.21.15.98:53944) 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 1 -- 172.21.15.98:0/2505772831 learned_addr learned my addr 172.21.15.98:0/2505772831 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f21a006fe30 0x7f21a0139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 10 monclient(hunting): get_auth_request con 0x7f21a0139b10 auth_method 0 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): get_auth_request con 0x7f21a006fe30 auth_method 0 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 1 -- 172.21.15.98:0/2505772831 >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 msgr2=0x7f21a00725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-11T03:07:57.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 1 -- 172.21.15.98:0/2505772831 >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 msgr2=0x7f21a00725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 1 --2- 172.21.15.98:0/2505772831 >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 0x7f21a00725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219ed9d640 1 --2- 172.21.15.98:0/2505772831 >> v2:172.21.15.190:3300/0 conn(0x7f21a00a7a10 0x7f21a00a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219e59c640 1 --2- 172.21.15.98:0/2505772831 >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 0x7f21a00725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219ed9d640 10 monclient(hunting): get_auth_request con 0x7f21a00a7a10 auth_method 0 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 10 monclient(hunting): handle_auth_done global_id 25433 payload 274 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 1 -- 172.21.15.98:0/2505772831 >> v2:172.21.15.190:3300/0 conn(0x7f21a00a7a10 msgr2=0x7f21a00a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:07:57.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.403+0000 7f219dd9b640 1 --2- 172.21.15.98:0/2505772831 >> v2:172.21.15.190:3300/0 conn(0x7f21a00a7a10 0x7f21a00a7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 1 -- 172.21.15.98:0/2505772831 >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 msgr2=0x7f21a00725d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-11T03:07:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 1 --2- 172.21.15.98:0/2505772831 >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 0x7f21a00725d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 10 monclient: _finish_hunting 0 2024-09-11T03:07:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 1 monclient: found mon.noname-e 2024-09-11T03:07:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:07:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:07:57.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 1 -- 172.21.15.98:0/2505772831 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a00702b0 con 0x7f21a006fe30 2024-09-11T03:07:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219dd9b640 1 --2- 172.21.15.98:0/2505772831 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f21a006fe30 0x7f21a0139ef0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f21a0072ba0 tx=0x7f219401c7f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a76c952175b5f91c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:07:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 1 -- 172.21.15.98:0/2505772831 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21940ac070 con 0x7f21a006fe30 2024-09-11T03:07:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:07:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: dump: 2024-09-11T03:07:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:57.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:57.419 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: _finish_auth 0 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: _check_auth_tickets 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 5 monclient: authenticate success, global_id 25433 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 1 -- 172.21.15.98:0/2505772831 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21940a68f0 con 0x7f21a006fe30 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 10 monclient: get_monmap_and_config success 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 4 set_mon_vals no callback set 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 1 -- 172.21.15.98:0/2505772831 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21940a7460 con 0x7f21a006fe30 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:57.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f219d59a640 10 monclient: dump: 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:57.421 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 10 monclient: shutdown 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 -- 172.21.15.98:0/2505772831 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f21a006fe30 msgr2=0x7f21a0139ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/2505772831 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f21a006fe30 0x7f21a0139ef0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f21a0072ba0 tx=0x7f219401c7f0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 -- 172.21.15.98:0/2505772831 shutdown_connections 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/2505772831 >> v2:172.21.15.190:3300/0 conn(0x7f21a00a7a10 0x7f21a00a7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/2505772831 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f21a006fe30 0x7f21a0139ef0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/2505772831 >> v2:172.21.15.98:3300/0 conn(0x7f21a0139b10 0x7f21a00725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 -- 172.21.15.98:0/2505772831 >> 172.21.15.98:0/2505772831 conn(0x7f21a00ac740 msgr2=0x7f21a00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 -- 172.21.15.98:0/2505772831 shutdown_connections 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 -- 172.21.15.98:0/2505772831 wait complete. 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 10 monclient: build_initial_monmap 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 10 monclient: monmap: 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:07:57.423 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 librados: starting msgr at 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 librados: starting objecter 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.407+0000 7f21a4ee7640 1 Processor -- start 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 -- start start 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 librados: setting wanted keys 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 librados: calling monclient init 2024-09-11T03:07:57.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: init 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _reopen_session rank -1 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _add_conns ranks=[2,7,1,8,0,4,3,5,6] 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21a00a7a10 0x7f21a00a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): picked mon.noname-b con 0x7f21a00a7a10 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21a0139b10 0x7f21a01d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): picked mon.noname-i con 0x7f21a0139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a01d2140 0x7f21a013a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): picked mon.noname-f con 0x7f21a01d2140 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f21a01435c0 con 0x7f21a01d2140 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f21a0143440 con 0x7f21a00a7a10 2024-09-11T03:07:57.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): start opening mon connection 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f21a0143740 con 0x7f21a0139b10 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): _renew_subs 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient(hunting): authenticate will time out at 2031.292531s 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21a00a7a10 0x7f21a00a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21a00a7a10 0x7f21a00a52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60360/0 (socket says 172.21.15.98:60360) 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 -- 172.21.15.98:0/3874285780 learned_addr learned my addr 172.21.15.98:0/3874285780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): get_auth_request con 0x7f21a00a7a10 auth_method 0 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:07:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219ed9d640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a01d2140 0x7f21a013a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient(hunting): handle_auth_done global_id 94304 payload 824 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219dd9b640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21a0139b10 0x7f21a01d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 -- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21a0139b10 msgr2=0x7f21a01d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21a0139b10 0x7f21a01d1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 -- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a01d2140 msgr2=0x7f21a013a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a01d2140 0x7f21a013a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient: _finish_hunting 0 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 monclient: found mon.noname-b 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 -- 172.21.15.98:0/3874285780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a0073600 con 0x7f21a00a7a10 2024-09-11T03:07:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219ed9d640 10 monclient: get_auth_request con 0x7f21a01d2140 auth_method 0 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219ed9d640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a01d2140 0x7f21a013a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219e59c640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21a00a7a10 0x7f21a00a52b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f219000e810 tx=0x7f219000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 client.?.objecter ms_handle_connect 0x7f21a00a7a10 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 1 -- 172.21.15.98:0/3874285780 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2190017070 con 0x7f21a00a7a10 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: dump: 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: _finish_auth 0 2024-09-11T03:07:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: _check_auth_tickets 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 5 monclient: authenticate success, global_id 94304 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 1 -- 172.21.15.98:0/3874285780 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219000ee00 con 0x7f21a00a7a10 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _renew_subs 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a0073140 con 0x7f21a00a7a10 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 1 -- 172.21.15.98:0/3874285780 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f219000f040 con 0x7f21a00a7a10 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219cd99640 4 set_mon_vals no callback set 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 client.94304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:07:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 monclient: dump: 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:07:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _renew_subs 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a013d3e0 con 0x7f21a00a7a10 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 librados: init done 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 1 -- 172.21.15.98:0/3874285780 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f2190012050 con 0x7f21a00a7a10 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:07:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21a00a7ea0 con 0x7f21a00a7a10 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f218003df40 0x7f2180040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 1 -- 172.21.15.98:0/3874285780 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f219004edf0 con 0x7f21a00a7a10 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 client.94304.objecter ms_dispatch 0x7f21a0000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 3 client.94304.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 3 client.94304.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 20 client.94304.objecter dump_active .. 0 homeless 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219dd9b640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f218003df40 0x7f2180040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219dd9b640 10 monclient: get_auth_request con 0x7f218003df40 auth_method 0 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f218effd640 10 client.94304.objecter ms_handle_connect 0x7f218003df40 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.411+0000 7f219dd9b640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f218003df40 0x7f2180040400 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f21940ae000 tx=0x7f21940a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:07:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.415+0000 7f218effd640 1 -- 172.21.15.98:0/3874285780 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f2190020030 con 0x7f21a00a7a10 2024-09-11T03:07:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.415+0000 7f218effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:07:57.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.415+0000 7f218effd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:07:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.647+0000 7f21a4ee7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:07:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.647+0000 7f21a4ee7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:07:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.647+0000 7f21a4ee7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:07:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.647+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f21a013bec0 con 0x7f21a00a7a10 2024-09-11T03:07:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f218effd640 1 -- 172.21.15.98:0/3874285780 <== mon.2 v2:172.21.15.98:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3187 (secure 0 0 0) 0x7f2190013410 con 0x7f21a00a7a10 2024-09-11T03:07:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f218effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:07:57.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f218effd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:07:57.651 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":120,"quorum":[1,2,3,4,5,6,7,8],"quorum_names":["f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:07:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 10 librados: watch_flush enter 2024-09-11T03:07:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 10 librados: watch_flush exit 2024-09-11T03:07:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 20 client.94304.objecter shutdown clearing up homeless session... 2024-09-11T03:07:57.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 10 client.94304.objecter successfully canceled tick 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f218003df40 msgr2=0x7f2180040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f218003df40 0x7f2180040400 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f21940ae000 tx=0x7f21940a7000 comp rx=0 tx=0).stop 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 10 monclient: shutdown 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21a00a7a10 msgr2=0x7f21a00a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21a00a7a10 0x7f21a00a52b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f219000e810 tx=0x7f219000ecd0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 shutdown_connections 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21a01d2140 0x7f21a013a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f21a0139b10 0x7f21a01d1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f218003df40 0x7f2180040400 unknown :-1 s=CLOSED pgs=3697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 --2- 172.21.15.98:0/3874285780 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f21a00a7a10 0x7f21a00a52b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 >> 172.21.15.98:0/3874285780 conn(0x7f21a00ac740 msgr2=0x7f21a0139340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:07:57.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 shutdown_connections 2024-09-11T03:07:57.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 -- 172.21.15.98:0/3874285780 wait complete. 2024-09-11T03:07:57.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:07:57.651+0000 7f21a4ee7640 1 librados: shutdown 2024-09-11T03:08:00.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:08:00.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient: get_monmap_and_config 2024-09-11T03:08:00.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient: build_initial_monmap 2024-09-11T03:08:00.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient: monmap: 2024-09-11T03:08:00.800 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:00.800 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:00.800 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:00.802947+0000 2024-09-11T03:08:00.800 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:00.802947+0000 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:00.801 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:00.802 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:00.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 Processor -- start 2024-09-11T03:08:00.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 -- start start 2024-09-11T03:08:00.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient: init 2024-09-11T03:08:00.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:00.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient: _add_conns ranks=[7,4,6,5,10,9,2,3,0,1,8] 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8ee80a3dc0 0x7f8ee80a41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): picked mon.noname-i con 0x7f8ee80a3dc0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80773b0 0x7f8ee80a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee80773b0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80a0410 0x7f8ee8139a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): picked mon.noname-d con 0x7f8ee80a0410 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8ee80a4c20 con 0x7f8ee80773b0 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8ee80a0bd0 con 0x7f8ee80a0410 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:00.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f8ee80a5730 con 0x7f8ee80a3dc0 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8ef00e8640 10 monclient(hunting): authenticate will time out at 2034.683246s 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80773b0 0x7f8ee80a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80773b0 0x7f8ee80a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57714/0 (socket says 172.21.15.98:57714) 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 -- 172.21.15.98:0/2172784767 learned_addr learned my addr 172.21.15.98:0/2172784767 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): get_auth_request con 0x7f8ee80773b0 auth_method 0 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:00.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eee685640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80a0410 0x7f8ee8139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eee685640 10 monclient(hunting): get_auth_request con 0x7f8ee80a0410 auth_method 0 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eee685640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eee685640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eee685640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient(hunting): handle_auth_done global_id 24862 payload 274 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 -- 172.21.15.98:0/2172784767 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8ee80a3dc0 msgr2=0x7f8ee80a41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8ee80a3dc0 0x7f8ee80a41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 -- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80a0410 msgr2=0x7f8ee8139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80a0410 0x7f8ee8139a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient: _finish_hunting 0 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 monclient: found mon.noname-c 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:00.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.799+0000 7f8eed683640 1 -- 172.21.15.98:0/2172784767 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee80a0eb0 con 0x7f8ee80773b0 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eed683640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80773b0 0x7f8ee80a46e0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80098d0 tx=0x7f8ed801c1e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=cc2ada9f72463285 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 1 -- 172.21.15.98:0/2172784767 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed80b0070 con 0x7f8ee80773b0 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: dump: 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:00.808 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: _finish_auth 0 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: _check_auth_tickets 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:00.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 1 -- 172.21.15.98:0/2172784767 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed80aa810 con 0x7f8ee80773b0 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 5 monclient: authenticate success, global_id 24862 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 1 -- 172.21.15.98:0/2172784767 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed80a9300 con 0x7f8ee80773b0 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 4 set_mon_vals no callback set 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8eece82640 10 monclient: dump: 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.810 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:00.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 10 monclient: shutdown 2024-09-11T03:08:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:00.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 -- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80773b0 msgr2=0x7f8ee80a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80773b0 0x7f8ee80a46e0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80098d0 tx=0x7f8ed801c1e0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 -- 172.21.15.98:0/2172784767 shutdown_connections 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80773b0 0x7f8ee80a46e0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80a0410 0x7f8ee8139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/2172784767 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f8ee80a3dc0 0x7f8ee80a41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 -- 172.21.15.98:0/2172784767 >> 172.21.15.98:0/2172784767 conn(0x7f8ee80a6430 msgr2=0x7f8ee806ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 -- 172.21.15.98:0/2172784767 shutdown_connections 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 -- 172.21.15.98:0/2172784767 wait complete. 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 10 monclient: build_initial_monmap 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 10 monclient: monmap: 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:00.817 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:00.818 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 librados: starting msgr at 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 librados: starting objecter 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 Processor -- start 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 -- start start 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 librados: setting wanted keys 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 1 librados: calling monclient init 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.803+0000 7f8ef00e8640 10 monclient: init 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:00.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _add_conns ranks=[6,4,0,8,1,5,7,3,2] 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80773b0 0x7f8ee81d12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): picked mon.noname-d con 0x7f8ee80773b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80a0410 0x7f8ee81d1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): picked mon.noname-c con 0x7f8ee80a0410 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80a3dc0 0x7f8ee81d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): picked mon.noname-a con 0x7f8ee80a3dc0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8143050 con 0x7f8ee80a3dc0 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f8ee8142ed0 con 0x7f8ee80a0410 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:00.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f8ee81431d0 con 0x7f8ee80773b0 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient(hunting): authenticate will time out at 2034.688266s 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80a0410 0x7f8ee81d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80a3dc0 0x7f8ee81d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80a0410 0x7f8ee81d1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57722/0 (socket says 172.21.15.98:57722) 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80773b0 0x7f8ee81d12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 1 -- 172.21.15.98:0/1423092817 learned_addr learned my addr 172.21.15.98:0/1423092817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient(hunting): get_auth_request con 0x7f8ee80a0410 auth_method 0 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): get_auth_request con 0x7f8ee80773b0 auth_method 0 2024-09-11T03:08:00.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 10 monclient(hunting): get_auth_request con 0x7f8ee80a3dc0 auth_method 0 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:00.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient(hunting): handle_auth_done global_id 84624 payload 824 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 -- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80a0410 msgr2=0x7f8ee81d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80a0410 0x7f8ee81d1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 -- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80a3dc0 msgr2=0x7f8ee81d1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80a3dc0 0x7f8ee81d1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient: _finish_hunting 0 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 monclient: found mon.noname-d 2024-09-11T03:08:00.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 -- 172.21.15.98:0/1423092817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee813eb20 con 0x7f8ee80773b0 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80a0410 0x7f8ee81d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eee685640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80a3dc0 0x7f8ee81d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eede84640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80773b0 0x7f8ee81d12d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400b410 tx=0x7f8ee400b810 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 client.?.objecter ms_handle_connect 0x7f8ee80773b0 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 1 -- 172.21.15.98:0/1423092817 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ee4015020 con 0x7f8ee80773b0 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: dump: 2024-09-11T03:08:00.824 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:00.825 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 1 -- 172.21.15.98:0/1423092817 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee4011aa0 con 0x7f8ee80773b0 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 1 -- 172.21.15.98:0/1423092817 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ee4011c90 con 0x7f8ee80773b0 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 monclient: dump: 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:00.826 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:00.827 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8edffff640 4 set_mon_vals no callback set 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 5 monclient: authenticate success, global_id 84624 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _renew_subs 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee80a74b0 con 0x7f8ee80773b0 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 client.84624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _renew_subs 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee813e840 con 0x7f8ee80773b0 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 librados: init done 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:00.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 1 -- 172.21.15.98:0/1423092817 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f8ee4011630 con 0x7f8ee80773b0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee8139a60 con 0x7f8ee80773b0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ecc03df70 0x7f8ecc040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 1 -- 172.21.15.98:0/1423092817 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f8ee404f4e0 con 0x7f8ee80773b0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 client.84624.objecter ms_dispatch 0x7f8ee8000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 3 client.84624.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 3 client.84624.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 20 client.84624.objecter dump_active .. 0 homeless 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ecc03df70 0x7f8ecc040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 10 monclient: get_auth_request con 0x7f8ecc03df70 auth_method 0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8ede7fc640 10 client.84624.objecter ms_handle_connect 0x7f8ecc03df70 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.807+0000 7f8eed683640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ecc03df70 0x7f8ecc040430 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80097c0 tx=0x7f8ed80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:00.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.811+0000 7f8ede7fc640 1 -- 172.21.15.98:0/1423092817 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f8ee401f020 con 0x7f8ee80773b0 2024-09-11T03:08:00.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.811+0000 7f8ede7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:00.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:00.811+0000 7f8ede7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:01.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.051+0000 7f8ef00e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:08:01.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.051+0000 7f8ef00e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:08:01.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.051+0000 7f8ef00e8640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.051+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8ee813d320 con 0x7f8ee80773b0 2024-09-11T03:08:01.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ede7fc640 1 -- 172.21.15.98:0/1423092817 <== mon.6 v2:172.21.15.98:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f8ee4019da0 con 0x7f8ee80773b0 2024-09-11T03:08:01.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ede7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:08:01.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ede7fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:01.054 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":122,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:08:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 10 librados: watch_flush enter 2024-09-11T03:08:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 10 librados: watch_flush exit 2024-09-11T03:08:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 20 client.84624.objecter shutdown clearing up homeless session... 2024-09-11T03:08:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 10 client.84624.objecter successfully canceled tick 2024-09-11T03:08:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ecc03df70 msgr2=0x7f8ecc040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ecc03df70 0x7f8ecc040430 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80097c0 tx=0x7f8ed80a6000 comp rx=0 tx=0).stop 2024-09-11T03:08:01.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 10 monclient: shutdown 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80773b0 msgr2=0x7f8ee81d12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80773b0 0x7f8ee81d12d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8ee400b410 tx=0x7f8ee400b810 comp rx=0 tx=0).stop 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 shutdown_connections 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f8ecc03df70 0x7f8ecc040430 unknown :-1 s=CLOSED pgs=3698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ee80a3dc0 0x7f8ee81d1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f8ee80a0410 0x7f8ee81d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 --2- 172.21.15.98:0/1423092817 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f8ee80773b0 0x7f8ee81d12d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 >> 172.21.15.98:0/1423092817 conn(0x7f8ee80a6430 msgr2=0x7f8ee809f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 shutdown_connections 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 -- 172.21.15.98:0/1423092817 wait complete. 2024-09-11T03:08:01.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.055+0000 7f8ef00e8640 1 librados: shutdown 2024-09-11T03:08:01.074 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:08:01.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-09-11T03:08:01.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-11T03:08:01.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.195+0000 7f02e7ee0640 10 monclient: get_monmap_and_config 2024-09-11T03:08:01.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.195+0000 7f02e7ee0640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.195+0000 7f02e7ee0640 10 monclient: monmap: 2024-09-11T03:08:01.198 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.198 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.198 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:01.200830+0000 2024-09-11T03:08:01.198 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:01.200830+0000 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:01.199 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 1 Processor -- start 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 1 -- start start 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient: init 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient: _add_conns ranks=[1,2,7,3,5,10,6,0,8,9,4] 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f02e0139b10 addr v1:172.21.15.190:6789/0 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02e0072600 0x7f02e013ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f02e0072600 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e013ef60 0x7f02e0141350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient(hunting): picked mon.noname-i con 0x7f02e013ef60 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02e013a600 con 0x7f02e0139b10 2024-09-11T03:08:01.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f02e00702b0 con 0x7f02e0072600 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f02e0071bf0 con 0x7f02e013ef60 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient(hunting): authenticate will time out at 2035.081144s 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02e0072600 0x7f02e013ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02e0072600 0x7f02e013ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60370/0 (socket says 172.21.15.98:60370) 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 -- 172.21.15.98:0/1809883004 learned_addr learned my addr 172.21.15.98:0/1809883004 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): get_auth_request con 0x7f02e0072600 auth_method 0 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e647d640 1 --2- 172.21.15.98:0/1809883004 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e013ef60 0x7f02e0141350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e647d640 10 monclient(hunting): get_auth_request con 0x7f02e013ef60 auth_method 0 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e647d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e647d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e647d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient(hunting): handle_auth_done global_id 94313 payload 274 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 -- 172.21.15.98:0/1809883004 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e013ef60 msgr2=0x7f02e0141350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 --2- 172.21.15.98:0/1809883004 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e013ef60 0x7f02e0141350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 -- 172.21.15.98:0/1809883004 >> v1:172.21.15.190:6789/0 conn(0x7f02e0139b10 legacy=0x7f02e0139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 monclient: found mon.noname-b 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 -- 172.21.15.98:0/1809883004 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e0071350 con 0x7f02e0072600 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e547b640 1 --2- 172.21.15.98:0/1809883004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02e0072600 0x7f02e013ea20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f02d00098a0 tx=0x7f02d001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47fbadff18cb1925 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 1 -- 172.21.15.98:0/1809883004 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02d00ab860 con 0x7f02e0072600 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: dump: 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.207 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.208 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 1 -- 172.21.15.98:0/1809883004 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02d00b0070 con 0x7f02e0072600 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 1 -- 172.21.15.98:0/1809883004 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02d00aa350 con 0x7f02e0072600 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e4c7a640 10 monclient: dump: 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.209 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 5 monclient: authenticate success, global_id 94313 2024-09-11T03:08:01.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.199+0000 7f02e7ee0640 4 set_mon_vals no callback set 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: shutdown 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/1809883004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02e0072600 msgr2=0x7f02e013ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 --2- 172.21.15.98:0/1809883004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02e0072600 0x7f02e013ea20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f02d00098a0 tx=0x7f02d001c410 comp rx=0 tx=0).stop 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/1809883004 shutdown_connections 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 --2- 172.21.15.98:0/1809883004 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e013ef60 0x7f02e0141350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 --2- 172.21.15.98:0/1809883004 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f02e0072600 0x7f02e013ea20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/1809883004 >> 172.21.15.98:0/1809883004 conn(0x7f02e00ac790 msgr2=0x7f02e00a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/1809883004 shutdown_connections 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/1809883004 wait complete. 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: monmap: 2024-09-11T03:08:01.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.212 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 librados: starting msgr at 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 librados: starting objecter 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 Processor -- start 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- start start 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 librados: setting wanted keys 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 librados: calling monclient init 2024-09-11T03:08:01.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: init 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: _add_conns ranks=[1,7,6,8,3,4,5,0,2] 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02e0072600 0x7f02e00a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): picked mon.noname-f con 0x7f02e0072600 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e0139b10 0x7f02e01d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): picked mon.noname-i con 0x7f02e0139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02e013ef60 0x7f02e00a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): picked mon.noname-d con 0x7f02e013ef60 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f02e0144230 con 0x7f02e0072600 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f02e01440b0 con 0x7f02e013ef60 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f02e01443b0 con 0x7f02e0139b10 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): authenticate will time out at 2035.086374s 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02e013ef60 0x7f02e00a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02e013ef60 0x7f02e00a2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54444/0 (socket says 172.21.15.98:54444) 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 -- 172.21.15.98:0/3182605785 learned_addr learned my addr 172.21.15.98:0/3182605785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): get_auth_request con 0x7f02e013ef60 auth_method 0 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e547b640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e0139b10 0x7f02e01d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e547b640 10 monclient(hunting): get_auth_request con 0x7f02e0139b10 auth_method 0 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e547b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e547b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e547b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e5c7c640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02e0072600 0x7f02e00a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e5c7c640 10 monclient(hunting): get_auth_request con 0x7f02e0072600 auth_method 0 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e5c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e5c7c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e5c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient(hunting): handle_auth_done global_id 84633 payload 824 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e0139b10 msgr2=0x7f02e01d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f02e0139b10 0x7f02e01d19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02e0072600 msgr2=0x7f02e00a5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02e0072600 0x7f02e00a5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 monclient: found mon.noname-d 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 -- 172.21.15.98:0/3182605785 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e0122cd0 con 0x7f02e013ef60 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e647d640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02e013ef60 0x7f02e00a2d10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009fd0 tx=0x7f02d800e4b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 client.?.objecter ms_handle_connect 0x7f02e013ef60 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02d800edd0 con 0x7f02e013ef60 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:01.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: dump: 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.224 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 5 monclient: authenticate success, global_id 84633 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: _renew_subs 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02d8019070 con 0x7f02e013ef60 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e0122fb0 con 0x7f02e013ef60 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02d8014d00 con 0x7f02e013ef60 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 client.84633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d77fe640 4 set_mon_vals no callback set 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02d67fc640 10 monclient: dump: 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.226 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: _renew_subs 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e00a8420 con 0x7f02e013ef60 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 librados: init done 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e00a3250 0x7f02e00a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.203+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02e006ca10 con 0x7f02e00a3250 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e00a3250 0x7f02e00a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient: get_auth_request con 0x7f02e00a3250 auth_method 0 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f02d8014420 con 0x7f02e013ef60 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_done global_id 104112 payload 824 2024-09-11T03:08:01.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f02c403df40 0x7f02c4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 1 -- 172.21.15.98:0/3182605785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02e006e680 con 0x7f02e00a3250 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f02d804fcb0 con 0x7f02e013ef60 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 client.84633.objecter ms_dispatch 0x7f02e0000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 3 client.84633.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 3 client.84633.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 20 client.84633.objecter dump_active .. 0 homeless 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 client.84633.objecter ms_handle_connect 0x7f02e00a3250 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 client.84633.objecter resend_mon_ops 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e5c7c640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e00a3250 0x7f02e00a3630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e500 tx=0x7f02dc00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02dc0052e0 con 0x7f02e00a3250 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e547b640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f02c403df40 0x7f02c4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 monclient: dump: 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.230 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e547b640 10 monclient: get_auth_request con 0x7f02c403df40 auth_method 0 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f02dc018070 con 0x7f02e00a3250 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e00a3250 msgr2=0x7f02e00a3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e00a3250 0x7f02e00a3630 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e500 tx=0x7f02dc00e9c0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e00a3250 msgr2=0x7f02e00a3630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e00a3250 0x7f02e00a3630 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e500 tx=0x7f02dc00e9c0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02d67fc640 10 client.84633.objecter ms_handle_connect 0x7f02c403df40 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.207+0000 7f02e547b640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f02c403df40 0x7f02c4040400 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7f02d00af000 tx=0x7f02d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e7ee0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e7ee0640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e013c7e0 0x7f02e013cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e7ee0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f02e006e870 con 0x7f02e013c7e0 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e013c7e0 0x7f02e013cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient: get_auth_request con 0x7f02e013c7e0 auth_method 0 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 10 monclient(hunting): handle_auth_done global_id 104121 payload 824 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 1 -- 172.21.15.98:0/3182605785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f02e00a3b70 con 0x7f02e013c7e0 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02e5c7c640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e013c7e0 0x7f02e013cbf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00ea00 tx=0x7f02dc000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02d67fc640 10 client.84633.objecter ms_handle_connect 0x7f02e013c7e0 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02d67fc640 10 client.84633.objecter resend_mon_ops 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02dc0076d0 con 0x7f02e013c7e0 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.211+0000 7f02d67fc640 10 monclient: dump: 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.233 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f02dc007260 con 0x7f02e013c7e0 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02d67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:01.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02d67fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e013c7e0 msgr2=0x7f02e013cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02d67fc640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e013c7e0 0x7f02e013cbf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00ea00 tx=0x7f02dc000ee0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02d67fc640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e013c7e0 msgr2=0x7f02e013cbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02d67fc640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f02e013c7e0 0x7f02e013cbf0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00ea00 tx=0x7f02dc000ee0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:01.235 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 27691, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.236 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.237 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.238 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.240 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:01.241 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.242 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.243 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.244 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.247 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:01.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.249 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.250 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.251 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.252 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.254 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:01.255 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.257 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:01.258 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 10 librados: watch_flush enter 2024-09-11T03:08:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 10 librados: watch_flush exit 2024-09-11T03:08:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 20 client.84633.objecter shutdown clearing up homeless session... 2024-09-11T03:08:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 10 client.84633.objecter successfully canceled tick 2024-09-11T03:08:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f02c403df40 msgr2=0x7f02c4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f02c403df40 0x7f02c4040400 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7f02d00af000 tx=0x7f02d0009040 comp rx=0 tx=0).stop 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 10 monclient: shutdown 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02e013ef60 msgr2=0x7f02e00a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 --2- 172.21.15.98:0/3182605785 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f02e013ef60 0x7f02e00a2d10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009fd0 tx=0x7f02d800e4b0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e647d640 1 -- 172.21.15.98:0/3182605785 reap_dead start 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e647d640 1 -- 172.21.15.98:0/3182605785 reap_dead start 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 shutdown_connections 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 >> 172.21.15.98:0/3182605785 conn(0x7f02e00ac790 msgr2=0x7f02e00a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 shutdown_connections 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 -- 172.21.15.98:0/3182605785 wait complete. 2024-09-11T03:08:01.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.215+0000 7f02e7ee0640 1 librados: shutdown 2024-09-11T03:08:01.262 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:01.262 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-09-11T03:08:01.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient: get_monmap_and_config 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient: monmap: 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:01.385643+0000 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:01.385643+0000 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.383 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:01.384 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 1 Processor -- start 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 1 -- start start 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient: init 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient: _add_conns ranks=[8,10,0,2,7,9,5,4,1,3,6] 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63e80a7aa0 0x7f63e80a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient(hunting): picked mon.noname-e con 0x7f63e80a7aa0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f63e80a12a0 0x7f63e80a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient(hunting): picked mon.noname-f con 0x7f63e80a12a0 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63e8139850 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:01.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63e80a8900 con 0x7f63e8139850 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f63e80a2b30 con 0x7f63e80a12a0 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f63e80a1720 con 0x7f63e80a7aa0 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63eee7a640 10 monclient(hunting): authenticate will time out at 2035.266453s 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63e80a7aa0 0x7f63e80a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63e80a7aa0 0x7f63e80a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49454/0 (socket says 172.21.15.98:49454) 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 -- 172.21.15.98:0/3813330270 learned_addr learned my addr 172.21.15.98:0/3813330270 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): get_auth_request con 0x7f63e80a7aa0 auth_method 0 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63dffff640 1 --2- 172.21.15.98:0/3813330270 >> v2:172.21.15.190:3300/0 conn(0x7f63e80a12a0 0x7f63e80a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63dffff640 10 monclient(hunting): get_auth_request con 0x7f63e80a12a0 auth_method 0 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient(hunting): handle_auth_done global_id 25442 payload 274 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 -- 172.21.15.98:0/3813330270 >> v2:172.21.15.190:3300/0 conn(0x7f63e80a12a0 msgr2=0x7f63e80a83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63df7fe640 1 -- 172.21.15.98:0/3813330270 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1761237542 0 0) 0x7f63e00014a0 con 0x7f63e8139850 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 --2- 172.21.15.98:0/3813330270 >> v2:172.21.15.190:3300/0 conn(0x7f63e80a12a0 0x7f63e80a83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 -- 172.21.15.98:0/3813330270 >> v1:172.21.15.98:6789/0 conn(0x7f63e8139850 legacy=0x7f63e8139cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 monclient: found mon.noname-e 2024-09-11T03:08:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63ecc16640 1 -- 172.21.15.98:0/3813330270 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e80a27c0 con 0x7f63e80a7aa0 2024-09-11T03:08:01.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.383+0000 7f63df7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:01.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ecc16640 1 --2- 172.21.15.98:0/3813330270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63e80a7aa0 0x7f63e80a7e80 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009870 tx=0x7f63e401c3c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f30c398f8300dd6c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 1 -- 172.21.15.98:0/3813330270 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63e40b0070 con 0x7f63e80a7aa0 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: dump: 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.397 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 1 -- 172.21.15.98:0/3813330270 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e40aa810 con 0x7f63e80a7aa0 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 5 monclient: authenticate success, global_id 25442 2024-09-11T03:08:01.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 1 -- 172.21.15.98:0/3813330270 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63e40a9300 con 0x7f63e80a7aa0 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 4 set_mon_vals no callback set 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63df7fe640 10 monclient: dump: 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.399 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient: shutdown 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- 172.21.15.98:0/3813330270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63e80a7aa0 msgr2=0x7f63e80a7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 --2- 172.21.15.98:0/3813330270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63e80a7aa0 0x7f63e80a7e80 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009870 tx=0x7f63e401c3c0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- 172.21.15.98:0/3813330270 shutdown_connections 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 --2- 172.21.15.98:0/3813330270 >> v2:172.21.15.190:3300/0 conn(0x7f63e80a12a0 0x7f63e80a83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 --2- 172.21.15.98:0/3813330270 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f63e80a7aa0 0x7f63e80a7e80 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- 172.21.15.98:0/3813330270 >> 172.21.15.98:0/3813330270 conn(0x7f63e80ac690 msgr2=0x7f63e80a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- 172.21.15.98:0/3813330270 shutdown_connections 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- 172.21.15.98:0/3813330270 wait complete. 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient: monmap: 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:01.401 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 librados: starting msgr at 2024-09-11T03:08:01.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 librados: starting objecter 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 Processor -- start 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- start start 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 librados: setting wanted keys 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 librados: calling monclient init 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient: init 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient: _add_conns ranks=[5,1,2,4,7,3,6,8,0] 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f63e80a12a0 0x7f63e81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): picked mon.noname-h con 0x7f63e80a12a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63e80a7aa0 0x7f63e81386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): picked mon.noname-f con 0x7f63e80a7aa0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:01.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e8139850 0x7f63e813d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): picked mon.noname-b con 0x7f63e8139850 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63e8144450 con 0x7f63e80a7aa0 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f63e81442d0 con 0x7f63e8139850 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f63e81445d0 con 0x7f63e80a12a0 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63eee7a640 10 monclient(hunting): authenticate will time out at 2035.271644s 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e8139850 0x7f63e813d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e8139850 0x7f63e813d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60376/0 (socket says 172.21.15.98:60376) 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 1 -- 172.21.15.98:0/594856755 learned_addr learned my addr 172.21.15.98:0/594856755 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 10 monclient(hunting): get_auth_request con 0x7f63e8139850 auth_method 0 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63ed417640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.387+0000 7f63dffff640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63e80a7aa0 0x7f63e81386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient(hunting): get_auth_request con 0x7f63e80a7aa0 auth_method 0 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f63e80a12a0 0x7f63e81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): get_auth_request con 0x7f63e80a12a0 auth_method 0 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 10 monclient(hunting): handle_auth_done global_id 94322 payload 824 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f63e80a12a0 msgr2=0x7f63e81381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f63e80a12a0 0x7f63e81381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63e80a7aa0 msgr2=0x7f63e81386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63e80a7aa0 0x7f63e81386e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 1 monclient: found mon.noname-b 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 1 -- 172.21.15.98:0/594856755 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63e8122e60 con 0x7f63e8139850 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ed417640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e8139850 0x7f63e813d850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c760 tx=0x7f63e000cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 client.?.objecter ms_handle_connect 0x7f63e8139850 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63e0003bb0 con 0x7f63e8139850 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: dump: 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.407 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.408 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e0003d70 con 0x7f63e8139850 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 5 monclient: authenticate success, global_id 94322 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 10 monclient: _renew_subs 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63e81228a0 con 0x7f63e8139850 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63e000a640 con 0x7f63e8139850 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:01.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: dump: 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.410 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63de7fc640 4 set_mon_vals no callback set 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 10 client.94322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 10 monclient: _renew_subs 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63e8122b80 con 0x7f63e8139850 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 1 librados: init done 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813dd90 0x7f63e813e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f63e806ca10 con 0x7f63e813dd90 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813dd90 0x7f63e813e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient: get_auth_request con 0x7f63e813dd90 auth_method 0 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f63e000a7e0 con 0x7f63e8139850 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f63d403df40 0x7f63d4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 10 monclient(hunting): handle_auth_done global_id 94331 payload 824 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 1 -- 172.21.15.98:0/594856755 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63e80a8f00 con 0x7f63e813dd90 2024-09-11T03:08:01.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f63e004f240 con 0x7f63e8139850 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 client.94322.objecter ms_dispatch 0x7f63e8000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 3 client.94322.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 3 client.94322.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 20 client.94322.objecter dump_active .. 0 homeless 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 client.94322.objecter ms_handle_connect 0x7f63e813dd90 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 client.94322.objecter resend_mon_ops 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63ecc16640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813dd90 0x7f63e813e170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f63e80a1ca0 tx=0x7f63e40191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f63d403df40 0x7f63d4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63e40b9070 con 0x7f63e813dd90 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:01.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: dump: 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.414 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 10 monclient: get_auth_request con 0x7f63d403df40 auth_method 0 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f63e40b1040 con 0x7f63e813dd90 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813dd90 msgr2=0x7f63e813e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813dd90 0x7f63e813e170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f63e80a1ca0 tx=0x7f63e40191c0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813dd90 msgr2=0x7f63e813e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813dd90 0x7f63e813e170 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f63e80a1ca0 tx=0x7f63e40191c0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dd7fa640 10 client.94322.objecter ms_handle_connect 0x7f63d403df40 2024-09-11T03:08:01.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.391+0000 7f63dffff640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f63d403df40 0x7f63d4040400 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 crypto rx=0x7f63e813ef30 tx=0x7f63d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813e6b0 0x7f63e8071710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f63e806e870 con 0x7f63e813e6b0 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813e6b0 0x7f63e8071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient: get_auth_request con 0x7f63e813e6b0 auth_method 0 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 10 monclient(hunting): handle_auth_done global_id 94340 payload 824 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 1 -- 172.21.15.98:0/594856755 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f63e8140ae0 con 0x7f63e813e6b0 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ecc16640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813e6b0 0x7f63e8071710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009870 tx=0x7f63e40a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 10 client.94322.objecter ms_handle_connect 0x7f63e813e6b0 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 10 client.94322.objecter resend_mon_ops 2024-09-11T03:08:01.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63e40b0050 con 0x7f63e813e6b0 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 10 monclient: dump: 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.419 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 <== mon.2 v2:172.21.15.98:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9602 (secure 0 0 0) 0x7f63e40b5400 con 0x7f63e813e6b0 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813e6b0 msgr2=0x7f63e8071710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813e6b0 0x7f63e8071710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009870 tx=0x7f63e40a8000 comp rx=0 tx=0).stop 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813e6b0 msgr2=0x7f63e8071710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63dd7fa640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e813e6b0 0x7f63e8071710 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f63e4009870 tx=0x7f63e40a8000 comp rx=0 tx=0).stop 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 179786, 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:01.421 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.422 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.423 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:01.424 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.425 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.426 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.427 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:01.429 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:01.430 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.432 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.433 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.434 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.435 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.436 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.437 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.438 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.439 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.440 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.442 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:08:01.443 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 4 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": [ 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:01.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.445 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:01.445 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:01.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 10 librados: watch_flush enter 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 10 librados: watch_flush exit 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 20 client.94322.objecter shutdown clearing up homeless session... 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 10 client.94322.objecter successfully canceled tick 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f63d403df40 msgr2=0x7f63d4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f63d403df40 0x7f63d4040400 secure :-1 s=READY pgs=3700 cs=0 l=1 rev1=1 crypto rx=0x7f63e813ef30 tx=0x7f63d8008040 comp rx=0 tx=0).stop 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 10 monclient: shutdown 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e8139850 msgr2=0x7f63e813d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e8139850 0x7f63e813d850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c760 tx=0x7f63e000cc20 comp rx=0 tx=0).stop 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63ed417640 1 -- 172.21.15.98:0/594856755 reap_dead start 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 shutdown_connections 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 --2- 172.21.15.98:0/594856755 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f63e8139850 0x7f63e813d850 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 >> 172.21.15.98:0/594856755 conn(0x7f63e80ac690 msgr2=0x7f63e80a1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 shutdown_connections 2024-09-11T03:08:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 -- 172.21.15.98:0/594856755 wait complete. 2024-09-11T03:08:01.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.399+0000 7f63eee7a640 1 librados: shutdown 2024-09-11T03:08:01.448 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:01.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-09-11T03:08:01.448 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-11T03:08:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient: get_monmap_and_config 2024-09-11T03:08:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient: monmap: 2024-09-11T03:08:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.567 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:01.569849+0000 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:01.569849+0000 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.568 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.569 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.569 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.569 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:01.569 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:01.569 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 Processor -- start 2024-09-11T03:08:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 -- start start 2024-09-11T03:08:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient: init 2024-09-11T03:08:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient: _add_conns ranks=[5,2,7,4,10,9,3,6,0,8,1] 2024-09-11T03:08:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67e40a7a10 0x7f67e40724c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): picked mon.noname-h con 0x7f67e40a7a10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67e40a7df0 0x7f67e40a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): picked mon.noname-b con 0x7f67e40a7df0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 0x7f67e4139c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): picked mon.noname-i con 0x7f67e41397c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f67e4073d30 con 0x7f67e40a7df0 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f67e4071ab0 con 0x7f67e40a7a10 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f67e4071210 con 0x7f67e41397c0 2024-09-11T03:08:01.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e8939640 10 monclient(hunting): authenticate will time out at 2035.450096s 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67e40a7df0 0x7f67e40a81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67e40a7df0 0x7f67e40a81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60392/0 (socket says 172.21.15.98:60392) 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 -- 172.21.15.98:0/2051172019 learned_addr learned my addr 172.21.15.98:0/2051172019 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): get_auth_request con 0x7f67e40a7df0 auth_method 0 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e2d9d640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 0x7f67e4139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e259c640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67e40a7a10 0x7f67e40724c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e259c640 10 monclient(hunting): get_auth_request con 0x7f67e40a7a10 auth_method 0 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e2d9d640 10 monclient(hunting): get_auth_request con 0x7f67e41397c0 auth_method 0 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_done global_id 94349 payload 274 2024-09-11T03:08:01.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 -- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 msgr2=0x7f67e4139c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 0x7f67e4139c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 -- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67e40a7a10 msgr2=0x7f67e40724c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67e40a7a10 0x7f67e40724c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 monclient: found mon.noname-b 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e1d9b640 1 -- 172.21.15.98:0/2051172019 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e40a98e0 con 0x7f67e40a7df0 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.567+0000 7f67e259c640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67e40a7a10 0x7f67e40724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67e40a7df0 0x7f67e40a81d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f67d40098a0 tx=0x7f67d401cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a60d169fb25ee01f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 1 -- 172.21.15.98:0/2051172019 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d40b2070 con 0x7f67e40a7df0 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: dump: 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.578 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 1 -- 172.21.15.98:0/2051172019 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d40ab7f0 con 0x7f67e40a7df0 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 1 -- 172.21.15.98:0/2051172019 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d40abe50 con 0x7f67e40a7df0 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e159a640 10 monclient: dump: 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.580 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.581 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 5 monclient: authenticate success, global_id 94349 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 4 set_mon_vals no callback set 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient: shutdown 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- 172.21.15.98:0/2051172019 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67e40a7df0 msgr2=0x7f67e40a81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67e40a7df0 0x7f67e40a81d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f67d40098a0 tx=0x7f67d401cea0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- 172.21.15.98:0/2051172019 shutdown_connections 2024-09-11T03:08:01.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 0x7f67e4139c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f67e40a7df0 0x7f67e40a81d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 --2- 172.21.15.98:0/2051172019 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f67e40a7a10 0x7f67e40724c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- 172.21.15.98:0/2051172019 >> 172.21.15.98:0/2051172019 conn(0x7f67e40a8b00 msgr2=0x7f67e40a8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- 172.21.15.98:0/2051172019 shutdown_connections 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- 172.21.15.98:0/2051172019 wait complete. 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient: monmap: 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.583 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.584 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 librados: starting msgr at 2024-09-11T03:08:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 librados: starting objecter 2024-09-11T03:08:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 Processor -- start 2024-09-11T03:08:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- start start 2024-09-11T03:08:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 librados: setting wanted keys 2024-09-11T03:08:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 librados: calling monclient init 2024-09-11T03:08:01.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient: init 2024-09-11T03:08:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient: _add_conns ranks=[6,8,7,5,4,1,0,2,3] 2024-09-11T03:08:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f67e40a7a10 0x7f67e41d17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): picked mon.noname-d con 0x7f67e40a7a10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f67e40a7df0 0x7f67e4138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): picked mon.noname-e con 0x7f67e40a7df0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 0x7f67e4138ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): picked mon.noname-i con 0x7f67e41397c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f67e40a8890 con 0x7f67e40a7a10 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f67e40a8710 con 0x7f67e41397c0 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f67e41434b0 con 0x7f67e40a7df0 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e8939640 10 monclient(hunting): authenticate will time out at 2035.455590s 2024-09-11T03:08:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f67e40a7df0 0x7f67e4138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f67e40a7df0 0x7f67e4138ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49466/0 (socket says 172.21.15.98:49466) 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 1 -- 172.21.15.98:0/3352286730 learned_addr learned my addr 172.21.15.98:0/3352286730 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 10 monclient(hunting): get_auth_request con 0x7f67e40a7df0 auth_method 0 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f67e40a7a10 0x7f67e41d17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 10 monclient(hunting): get_auth_request con 0x7f67e40a7a10 auth_method 0 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.571+0000 7f67e2d9d640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 0x7f67e4138ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e2d9d640 10 monclient(hunting): get_auth_request con 0x7f67e41397c0 auth_method 0 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 10 monclient(hunting): handle_auth_done global_id 25451 payload 824 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 msgr2=0x7f67e4138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f67e41397c0 0x7f67e4138ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f67e40a7a10 msgr2=0x7f67e41d17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f67e40a7a10 0x7f67e41d17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 1 monclient: found mon.noname-e 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 1 -- 172.21.15.98:0/3352286730 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e4140100 con 0x7f67e40a7df0 2024-09-11T03:08:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f67e40a7a10 0x7f67e41d17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e1d9b640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f67e40a7df0 0x7f67e4138ab0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f67d40b0040 tx=0x7f67d40191e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 client.?.objecter ms_handle_connect 0x7f67e40a7df0 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d40bd070 con 0x7f67e40a7df0 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: dump: 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d40abe10 con 0x7f67e40a7df0 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 5 monclient: authenticate success, global_id 25451 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 10 monclient: _renew_subs 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e4124550 con 0x7f67e40a7df0 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d40b2070 con 0x7f67e40a7df0 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: dump: 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e0d99640 4 set_mon_vals no callback set 2024-09-11T03:08:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 10 client.25451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 10 monclient: _renew_subs 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e413f820 con 0x7f67e40a7df0 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 1 librados: init done 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413dd30 0x7f67e413e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f67e406ca10 con 0x7f67e413dd30 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413dd30 0x7f67e413e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient: get_auth_request con 0x7f67e413dd30 auth_method 0 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f67d4004250 con 0x7f67e40a7df0 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 10 monclient(hunting): handle_auth_done global_id 24880 payload 824 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 1 -- 172.21.15.98:0/3352286730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67e413fcb0 con 0x7f67e413dd30 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f67d40e91c0 con 0x7f67e40a7df0 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 client.25451.objecter ms_dispatch 0x7f67e4000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:01.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 3 client.25451.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 3 client.25451.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 20 client.25451.objecter dump_active .. 0 homeless 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 client.25451.objecter ms_handle_connect 0x7f67e413dd30 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 client.25451.objecter resend_mon_ops 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e259c640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413dd30 0x7f67e413e110 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f67d80119d0 tx=0x7f67d8011e90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d8019070 con 0x7f67e413dd30 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: dump: 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.603 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e2d9d640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e2d9d640 10 monclient: get_auth_request con 0x7f67c403df40 auth_method 0 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f67d800fb60 con 0x7f67e413dd30 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413dd30 msgr2=0x7f67e413e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413dd30 0x7f67e413e110 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f67d80119d0 tx=0x7f67d8011e90 comp rx=0 tx=0).stop 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413dd30 msgr2=0x7f67e413e110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413dd30 0x7f67e413e110 secure :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f67d80119d0 tx=0x7f67d8011e90 comp rx=0 tx=0).stop 2024-09-11T03:08:01.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67d2ffd640 10 client.25451.objecter ms_handle_connect 0x7f67c403df40 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.575+0000 7f67e2d9d640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 secure :-1 s=READY pgs=3701 cs=0 l=1 rev1=1 crypto rx=0x7f67e413fc50 tx=0x7f67dc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413e650 0x7f67e40a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f67e41d2710 con 0x7f67e413e650 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413e650 0x7f67e40a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient: get_auth_request con 0x7f67e413e650 auth_method 0 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 10 monclient(hunting): handle_auth_done global_id 24889 payload 824 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 1 -- 172.21.15.98:0/3352286730 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f67e40a81d0 con 0x7f67e413e650 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 10 client.25451.objecter ms_handle_connect 0x7f67e413e650 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e259c640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413e650 0x7f67e40a3e80 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f67d80119d0 tx=0x7f67d8004740 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 10 client.25451.objecter resend_mon_ops 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67d800f040 con 0x7f67e413e650 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 10 monclient: dump: 2024-09-11T03:08:01.607 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.608 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 <== mon.4 v2:172.21.15.98:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9269 (secure 0 0 0) 0x7f67d80143e0 con 0x7f67e413e650 2024-09-11T03:08:01.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413e650 msgr2=0x7f67e40a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413e650 0x7f67e40a3e80 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f67d80119d0 tx=0x7f67d8004740 comp rx=0 tx=0).stop 2024-09-11T03:08:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413e650 msgr2=0x7f67e40a3e80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67d2ffd640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f67e413e650 0x7f67e40a3e80 secure :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f67d80119d0 tx=0x7f67d8004740 comp rx=0 tx=0).stop 2024-09-11T03:08:01.610 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:01.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 1097368, 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:01.611 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.612 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.613 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.614 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.615 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:01.616 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.617 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:01.618 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.619 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.620 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.621 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.623 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:01.624 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.625 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.626 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.627 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:01.628 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.629 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:01.630 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.631 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:01.632 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.633 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.634 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.635 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:01.636 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.637 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:01.638 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 10 librados: watch_flush enter 2024-09-11T03:08:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 10 librados: watch_flush exit 2024-09-11T03:08:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 20 client.25451.objecter shutdown clearing up homeless session... 2024-09-11T03:08:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 10 client.25451.objecter successfully canceled tick 2024-09-11T03:08:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 msgr2=0x7f67c4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f67c403df40 0x7f67c4040400 secure :-1 s=READY pgs=3701 cs=0 l=1 rev1=1 crypto rx=0x7f67e413fc50 tx=0x7f67dc00b040 comp rx=0 tx=0).stop 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 10 monclient: shutdown 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f67e40a7df0 msgr2=0x7f67e4138ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f67e40a7df0 0x7f67e4138ab0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f67d40b0040 tx=0x7f67d40191e0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e2d9d640 1 -- 172.21.15.98:0/3352286730 reap_dead start 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 shutdown_connections 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 --2- 172.21.15.98:0/3352286730 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f67e40a7df0 0x7f67e4138ab0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 >> 172.21.15.98:0/3352286730 conn(0x7f67e40a8b00 msgr2=0x7f67e40a9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 shutdown_connections 2024-09-11T03:08:01.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 -- 172.21.15.98:0/3352286730 wait complete. 2024-09-11T03:08:01.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.583+0000 7f67e8939640 1 librados: shutdown 2024-09-11T03:08:01.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:01.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-09-11T03:08:01.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-09-11T03:08:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient: get_monmap_and_config 2024-09-11T03:08:01.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient: monmap: 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:01.761454+0000 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:01.761454+0000 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.759 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:01.760 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 Processor -- start 2024-09-11T03:08:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 -- start start 2024-09-11T03:08:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient: init 2024-09-11T03:08:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient: _add_conns ranks=[8,4,10,0,1,5,7,3,2,6,9] 2024-09-11T03:08:01.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f064409b8c0 0x7f064409bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): picked mon.noname-e con 0x7f064409b8c0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f064409df80 0x7f06440a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f064409df80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:01.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f06440a4320 0x7f06440a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): picked mon.noname-f con 0x7f06440a4320 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f064409d2c0 con 0x7f06440a4320 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f064409d5a0 con 0x7f064409df80 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f064409d880 con 0x7f064409b8c0 2024-09-11T03:08:01.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0649bd4640 10 monclient(hunting): authenticate will time out at 2035.641757s 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f064409b8c0 0x7f064409bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f064409df80 0x7f06440a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f064409b8c0 0x7f064409bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49472/0 (socket says 172.21.15.98:49472) 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 10 monclient(hunting): get_auth_request con 0x7f064409df80 auth_method 0 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 -- 172.21.15.98:0/3056638984 learned_addr learned my addr 172.21.15.98:0/3056638984 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): get_auth_request con 0x7f064409b8c0 auth_method 0 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0643fff640 1 --2- 172.21.15.98:0/3056638984 >> v2:172.21.15.190:3300/0 conn(0x7f06440a4320 0x7f06440a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient(hunting): handle_auth_done global_id 25460 payload 274 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 -- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f064409df80 msgr2=0x7f06440a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 --2- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f064409df80 0x7f06440a3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 -- 172.21.15.98:0/3056638984 >> v2:172.21.15.190:3300/0 conn(0x7f06440a4320 msgr2=0x7f06440a47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:01.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 --2- 172.21.15.98:0/3056638984 >> v2:172.21.15.190:3300/0 conn(0x7f06440a4320 0x7f06440a47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 monclient: found mon.noname-e 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f06437fe640 1 -- 172.21.15.98:0/3056638984 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06440a5d10 con 0x7f064409b8c0 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0643fff640 10 monclient: get_auth_request con 0x7f06440a4320 auth_method 0 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0642ffd640 1 --2- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f064409df80 0x7f06440a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.759+0000 7f0643fff640 1 --2- 172.21.15.98:0/3056638984 >> v2:172.21.15.190:3300/0 conn(0x7f06440a4320 0x7f06440a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 --2- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f064409b8c0 0x7f064409bca0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f063c009e50 tx=0x7f063c01c8f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b2dd0f2c52aced06 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 1 -- 172.21.15.98:0/3056638984 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f063c0090d0 con 0x7f064409b8c0 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: dump: 2024-09-11T03:08:01.771 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.772 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 1 -- 172.21.15.98:0/3056638984 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f063c0a5a20 con 0x7f064409b8c0 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 1 -- 172.21.15.98:0/3056638984 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f063c0a5c10 con 0x7f064409b8c0 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06427fc640 10 monclient: dump: 2024-09-11T03:08:01.773 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 5 monclient: authenticate success, global_id 25460 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 4 set_mon_vals no callback set 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: shutdown 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f064409b8c0 msgr2=0x7f064409bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 --2- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f064409b8c0 0x7f064409bca0 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f063c009e50 tx=0x7f063c01c8f0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- 172.21.15.98:0/3056638984 shutdown_connections 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 --2- 172.21.15.98:0/3056638984 >> v2:172.21.15.190:3300/0 conn(0x7f06440a4320 0x7f06440a47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 --2- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f064409df80 0x7f06440a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 --2- 172.21.15.98:0/3056638984 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f064409b8c0 0x7f064409bca0 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- 172.21.15.98:0/3056638984 >> 172.21.15.98:0/3056638984 conn(0x7f06440a4ed0 msgr2=0x7f06440a52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- 172.21.15.98:0/3056638984 shutdown_connections 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- 172.21.15.98:0/3056638984 wait complete. 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: monmap: 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 librados: starting msgr at 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 librados: starting objecter 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 Processor -- start 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- start start 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 librados: setting wanted keys 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 librados: calling monclient init 2024-09-11T03:08:01.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: init 2024-09-11T03:08:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: _add_conns ranks=[6,0,7,8,4,3,2,5,1] 2024-09-11T03:08:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f064409b8c0 0x7f064413b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): picked mon.noname-d con 0x7f064409b8c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064409df80 0x7f064413bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): picked mon.noname-a con 0x7f064409df80 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f06440a4320 0x7f064413bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): picked mon.noname-i con 0x7f06440a4320 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0644142f30 con 0x7f064409df80 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0644142db0 con 0x7f064409b8c0 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f06441430b0 con 0x7f06440a4320 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient(hunting): authenticate will time out at 2035.646637s 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f064409b8c0 0x7f064413b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064409df80 0x7f064413bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f064409b8c0 0x7f064413b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54448/0 (socket says 172.21.15.98:54448) 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 -- 172.21.15.98:0/2302968009 learned_addr learned my addr 172.21.15.98:0/2302968009 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): get_auth_request con 0x7f064409b8c0 auth_method 0 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 10 monclient(hunting): get_auth_request con 0x7f064409df80 auth_method 0 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient(hunting): handle_auth_done global_id 84651 payload 824 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f06440a4320 msgr2=0x7f064413bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f06440a4320 0x7f064413bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064409df80 msgr2=0x7f064413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f064409df80 0x7f064413bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 monclient: found mon.noname-d 2024-09-11T03:08:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 -- 172.21.15.98:0/2302968009 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064413c610 con 0x7f064409b8c0 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f06437fe640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f064409b8c0 0x7f064413b570 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f063c01d190 tx=0x7f063c019690 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 client.?.objecter ms_handle_connect 0x7f064409b8c0 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f063c0bc020 con 0x7f064409b8c0 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: dump: 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.787 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f063c003e10 con 0x7f064409b8c0 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f063c0a57c0 con 0x7f064409b8c0 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 5 monclient: authenticate success, global_id 84651 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: _renew_subs 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0644094e80 con 0x7f064409b8c0 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0641ffb640 4 set_mon_vals no callback set 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:01.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f063bfff640 10 monclient: dump: 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.790 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 client.84651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: _renew_subs 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0644093680 con 0x7f064409b8c0 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.763+0000 7f0649bd4640 1 librados: init done 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0649bd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0649bd4640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f06440923f0 0x7f06440927d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0644094820 con 0x7f06440923f0 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f06440923f0 0x7f06440927d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient: get_auth_request con 0x7f06440923f0 auth_method 0 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f063c0039a0 con 0x7f064409b8c0 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 10 monclient(hunting): handle_auth_done global_id 84660 payload 824 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f062403df40 0x7f0624040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 1 -- 172.21.15.98:0/2302968009 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06440a3620 con 0x7f06440923f0 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f063c0ea510 con 0x7f064409b8c0 2024-09-11T03:08:01.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 client.84651.objecter ms_dispatch 0x7f0644000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 3 client.84651.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 3 client.84651.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 20 client.84651.objecter dump_active .. 0 homeless 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0642ffd640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f06440923f0 0x7f06440927d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f06440734b0 tx=0x7f062c011d80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 client.84651.objecter ms_handle_connect 0x7f06440923f0 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 client.84651.objecter resend_mon_ops 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f062c014440 con 0x7f06440923f0 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 monclient: dump: 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.793 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.794 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0643fff640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f062403df40 0x7f0624040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0643fff640 10 monclient: get_auth_request con 0x7f062403df40 auth_method 0 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f062c00fb30 con 0x7f06440923f0 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f06440923f0 msgr2=0x7f06440927d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f06440923f0 0x7f06440927d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f06440734b0 tx=0x7f062c011d80 comp rx=0 tx=0).stop 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f06440923f0 msgr2=0x7f06440927d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f06440923f0 0x7f06440927d0 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f06440734b0 tx=0x7f062c011d80 comp rx=0 tx=0).stop 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f063bfff640 10 client.84651.objecter ms_handle_connect 0x7f062403df40 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.767+0000 7f0643fff640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f062403df40 0x7f0624040400 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7f063400a790 tx=0x7f0634008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0649bd4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:01.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0649bd4640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0644092d10 0x7f06440707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0649bd4640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f064406ca10 con 0x7f0644092d10 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0644092d10 0x7f06440707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient: get_auth_request con 0x7f0644092d10 auth_method 0 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 10 monclient(hunting): handle_auth_done global_id 84669 payload 824 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.771+0000 7f0642ffd640 1 -- 172.21.15.98:0/2302968009 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06440a48d0 con 0x7f0644092d10 2024-09-11T03:08:01.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0642ffd640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0644092d10 0x7f06440707b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f062c01c000 tx=0x7f062c0046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 10 client.84651.objecter ms_handle_connect 0x7f0644092d10 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 10 client.84651.objecter resend_mon_ops 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f062c004490 con 0x7f0644092d10 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 10 monclient: dump: 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.797 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 <== mon.6 v2:172.21.15.98:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7f062c01d400 con 0x7f0644092d10 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0644092d10 msgr2=0x7f06440707b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0644092d10 0x7f06440707b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f062c01c000 tx=0x7f062c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0644092d10 msgr2=0x7f06440707b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f063bfff640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0644092d10 0x7f06440707b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 307588, 2024-09-11T03:08:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:08:01.800 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.801 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.802 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.804 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:01.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:01.806 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:01.807 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.810 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.812 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.813 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.814 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:01.815 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:01.818 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:01.821 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 2 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:01.822 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:01.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 10 librados: watch_flush enter 2024-09-11T03:08:01.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 10 librados: watch_flush exit 2024-09-11T03:08:01.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 20 client.84651.objecter shutdown clearing up homeless session... 2024-09-11T03:08:01.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 10 client.84651.objecter successfully canceled tick 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f062403df40 msgr2=0x7f0624040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f062403df40 0x7f0624040400 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7f063400a790 tx=0x7f0634008040 comp rx=0 tx=0).stop 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 10 monclient: shutdown 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f064409b8c0 msgr2=0x7f064413b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 --2- 172.21.15.98:0/2302968009 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f064409b8c0 0x7f064413b570 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f063c01d190 tx=0x7f063c019690 comp rx=0 tx=0).stop 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0643fff640 1 -- 172.21.15.98:0/2302968009 reap_dead start 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0643fff640 1 -- 172.21.15.98:0/2302968009 reap_dead start 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 shutdown_connections 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 >> 172.21.15.98:0/2302968009 conn(0x7f06440a4ed0 msgr2=0x7f06440a1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 shutdown_connections 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 -- 172.21.15.98:0/2302968009 wait complete. 2024-09-11T03:08:01.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.775+0000 7f0649bd4640 1 librados: shutdown 2024-09-11T03:08:01.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:01.826 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-09-11T03:08:01.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient: get_monmap_and_config 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient: monmap: 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:01.945869+0000 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:01.945869+0000 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.943 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:01.944 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 1 Processor -- start 2024-09-11T03:08:01.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 1 -- start start 2024-09-11T03:08:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient: init 2024-09-11T03:08:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient: _add_conns ranks=[0,6,7,2,10,5,8,9,1,4,3] 2024-09-11T03:08:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05e8139b80 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:01.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05e806fe50 0x7f05e813ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient(hunting): picked mon.noname-d con 0x7f05e806fe50 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05e8072640 0x7f05e8140f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient(hunting): picked mon.noname-i con 0x7f05e8072640 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05e8071370 con 0x7f05e8139b80 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f05e80719c0 con 0x7f05e806fe50 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f05e8071c10 con 0x7f05e8072640 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05eebbd640 10 monclient(hunting): authenticate will time out at 2035.826893s 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05ec959640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f05e8139b80 0x7f05e8139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39670/0 (socket says 172.21.15.98:39670) 2024-09-11T03:08:01.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05ec959640 1 -- 172.21.15.98:0/3164728238 learned_addr learned my addr 172.21.15.98:0/3164728238 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 1 --2- 172.21.15.98:0/3164728238 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05e806fe50 0x7f05e813ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): get_auth_request con 0x7f05e806fe50 auth_method 0 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 1 -- 172.21.15.98:0/3164728238 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796718529 0 0) 0x7f05e8071370 con 0x7f05e8139b80 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 10 monclient(hunting): _negotiate 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 10 monclient(hunting): my global_id is 104148 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05ed15a640 1 --2- 172.21.15.98:0/3164728238 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05e8072640 0x7f05e8140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 1 -- 172.21.15.98:0/3164728238 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05d0003530 con 0x7f05e8139b80 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05ed15a640 10 monclient(hunting): get_auth_request con 0x7f05e8072640 auth_method 0 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05ed15a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05ed15a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05ed15a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient(hunting): handle_auth_done global_id 84678 payload 274 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 1 -- 172.21.15.98:0/3164728238 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05e8072640 msgr2=0x7f05e8140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 1 --2- 172.21.15.98:0/3164728238 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05e8072640 0x7f05e8140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 1 -- 172.21.15.98:0/3164728238 >> v1:172.21.15.98:6789/0 conn(0x7f05e8139b80 legacy=0x7f05e8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 1 -- 172.21.15.98:0/3164728238 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1480907890 0 0) 0x7f05d0003530 con 0x7f05e8139b80 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 1 monclient: found mon.noname-d 2024-09-11T03:08:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e7fff640 1 -- 172.21.15.98:0/3164728238 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e80a9600 con 0x7f05e806fe50 2024-09-11T03:08:01.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.943+0000 7f05e77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e7fff640 1 --2- 172.21.15.98:0/3164728238 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05e806fe50 0x7f05e813ea00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0097f0 tx=0x7f05dc01c2c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6070187cbcf074a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 1 -- 172.21.15.98:0/3164728238 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05dc0b0070 con 0x7f05e806fe50 2024-09-11T03:08:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: dump: 2024-09-11T03:08:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.957 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.958 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 1 -- 172.21.15.98:0/3164728238 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05dc0aa810 con 0x7f05e806fe50 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 1 -- 172.21.15.98:0/3164728238 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05d8001cf0 con 0x7f05e806fe50 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 5 monclient: authenticate success, global_id 84678 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 4 set_mon_vals no callback set 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05e77fe640 10 monclient: dump: 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.960 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 10 monclient: shutdown 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 -- 172.21.15.98:0/3164728238 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05e806fe50 msgr2=0x7f05e813ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3164728238 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05e806fe50 0x7f05e813ea00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0097f0 tx=0x7f05dc01c2c0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 -- 172.21.15.98:0/3164728238 shutdown_connections 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3164728238 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f05e8072640 0x7f05e8140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3164728238 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f05e806fe50 0x7f05e813ea00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 -- 172.21.15.98:0/3164728238 >> 172.21.15.98:0/3164728238 conn(0x7f05e80ac730 msgr2=0x7f05e80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 -- 172.21.15.98:0/3164728238 shutdown_connections 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 -- 172.21.15.98:0/3164728238 wait complete. 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 10 monclient: build_initial_monmap 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 10 monclient: monmap: 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:01.962 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 librados: starting msgr at 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 librados: starting objecter 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 Processor -- start 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 -- start start 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 librados: setting wanted keys 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 1 librados: calling monclient init 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.947+0000 7f05eebbd640 10 monclient: init 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient: _add_conns ranks=[2,0,4,3,8,5,1,7,6] 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e806fe50 0x7f05e81d1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): picked mon.noname-b con 0x7f05e806fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e8072640 0x7f05e81d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): picked mon.noname-a con 0x7f05e8072640 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05e8139b80 0x7f05e813d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): picked mon.noname-c con 0x7f05e8139b80 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f05e8144040 con 0x7f05e8072640 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f05e8143ec0 con 0x7f05e806fe50 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f05e81441c0 con 0x7f05e8139b80 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): authenticate will time out at 2035.832134s 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e806fe50 0x7f05e81d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e8072640 0x7f05e81d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e806fe50 0x7f05e81d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60408/0 (socket says 172.21.15.98:60408) 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05e8139b80 0x7f05e813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e8072640 0x7f05e81d18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54014/0 (socket says 172.21.15.98:54014) 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 1 -- 172.21.15.98:0/3483328556 learned_addr learned my addr 172.21.15.98:0/3483328556 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): get_auth_request con 0x7f05e806fe50 auth_method 0 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient(hunting): get_auth_request con 0x7f05e8072640 auth_method 0 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): get_auth_request con 0x7f05e8139b80 auth_method 0 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient(hunting): handle_auth_done global_id 24907 payload 824 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e806fe50 msgr2=0x7f05e81d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e806fe50 0x7f05e81d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e8072640 msgr2=0x7f05e81d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e8072640 0x7f05e81d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient: _finish_hunting 0 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 monclient: found mon.noname-c 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 -- 172.21.15.98:0/3483328556 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e80a8da0 con 0x7f05e8139b80 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05e8072640 0x7f05e81d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f05e806fe50 0x7f05e81d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:01.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ed15a640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05e8139b80 0x7f05e813d410 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f05e000d530 tx=0x7f05e000d9f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 client.?.objecter ms_handle_connect 0x7f05e8139b80 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05e0012a60 con 0x7f05e8139b80 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: dump: 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.969 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: _finish_auth 0 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: _check_auth_tickets 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:01.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e0019070 con 0x7f05e8139b80 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05e0012c60 con 0x7f05e8139b80 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 5 monclient: authenticate success, global_id 24907 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient: _renew_subs 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e6ffd640 4 set_mon_vals no callback set 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e80a8870 con 0x7f05e8139b80 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: dump: 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.971 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 client.24907.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient: _renew_subs 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e8122c10 con 0x7f05e8139b80 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 librados: init done 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813d950 0x7f05e813dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f05e813a420 con 0x7f05e813d950 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813d950 0x7f05e813dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f05e0013070 con 0x7f05e8139b80 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient: get_auth_request con 0x7f05e813d950 auth_method 0 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f05d403df40 0x7f05d4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f05e000f040 con 0x7f05e8139b80 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 client.24907.objecter ms_dispatch 0x7f05e8000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 3 client.24907.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 3 client.24907.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 10 monclient(hunting): handle_auth_done global_id 25469 payload 824 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 1 -- 172.21.15.98:0/3483328556 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05e80a6c90 con 0x7f05e813d950 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 20 client.24907.objecter dump_active .. 0 homeless 2024-09-11T03:08:01.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 client.24907.objecter ms_handle_connect 0x7f05e813d950 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05ec959640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813d950 0x7f05e813dd30 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f05d800f430 tx=0x7f05d800f8f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 client.24907.objecter resend_mon_ops 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f05d403df40 0x7f05d4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 10 monclient: get_auth_request con 0x7f05d403df40 auth_method 0 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05d8007750 con 0x7f05e813d950 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:01.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: dump: 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.977 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f05d8018070 con 0x7f05e813d950 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813d950 msgr2=0x7f05e813dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813d950 0x7f05e813dd30 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f05d800f430 tx=0x7f05d800f8f0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813d950 msgr2=0x7f05e813dd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813d950 0x7f05e813dd30 secure :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f05d800f430 tx=0x7f05d800f8f0 comp rx=0 tx=0).stop 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e57fa640 10 client.24907.objecter ms_handle_connect 0x7f05d403df40 2024-09-11T03:08:01.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.951+0000 7f05e7fff640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f05d403df40 0x7f05d4040400 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0ad000 tx=0x7f05dc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05eebbd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813e270 0x7f05e80a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05eebbd640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:01.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f05e806ca10 con 0x7f05e813e270 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813e270 0x7f05e80a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient: get_auth_request con 0x7f05e813e270 auth_method 0 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 10 monclient(hunting): handle_auth_done global_id 25478 payload 824 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 1 -- 172.21.15.98:0/3483328556 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05e8139f60 con 0x7f05e813e270 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05e57fa640 10 client.24907.objecter ms_handle_connect 0x7f05e813e270 2024-09-11T03:08:01.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05e57fa640 10 client.24907.objecter resend_mon_ops 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.959+0000 7f05ec959640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813e270 0x7f05e80a4300 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f05d800f930 tx=0x7f05d8001260 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05d80079b0 con 0x7f05e813e270 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 10 monclient: dump: 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:01.982 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 <== mon.8 v2:172.21.15.98:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9436 (secure 0 0 0) 0x7f05d80079b0 con 0x7f05e813e270 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813e270 msgr2=0x7f05e80a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813e270 0x7f05e80a4300 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f05d800f930 tx=0x7f05d8001260 comp rx=0 tx=0).stop 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813e270 msgr2=0x7f05e80a4300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:01.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05e57fa640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f05e813e270 0x7f05e80a4300 secure :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f05d800f930 tx=0x7f05d8001260 comp rx=0 tx=0).stop 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 1097756, 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:01.984 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 2, 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.985 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.987 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:01.988 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:01.989 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.991 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.992 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.993 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.994 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.996 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:01.999 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.000 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.002 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.003 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.004 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.005 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:08:02.006 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.007 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:02.008 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 10 librados: watch_flush enter 2024-09-11T03:08:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 10 librados: watch_flush exit 2024-09-11T03:08:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 20 client.24907.objecter shutdown clearing up homeless session... 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 10 client.24907.objecter successfully canceled tick 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f05d403df40 msgr2=0x7f05d4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f05d403df40 0x7f05d4040400 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0ad000 tx=0x7f05dc0a6000 comp rx=0 tx=0).stop 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 10 monclient: shutdown 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05e8139b80 msgr2=0x7f05e813d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05e8139b80 0x7f05e813d410 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f05e000d530 tx=0x7f05e000d9f0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05ed15a640 1 -- 172.21.15.98:0/3483328556 reap_dead start 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 shutdown_connections 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 --2- 172.21.15.98:0/3483328556 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f05e8139b80 0x7f05e813d410 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 >> 172.21.15.98:0/3483328556 conn(0x7f05e80ac730 msgr2=0x7f05e80aad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 shutdown_connections 2024-09-11T03:08:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 -- 172.21.15.98:0/3483328556 wait complete. 2024-09-11T03:08:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:01.963+0000 7f05eebbd640 1 librados: shutdown 2024-09-11T03:08:02.011 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:02.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-09-11T03:08:02.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.127+0000 7fd15a9aa640 10 monclient: get_monmap_and_config 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.127+0000 7fd15a9aa640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.127+0000 7fd15a9aa640 10 monclient: monmap: 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:02.131916+0000 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:02.131916+0000 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:02.130 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.127+0000 7fd15a9aa640 1 Processor -- start 2024-09-11T03:08:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 1 -- start start 2024-09-11T03:08:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient: init 2024-09-11T03:08:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient: _add_conns ranks=[10,3,2,7,8,4,6,1,5,0,9] 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fd154139b10 0x7fd154072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): picked mon.noname-f con 0x7fd154139b10 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd154139ef0 0x7fd1540a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): picked mon.noname-g con 0x7fd154139ef0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd15413a2d0 0x7fd1540a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): picked mon.noname-b con 0x7fd15413a2d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fd1540702b0 con 0x7fd154139b10 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fd154071bf0 con 0x7fd15413a2d0 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fd154071350 con 0x7fd154139ef0 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient(hunting): authenticate will time out at 2036.012788s 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd15413a2d0 0x7fd1540a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd15413a2d0 0x7fd1540a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60412/0 (socket says 172.21.15.98:60412) 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 -- 172.21.15.98:0/936331517 learned_addr learned my addr 172.21.15.98:0/936331517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): get_auth_request con 0x7fd15413a2d0 auth_method 0 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd14b7fe640 1 --2- 172.21.15.98:0/936331517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd154139ef0 0x7fd1540a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd153fff640 1 --2- 172.21.15.98:0/936331517 >> v2:172.21.15.190:3300/0 conn(0x7fd154139b10 0x7fd154072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient(hunting): handle_auth_done global_id 94367 payload 274 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 -- 172.21.15.98:0/936331517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd154139ef0 msgr2=0x7fd1540a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 --2- 172.21.15.98:0/936331517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd154139ef0 0x7fd1540a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 -- 172.21.15.98:0/936331517 >> v2:172.21.15.190:3300/0 conn(0x7fd154139b10 msgr2=0x7fd154072600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 --2- 172.21.15.98:0/936331517 >> v2:172.21.15.190:3300/0 conn(0x7fd154139b10 0x7fd154072600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 monclient: found mon.noname-b 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 -- 172.21.15.98:0/936331517 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1540a9620 con 0x7fd15413a2d0 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd14b7fe640 10 monclient: get_auth_request con 0x7fd154139ef0 auth_method 0 2024-09-11T03:08:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd153fff640 10 monclient: get_auth_request con 0x7fd154139b10 auth_method 0 2024-09-11T03:08:02.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd14b7fe640 1 --2- 172.21.15.98:0/936331517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd154139ef0 0x7fd1540a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:02.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd153fff640 1 --2- 172.21.15.98:0/936331517 >> v2:172.21.15.190:3300/0 conn(0x7fd154139b10 0x7fd154072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd158f47640 1 --2- 172.21.15.98:0/936331517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd15413a2d0 0x7fd1540a7f50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0098a0 tx=0x7fd14c01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7533e1266d378918 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 1 -- 172.21.15.98:0/936331517 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd14c0b0070 con 0x7fd15413a2d0 2024-09-11T03:08:02.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: dump: 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.143 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 1 -- 172.21.15.98:0/936331517 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd14c0aa810 con 0x7fd15413a2d0 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 5 monclient: authenticate success, global_id 94367 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 1 -- 172.21.15.98:0/936331517 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd14c0a9300 con 0x7fd15413a2d0 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd1537fe640 10 monclient: dump: 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.145 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.131+0000 7fd15a9aa640 4 set_mon_vals no callback set 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: shutdown 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/936331517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd15413a2d0 msgr2=0x7fd1540a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/936331517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd15413a2d0 0x7fd1540a7f50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0098a0 tx=0x7fd14c01c490 comp rx=0 tx=0).stop 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/936331517 shutdown_connections 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/936331517 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fd15413a2d0 0x7fd1540a7f50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/936331517 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fd154139ef0 0x7fd1540a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/936331517 >> v2:172.21.15.190:3300/0 conn(0x7fd154139b10 0x7fd154072600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/936331517 >> 172.21.15.98:0/936331517 conn(0x7fd1540ac790 msgr2=0x7fd1540a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/936331517 shutdown_connections 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/936331517 wait complete. 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: monmap: 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:02.147 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 librados: starting msgr at 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 librados: starting objecter 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 Processor -- start 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- start start 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 librados: setting wanted keys 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 librados: calling monclient init 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: init 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: _add_conns ranks=[6,0,5,7,3,2,8,4,1] 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd154139b10 0x7fd154134e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): picked mon.noname-d con 0x7fd154139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd154139ef0 0x7fd15413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): picked mon.noname-a con 0x7fd154139ef0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd15413a2d0 0x7fd15413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): picked mon.noname-h con 0x7fd15413a2d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd154143460 con 0x7fd154139ef0 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fd1540a88a0 con 0x7fd15413a2d0 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fd1541435e0 con 0x7fd154139b10 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): authenticate will time out at 2036.018086s 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd154139b10 0x7fd154134e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd154139ef0 0x7fd15413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd154139b10 0x7fd154134e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54476/0 (socket says 172.21.15.98:54476) 2024-09-11T03:08:02.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 -- 172.21.15.98:0/2732652949 learned_addr learned my addr 172.21.15.98:0/2732652949 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): get_auth_request con 0x7fd154139b10 auth_method 0 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 10 monclient(hunting): get_auth_request con 0x7fd154139ef0 auth_method 0 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd158f47640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd15413a2d0 0x7fd15413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient(hunting): handle_auth_done global_id 84687 payload 824 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd15413a2d0 msgr2=0x7fd15413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd15413a2d0 0x7fd15413ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd154139ef0 msgr2=0x7fd15413a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd154139ef0 0x7fd15413a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 monclient: found mon.noname-d 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 -- 172.21.15.98:0/2732652949 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1541358a0 con 0x7fd154139b10 2024-09-11T03:08:02.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd158f47640 10 monclient: get_auth_request con 0x7fd15413a2d0 auth_method 0 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd14b7fe640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd154139ef0 0x7fd15413a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd158f47640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fd15413a2d0 0x7fd15413ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd153fff640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd154139b10 0x7fd154134e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd14400cf90 tx=0x7fd14400a7d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 client.?.objecter ms_handle_connect 0x7fd154139b10 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd14401a070 con 0x7fd154139b10 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: dump: 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.153 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd14400dab0 con 0x7fd154139b10 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 5 monclient: authenticate success, global_id 84687 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: _renew_subs 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1540aa3c0 con 0x7fd154139b10 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd14400dc60 con 0x7fd154139b10 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1527fc640 4 set_mon_vals no callback set 2024-09-11T03:08:02.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 client.84687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 10 monclient: dump: 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.156 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: _renew_subs 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fd15413dd40 con 0x7fd154139b10 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 librados: init done 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154135360 0x7fd15413b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd15413d830 con 0x7fd154135360 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.135+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fd14400d640 con 0x7fd154139b10 2024-09-11T03:08:02.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd13403df40 0x7fd134040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154135360 0x7fd15413b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fd14404fc90 con 0x7fd154139b10 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 client.84687.objecter ms_dispatch 0x7fd154000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 3 client.84687.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 3 client.84687.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient: get_auth_request con 0x7fd154135360 auth_method 0 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 20 client.84687.objecter dump_active .. 0 homeless 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd158f47640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd13403df40 0x7fd134040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd158f47640 10 monclient: get_auth_request con 0x7fd13403df40 auth_method 0 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_done global_id 26326 payload 824 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 1 -- 172.21.15.98:0/2732652949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd15406e680 con 0x7fd154135360 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 client.84687.objecter ms_handle_connect 0x7fd154135360 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 client.84687.objecter resend_mon_ops 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd14b7fe640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154135360 0x7fd15413b160 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fd1540a5e80 tx=0x7fd13c00d590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 client.84687.objecter ms_handle_connect 0x7fd13403df40 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd158f47640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd13403df40 0x7fd134040400 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0ad000 tx=0x7fd14c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd13c00fc60 con 0x7fd154135360 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:08:02.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 monclient: dump: 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.160 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd13c00fc60 con 0x7fd154135360 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154135360 msgr2=0x7fd15413b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154135360 0x7fd15413b160 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fd1540a5e80 tx=0x7fd13c00d590 comp rx=0 tx=0).stop 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154135360 msgr2=0x7fd15413b160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.139+0000 7fd1517fa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd154135360 0x7fd15413b160 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.143+0000 7fd15a9aa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.143+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15413b6a0 0x7fd1540a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd15406ca10 con 0x7fd15413b6a0 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15413b6a0 0x7fd1540a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient: get_auth_request con 0x7fd15413b6a0 auth_method 0 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 10 monclient(hunting): handle_auth_done global_id 26335 payload 824 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 1 -- 172.21.15.98:0/2732652949 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1540aa5f0 con 0x7fd15413b6a0 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd14b7fe640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15413b6a0 0x7fd1540a0e60 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fd1540a5e80 tx=0x7fd13c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 10 client.84687.objecter ms_handle_connect 0x7fd15413b6a0 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 10 client.84687.objecter resend_mon_ops 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd13c020bf0 con 0x7fd15413b6a0 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 10 monclient: dump: 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.164 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9438 (secure 0 0 0) 0x7fd13c02a450 con 0x7fd15413b6a0 2024-09-11T03:08:02.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15413b6a0 msgr2=0x7fd1540a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15413b6a0 0x7fd1540a0e60 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fd1540a5e80 tx=0x7fd13c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15413b6a0 msgr2=0x7fd1540a0e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd1517fa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd15413b6a0 0x7fd1540a0e60 secure :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fd1540a5e80 tx=0x7fd13c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "leader", 2024-09-11T03:08:02.166 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 1097919, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.167 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.169 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.171 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:02.172 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:02.173 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.174 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.176 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.177 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:02.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.180 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.181 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.182 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:02.183 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.184 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.185 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:02.186 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.188 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": [ 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.189 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.190 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.190 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:02.190 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:02.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 10 librados: watch_flush enter 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 10 librados: watch_flush exit 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 20 client.84687.objecter shutdown clearing up homeless session... 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 10 client.84687.objecter successfully canceled tick 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd13403df40 msgr2=0x7fd134040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fd13403df40 0x7fd134040400 secure :-1 s=READY pgs=3704 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0ad000 tx=0x7fd14c0a6000 comp rx=0 tx=0).stop 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 10 monclient: shutdown 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd154139b10 msgr2=0x7fd154134e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 --2- 172.21.15.98:0/2732652949 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fd154139b10 0x7fd154134e20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd14400cf90 tx=0x7fd14400a7d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd158f47640 1 -- 172.21.15.98:0/2732652949 reap_dead start 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd158f47640 1 -- 172.21.15.98:0/2732652949 reap_dead start 2024-09-11T03:08:02.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 shutdown_connections 2024-09-11T03:08:02.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 >> 172.21.15.98:0/2732652949 conn(0x7fd1540ac790 msgr2=0x7fd154135ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 shutdown_connections 2024-09-11T03:08:02.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 -- 172.21.15.98:0/2732652949 wait complete. 2024-09-11T03:08:02.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.147+0000 7fd15a9aa640 1 librados: shutdown 2024-09-11T03:08:02.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:02.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-09-11T03:08:02.193 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient: get_monmap_and_config 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient: monmap: 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:02.307732+0000 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:02.307732+0000 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.305 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:02.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 1 Processor -- start 2024-09-11T03:08:02.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 1 -- start start 2024-09-11T03:08:02.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient: init 2024-09-11T03:08:02.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient: _add_conns ranks=[4,0,3,5,2,8,9,7,10,6,1] 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10b8139b10 0x7f10b8072600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient(hunting): picked mon.noname-c con 0x7f10b8139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10b8139ef0 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813a2d0 0x7f10b80a8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient(hunting): picked mon.noname-g con 0x7f10b813a2d0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b80702b0 con 0x7f10b8139ef0 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f10b8071bf0 con 0x7f10b813a2d0 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f10b8071350 con 0x7f10b8139b10 2024-09-11T03:08:02.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.303+0000 7f10c056f640 10 monclient(hunting): authenticate will time out at 2036.187985s 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10b8139b10 0x7f10b8072600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10b8139b10 0x7f10b8072600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57780/0 (socket says 172.21.15.98:57780) 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bdb0a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f10b8139ef0 0x7f10b80a7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39682/0 (socket says 172.21.15.98:39682) 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 -- 172.21.15.98:0/1374373741 learned_addr learned my addr 172.21.15.98:0/1374373741 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): get_auth_request con 0x7f10b8139b10 auth_method 0 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient(hunting): handle_auth_done global_id 24916 payload 274 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 -- 172.21.15.98:0/1374373741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813a2d0 msgr2=0x7f10b80a8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 --2- 172.21.15.98:0/1374373741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813a2d0 0x7f10b80a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 -- 172.21.15.98:0/1374373741 >> v1:172.21.15.98:6789/0 conn(0x7f10b8139ef0 legacy=0x7f10b80a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 1 -- 172.21.15.98:0/1374373741 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988050842 0 0) 0x7f10b4001120 con 0x7f10b8139ef0 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 monclient: found mon.noname-c 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 -- 172.21.15.98:0/1374373741 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b80a9620 con 0x7f10b8139b10 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10be30b640 1 --2- 172.21.15.98:0/1374373741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10b8139b10 0x7f10b8072600 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f10a80098d0 tx=0x7f10a801c4d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=290e3b3ea66ec8fa server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 1 -- 172.21.15.98:0/1374373741 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10a80b0070 con 0x7f10b8139b10 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: dump: 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.320 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 5 monclient: authenticate success, global_id 24916 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 1 -- 172.21.15.98:0/1374373741 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a80aa810 con 0x7f10b8139b10 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:02.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 1 -- 172.21.15.98:0/1374373741 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10a80a9300 con 0x7f10b8139b10 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 4 set_mon_vals no callback set 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10bd309640 10 monclient: dump: 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.322 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 10 monclient: shutdown 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 -- 172.21.15.98:0/1374373741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10b8139b10 msgr2=0x7f10b8072600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 --2- 172.21.15.98:0/1374373741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10b8139b10 0x7f10b8072600 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f10a80098d0 tx=0x7f10a801c4d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 -- 172.21.15.98:0/1374373741 shutdown_connections 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 --2- 172.21.15.98:0/1374373741 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813a2d0 0x7f10b80a8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 --2- 172.21.15.98:0/1374373741 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f10b8139b10 0x7f10b8072600 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 -- 172.21.15.98:0/1374373741 >> 172.21.15.98:0/1374373741 conn(0x7f10b80ac790 msgr2=0x7f10b80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 -- 172.21.15.98:0/1374373741 shutdown_connections 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 1 -- 172.21.15.98:0/1374373741 wait complete. 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.307+0000 7f10c056f640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: monmap: 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 librados: starting msgr at 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 librados: starting objecter 2024-09-11T03:08:02.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 Processor -- start 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 -- start start 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 librados: setting wanted keys 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 librados: calling monclient init 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: init 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: _add_conns ranks=[8,6,7,3,0,5,1,2,4] 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10b8139b10 0x7f10b80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): picked mon.noname-e con 0x7f10b8139b10 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f10b8139ef0 0x7f10b813ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): picked mon.noname-d con 0x7f10b8139ef0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f10b813a2d0 0x7f10b813b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): picked mon.noname-i con 0x7f10b813a2d0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:02.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f10b8144470 con 0x7f10b8139ef0 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f10b80a8870 con 0x7f10b813a2d0 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f10b81445f0 con 0x7f10b8139b10 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): authenticate will time out at 2036.192926s 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10b8139b10 0x7f10b80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f10b8139ef0 0x7f10b813ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10b8139b10 0x7f10b80a4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49504/0 (socket says 172.21.15.98:49504) 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 1 -- 172.21.15.98:0/2352484360 learned_addr learned my addr 172.21.15.98:0/2352484360 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): get_auth_request con 0x7f10b8139b10 auth_method 0 2024-09-11T03:08:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): get_auth_request con 0x7f10b8139ef0 auth_method 0 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient(hunting): handle_auth_done global_id 84696 payload 824 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10b8139b10 msgr2=0x7f10b80a4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10b8139b10 0x7f10b80a4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f10b813a2d0 msgr2=0x7f10b813b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f10b813a2d0 0x7f10b813b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 monclient: found mon.noname-d 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 -- 172.21.15.98:0/2352484360 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b80aa310 con 0x7f10b8139ef0 2024-09-11T03:08:02.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f10b8139b10 0x7f10b80a4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bdb0a640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f10b8139ef0 0x7f10b813ac90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f10b400c400 tx=0x7f10b400c8c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 client.?.objecter ms_handle_connect 0x7f10b8139ef0 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10b40047f0 con 0x7f10b8139ef0 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: dump: 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.330 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b40049b0 con 0x7f10b8139ef0 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10b40133c0 con 0x7f10b8139ef0 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 monclient: dump: 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10bcb08640 4 set_mon_vals no callback set 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 5 monclient: authenticate success, global_id 84696 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: _renew_subs 2024-09-11T03:08:02.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b813d080 con 0x7f10b8139ef0 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 client.84696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: _renew_subs 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b813bb60 con 0x7f10b8139ef0 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 librados: init done 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813b710 0x7f10b81d1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f10b806ca10 con 0x7f10b813b710 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f10b4004b60 con 0x7f10b8139ef0 2024-09-11T03:08:02.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f109c03df40 0x7f109c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f10b404f1a0 con 0x7f10b8139ef0 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813b710 0x7f10b81d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 client.84696.objecter ms_dispatch 0x7f10b8000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 3 client.84696.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 3 client.84696.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient: get_auth_request con 0x7f10b813b710 auth_method 0 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 20 client.84696.objecter dump_active .. 0 homeless 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10beb0c640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f109c03df40 0x7f109c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10beb0c640 10 monclient: get_auth_request con 0x7f109c03df40 auth_method 0 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 10 monclient(hunting): handle_auth_done global_id 104115 payload 824 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 1 -- 172.21.15.98:0/2352484360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10b80a9f00 con 0x7f10b813b710 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10be30b640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813b710 0x7f10b81d1570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f10a801cd70 tx=0x7f10a8019270 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 client.84696.objecter ms_handle_connect 0x7f10b813b710 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.311+0000 7f10aeffd640 10 client.84696.objecter resend_mon_ops 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 10 client.84696.objecter ms_handle_connect 0x7f109c03df40 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10beb0c640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f109c03df40 0x7f109c040400 secure :-1 s=READY pgs=3705 cs=0 l=1 rev1=1 crypto rx=0x7f10b80a5590 tx=0x7f10b0009250 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10a80b0040 con 0x7f10b813b710 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 10 monclient: dump: 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.337 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.338 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.338 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f10a80aa810 con 0x7f10b813b710 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813b710 msgr2=0x7f10b81d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813b710 0x7f10b81d1570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f10a801cd70 tx=0x7f10a8019270 comp rx=0 tx=0).stop 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813b710 msgr2=0x7f10b81d1570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.315+0000 7f10aeffd640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b813b710 0x7f10b81d1570 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f10a801cd70 tx=0x7f10a8019270 comp rx=0 tx=0).stop 2024-09-11T03:08:02.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10c056f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:02.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10c056f640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b81d1ab0 0x7f10b81d1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10c056f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f10b81d2400 con 0x7f10b81d1ab0 2024-09-11T03:08:02.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b81d1ab0 0x7f10b81d1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient: get_auth_request con 0x7f10b81d1ab0 auth_method 0 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 10 monclient(hunting): handle_auth_done global_id 104124 payload 824 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.319+0000 7f10be30b640 1 -- 172.21.15.98:0/2352484360 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10b80aa540 con 0x7f10b81d1ab0 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10be30b640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b81d1ab0 0x7f10b81d1ec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f10b8141880 tx=0x7f10a80a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 10 client.84696.objecter ms_handle_connect 0x7f10b81d1ab0 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 10 client.84696.objecter resend_mon_ops 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.3 v2:172.21.15.190:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10a80b0040 con 0x7f10b81d1ab0 2024-09-11T03:08:02.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 10 monclient: dump: 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.344 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 <== mon.3 v2:172.21.15.190:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f10a80c0e50 con 0x7f10b81d1ab0 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b81d1ab0 msgr2=0x7f10b81d1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b81d1ab0 0x7f10b81d1ec0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f10b8141880 tx=0x7f10a80a8000 comp rx=0 tx=0).stop 2024-09-11T03:08:02.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b81d1ab0 msgr2=0x7f10b81d1ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10aeffd640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f10b81d1ab0 0x7f10b81d1ec0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f10b8141880 tx=0x7f10a80a8000 comp rx=0 tx=0).stop 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 28759, 2024-09-11T03:08:02.346 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T03:08:02.347 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.348 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.349 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:02.350 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.351 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.352 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.353 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.355 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.358 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:02.361 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.363 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.367 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:02.368 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 10 librados: watch_flush enter 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 10 librados: watch_flush exit 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 20 client.84696.objecter shutdown clearing up homeless session... 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 10 client.84696.objecter successfully canceled tick 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f109c03df40 msgr2=0x7f109c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f109c03df40 0x7f109c040400 secure :-1 s=READY pgs=3705 cs=0 l=1 rev1=1 crypto rx=0x7f10b80a5590 tx=0x7f10b0009250 comp rx=0 tx=0).stop 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 10 monclient: shutdown 2024-09-11T03:08:02.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f10b8139ef0 msgr2=0x7f10b813ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 --2- 172.21.15.98:0/2352484360 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f10b8139ef0 0x7f10b813ac90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f10b400c400 tx=0x7f10b400c8c0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10beb0c640 1 -- 172.21.15.98:0/2352484360 reap_dead start 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10beb0c640 1 -- 172.21.15.98:0/2352484360 reap_dead start 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 shutdown_connections 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 >> 172.21.15.98:0/2352484360 conn(0x7f10b80ac790 msgr2=0x7f10b80a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 shutdown_connections 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 -- 172.21.15.98:0/2352484360 wait complete. 2024-09-11T03:08:02.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.323+0000 7f10c056f640 1 librados: shutdown 2024-09-11T03:08:02.371 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:02.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-09-11T03:08:02.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-09-11T03:08:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient: get_monmap_and_config 2024-09-11T03:08:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient: monmap: 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:02.486263+0000 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:02.486263+0000 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:02.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 Processor -- start 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 -- start start 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient: init 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient: _add_conns ranks=[8,10,6,1,4,3,7,5,0,2,9] 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd680a09d0 0x7fcd680a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): picked mon.noname-e con 0x7fcd680a09d0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fcd680a0200 0x7fcd681365c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): picked mon.noname-f con 0x7fcd680a0200 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd68136b00 0x7fcd68136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): picked mon.noname-d con 0x7fcd68136b00 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fcd6809ff40 con 0x7fcd680a0200 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcd680a50c0 con 0x7fcd68136b00 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fcd680a5340 con 0x7fcd680a09d0 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6e97f640 10 monclient(hunting): authenticate will time out at 2036.366416s 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd68136b00 0x7fcd68136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd680a09d0 0x7fcd680a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd68136b00 0x7fcd68136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54494/0 (socket says 172.21.15.98:54494) 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd680a09d0 0x7fcd680a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49508/0 (socket says 172.21.15.98:49508) 2024-09-11T03:08:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 -- 172.21.15.98:0/4207217530 learned_addr learned my addr 172.21.15.98:0/4207217530 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): get_auth_request con 0x7fcd68136b00 auth_method 0 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 10 monclient(hunting): get_auth_request con 0x7fcd680a09d0 auth_method 0 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd677fe640 1 --2- 172.21.15.98:0/4207217530 >> v2:172.21.15.190:3300/0 conn(0x7fcd680a0200 0x7fcd681365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd677fe640 10 monclient(hunting): get_auth_request con 0x7fcd680a0200 auth_method 0 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient(hunting): handle_auth_done global_id 84705 payload 274 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 -- 172.21.15.98:0/4207217530 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd680a09d0 msgr2=0x7fcd680a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 --2- 172.21.15.98:0/4207217530 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd680a09d0 0x7fcd680a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 -- 172.21.15.98:0/4207217530 >> v2:172.21.15.190:3300/0 conn(0x7fcd680a0200 msgr2=0x7fcd681365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 --2- 172.21.15.98:0/4207217530 >> v2:172.21.15.190:3300/0 conn(0x7fcd680a0200 0x7fcd681365c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 monclient: found mon.noname-d 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.483+0000 7fcd6cf1c640 1 -- 172.21.15.98:0/4207217530 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd680a1f20 con 0x7fcd68136b00 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6cf1c640 1 --2- 172.21.15.98:0/4207217530 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd68136b00 0x7fcd68136fb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcd580098a0 tx=0x7fcd5801c400 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8c0d9d48736deba9 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 1 -- 172.21.15.98:0/4207217530 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd580b0070 con 0x7fcd68136b00 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: dump: 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.499 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 1 -- 172.21.15.98:0/4207217530 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd580aa810 con 0x7fcd68136b00 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 5 monclient: authenticate success, global_id 84705 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 1 -- 172.21.15.98:0/4207217530 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd580a9300 con 0x7fcd68136b00 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 4 set_mon_vals no callback set 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd66ffd640 10 monclient: dump: 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient: shutdown 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- 172.21.15.98:0/4207217530 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd68136b00 msgr2=0x7fcd68136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/4207217530 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd68136b00 0x7fcd68136fb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcd580098a0 tx=0x7fcd5801c400 comp rx=0 tx=0).stop 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- 172.21.15.98:0/4207217530 shutdown_connections 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/4207217530 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd68136b00 0x7fcd68136fb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/4207217530 >> v2:172.21.15.190:3300/0 conn(0x7fcd680a0200 0x7fcd681365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/4207217530 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fcd680a09d0 0x7fcd680a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- 172.21.15.98:0/4207217530 >> 172.21.15.98:0/4207217530 conn(0x7fcd6806eed0 msgr2=0x7fcd6806f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- 172.21.15.98:0/4207217530 shutdown_connections 2024-09-11T03:08:02.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- 172.21.15.98:0/4207217530 wait complete. 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient: monmap: 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.503 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 librados: starting msgr at 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 librados: starting objecter 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 Processor -- start 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- start start 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 librados: setting wanted keys 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 librados: calling monclient init 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient: init 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient: _add_conns ranks=[6,5,7,3,1,2,8,0,4] 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd680a0200 0x7fcd68133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): picked mon.noname-d con 0x7fcd680a0200 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd680a09d0 0x7fcd68134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): picked mon.noname-h con 0x7fcd680a09d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd68136b00 0x7fcd6809cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): picked mon.noname-i con 0x7fcd68136b00 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcd68140520 con 0x7fcd680a09d0 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fcd680a07f0 con 0x7fcd680a0200 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fcd681406a0 con 0x7fcd68136b00 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd6e97f640 10 monclient(hunting): authenticate will time out at 2036.371328s 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd680a0200 0x7fcd68133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd680a0200 0x7fcd68133cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54504/0 (socket says 172.21.15.98:54504) 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 -- 172.21.15.98:0/2647800109 learned_addr learned my addr 172.21.15.98:0/2647800109 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): get_auth_request con 0x7fcd680a0200 auth_method 0 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient(hunting): handle_auth_done global_id 84714 payload 824 2024-09-11T03:08:02.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd68136b00 msgr2=0x7fcd6809cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fcd68136b00 0x7fcd6809cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd680a09d0 msgr2=0x7fcd68134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd680a09d0 0x7fcd68134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 monclient: found mon.noname-d 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 -- 172.21.15.98:0/2647800109 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6813ca70 con 0x7fcd680a0200 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd67fff640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd680a0200 0x7fcd68133cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000efc0 tx=0x7fcd6000c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd64ff9640 10 client.?.objecter ms_handle_connect 0x7fcd680a0200 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.487+0000 7fcd64ff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd60018020 con 0x7fcd680a0200 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: dump: 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.508 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 5 monclient: authenticate success, global_id 84714 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd6000cdb0 con 0x7fcd680a0200 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 10 monclient: _renew_subs 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd6813cd50 con 0x7fcd680a0200 2024-09-11T03:08:02.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd60014d60 con 0x7fcd680a0200 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd667fc640 4 set_mon_vals no callback set 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 10 client.84714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: dump: 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.510 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 10 monclient: _renew_subs 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd6813c790 con 0x7fcd680a0200 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 1 librados: init done 2024-09-11T03:08:02.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d080 0x7fcd6809d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcd6806ca10 con 0x7fcd6809d080 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fcd60014480 con 0x7fcd680a0200 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d080 0x7fcd6809d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient: get_auth_request con 0x7fcd6809d080 auth_method 0 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd5403df70 0x7fcd54040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fcd60050180 con 0x7fcd680a0200 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 client.84714.objecter ms_dispatch 0x7fcd68000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 3 client.84714.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:02.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 3 client.84714.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 20 client.84714.objecter dump_active .. 0 homeless 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6cf1c640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd5403df70 0x7fcd54040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6cf1c640 10 monclient: get_auth_request con 0x7fcd5403df70 auth_method 0 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 10 monclient(hunting): handle_auth_done global_id 74723 payload 824 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 1 -- 172.21.15.98:0/2647800109 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd680a4280 con 0x7fcd6809d080 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 client.84714.objecter ms_handle_connect 0x7fcd6809d080 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 client.84714.objecter resend_mon_ops 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd677fe640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d080 0x7fcd6809d460 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd6809ca50 tx=0x7fcd5c00d980 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 client.84714.objecter ms_handle_connect 0x7fcd5403df70 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd6cf1c640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd5403df70 0x7fcd54040430 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7fcd580ad000 tx=0x7fcd58004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd5c012a60 con 0x7fcd6809d080 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: dump: 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.514 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd5c019070 con 0x7fcd6809d080 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:02.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d080 msgr2=0x7fcd6809d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d080 0x7fcd6809d460 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcd6809ca50 tx=0x7fcd5c00d980 comp rx=0 tx=0).stop 2024-09-11T03:08:02.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d080 msgr2=0x7fcd6809d460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.491+0000 7fcd64ff9640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d080 0x7fcd6809d460 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:02.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d9a0 0x7fcd68137610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fcd6806e870 con 0x7fcd6809d9a0 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d9a0 0x7fcd68137610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient: get_auth_request con 0x7fcd6809d9a0 auth_method 0 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 10 monclient(hunting): handle_auth_done global_id 74732 payload 824 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 1 -- 172.21.15.98:0/2647800109 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd680a0db0 con 0x7fcd6809d9a0 2024-09-11T03:08:02.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd677fe640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d9a0 0x7fcd68137610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00b150 tx=0x7fcd5c0046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 10 client.84714.objecter ms_handle_connect 0x7fcd6809d9a0 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 10 client.84714.objecter resend_mon_ops 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.5 v2:172.21.15.190:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd5c029480 con 0x7fcd6809d9a0 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 10 monclient: dump: 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.519 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 <== mon.5 v2:172.21.15.190:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9268 (secure 0 0 0) 0x7fcd5c012b00 con 0x7fcd6809d9a0 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:02.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d9a0 msgr2=0x7fcd68137610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d9a0 0x7fcd68137610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c00b150 tx=0x7fcd5c0046d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d9a0 msgr2=0x7fcd68137610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd64ff9640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fcd6809d9a0 0x7fcd68137610 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 439429, 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:02.521 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.522 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.523 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:02.524 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.525 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.526 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.527 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:02.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.530 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.531 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:02.534 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:02.535 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.537 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.538 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.539 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.540 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.541 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.542 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 3 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:02.543 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:02.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 10 librados: watch_flush enter 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 10 librados: watch_flush exit 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 20 client.84714.objecter shutdown clearing up homeless session... 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 10 client.84714.objecter successfully canceled tick 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd5403df70 msgr2=0x7fcd54040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fcd5403df70 0x7fcd54040430 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7fcd580ad000 tx=0x7fcd58004740 comp rx=0 tx=0).stop 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 10 monclient: shutdown 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd680a0200 msgr2=0x7fcd68133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 --2- 172.21.15.98:0/2647800109 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fcd680a0200 0x7fcd68133cf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000efc0 tx=0x7fcd6000c490 comp rx=0 tx=0).stop 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6cf1c640 1 -- 172.21.15.98:0/2647800109 reap_dead start 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6cf1c640 1 -- 172.21.15.98:0/2647800109 reap_dead start 2024-09-11T03:08:02.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 shutdown_connections 2024-09-11T03:08:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 >> 172.21.15.98:0/2647800109 conn(0x7fcd6806eed0 msgr2=0x7fcd680727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 shutdown_connections 2024-09-11T03:08:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.499+0000 7fcd6e97f640 1 -- 172.21.15.98:0/2647800109 wait complete. 2024-09-11T03:08:02.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.503+0000 7fcd6e97f640 1 librados: shutdown 2024-09-11T03:08:02.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:02.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-09-11T03:08:02.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-09-11T03:08:02.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.663+0000 7f601f96c640 10 monclient: get_monmap_and_config 2024-09-11T03:08:02.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.663+0000 7f601f96c640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient: monmap: 2024-09-11T03:08:02.666 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:02.669296+0000 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:02.669296+0000 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.667 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.668 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.668 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:02.668 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:02.668 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 Processor -- start 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 -- start start 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient: init 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient: _add_conns ranks=[5,3,10,2,0,8,9,7,4,1,6] 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f60180a46a0 0x7f60180a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): picked mon.noname-h con 0x7f60180a46a0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:02.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60180a4fc0 0x7f6018139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): picked mon.noname-g con 0x7f60180a4fc0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6018139e70 0x7f601813a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): picked mon.noname-f con 0x7f6018139e70 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f60180a0750 con 0x7f6018139e70 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f60180a17b0 con 0x7f60180a4fc0 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f60180775d0 con 0x7f60180a46a0 2024-09-11T03:08:02.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601f96c640 10 monclient(hunting): authenticate will time out at 2036.549507s 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60180a4fc0 0x7f6018139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f60180a46a0 0x7f60180a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f6018139e70 0x7f601813a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60180a4fc0 0x7f6018139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3301/0 says I am v2:172.21.15.98:37964/0 (socket says 172.21.15.98:37964) 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f60180a46a0 0x7f60180a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3302/0 says I am v2:172.21.15.98:42784/0 (socket says 172.21.15.98:42784) 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 1 -- 172.21.15.98:0/2666986247 learned_addr learned my addr 172.21.15.98:0/2666986247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 10 monclient(hunting): get_auth_request con 0x7f60180a4fc0 auth_method 0 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 10 monclient(hunting): get_auth_request con 0x7f60180a46a0 auth_method 0 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): get_auth_request con 0x7f6018139e70 auth_method 0 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601cf07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient(hunting): handle_auth_done global_id 26353 payload 274 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 1 -- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f60180a46a0 msgr2=0x7f60180a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 1 --2- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f60180a46a0 0x7f60180a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 1 -- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60180a4fc0 msgr2=0x7f6018139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 1 --2- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60180a4fc0 0x7f6018139930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 1 monclient: found mon.noname-f 2024-09-11T03:08:02.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:08:02.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.667+0000 7f601df09640 1 -- 172.21.15.98:0/2666986247 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60180a0200 con 0x7f6018139e70 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 --2- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6018139e70 0x7f601813a2d0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f6010009d90 tx=0x7f601001c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edac1a1fc5217188 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 1 -- 172.21.15.98:0/2666986247 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60100ac070 con 0x7f6018139e70 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: dump: 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.682 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.683 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 1 -- 172.21.15.98:0/2666986247 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60100a68f0 con 0x7f6018139e70 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 5 monclient: authenticate success, global_id 26353 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 4 set_mon_vals no callback set 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 1 -- 172.21.15.98:0/2666986247 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60100a6b20 con 0x7f6018139e70 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f600ffff640 10 monclient: dump: 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.685 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient: shutdown 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6018139e70 msgr2=0x7f601813a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 --2- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6018139e70 0x7f601813a2d0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f6010009d90 tx=0x7f601001c830 comp rx=0 tx=0).stop 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- 172.21.15.98:0/2666986247 shutdown_connections 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 --2- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6018139e70 0x7f601813a2d0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 --2- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f60180a4fc0 0x7f6018139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 --2- 172.21.15.98:0/2666986247 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f60180a46a0 0x7f60180a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- 172.21.15.98:0/2666986247 >> 172.21.15.98:0/2666986247 conn(0x7f601806ed00 msgr2=0x7f601806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- 172.21.15.98:0/2666986247 shutdown_connections 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- 172.21.15.98:0/2666986247 wait complete. 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient: monmap: 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:02.687 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 librados: starting msgr at 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 librados: starting objecter 2024-09-11T03:08:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 Processor -- start 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- start start 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 librados: setting wanted keys 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 librados: calling monclient init 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient: init 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient: _add_conns ranks=[8,4,2,3,1,5,6,7,0] 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60180a46a0 0x7f601813a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): picked mon.noname-e con 0x7f60180a46a0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60180a4fc0 0x7f601813aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): picked mon.noname-c con 0x7f60180a4fc0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6018139e70 0x7f601813b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): picked mon.noname-b con 0x7f6018139e70 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f60180779d0 con 0x7f6018139e70 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f6018077850 con 0x7f60180a4fc0 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6018143410 con 0x7f60180a46a0 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601f96c640 10 monclient(hunting): authenticate will time out at 2036.555254s 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60180a46a0 0x7f601813a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6018139e70 0x7f601813b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60180a4fc0 0x7f601813aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60180a46a0 0x7f601813a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49514/0 (socket says 172.21.15.98:49514) 2024-09-11T03:08:02.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 1 -- 172.21.15.98:0/1880514843 learned_addr learned my addr 172.21.15.98:0/1880514843 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): get_auth_request con 0x7f6018139e70 auth_method 0 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 10 monclient(hunting): get_auth_request con 0x7f60180a4fc0 auth_method 0 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 10 monclient(hunting): get_auth_request con 0x7f60180a46a0 auth_method 0 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient(hunting): handle_auth_done global_id 94376 payload 824 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60180a46a0 msgr2=0x7f601813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f60180a46a0 0x7f601813a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60180a4fc0 msgr2=0x7f601813aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60180a4fc0 0x7f601813aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 monclient: found mon.noname-b 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601df09640 1 -- 172.21.15.98:0/1880514843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f601813dc40 con 0x7f6018139e70 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.671+0000 7f601cf07640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f60180a4fc0 0x7f601813aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601df09640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6018139e70 0x7f601813b430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f60100ab000 tx=0x7f60100195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 client.?.objecter ms_handle_connect 0x7f6018139e70 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60100ac030 con 0x7f6018139e70 2024-09-11T03:08:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: dump: 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.694 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 5 monclient: authenticate success, global_id 94376 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6010003e10 con 0x7f6018139e70 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60100044b0 con 0x7f6018139e70 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 10 monclient: _renew_subs 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f601813df20 con 0x7f6018139e70 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600f7fe640 4 set_mon_vals no callback set 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: dump: 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.696 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 10 client.94376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:02.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 10 monclient: _renew_subs 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f601813ecc0 con 0x7f6018139e70 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 1 librados: init done 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601813f9c0 0x7f6018136fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f601813ee90 con 0x7f601813f9c0 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f60100b9070 con 0x7f6018139e70 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601813f9c0 0x7f6018136fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5ffc03df30 0x7f5ffc0403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient: get_auth_request con 0x7f601813f9c0 auth_method 0 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f60100e98f0 con 0x7f6018139e70 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 client.94376.objecter ms_dispatch 0x7f6018000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 3 client.94376.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 3 client.94376.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 20 client.94376.objecter dump_active .. 0 homeless 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601cf07640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5ffc03df30 0x7f5ffc0403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601cf07640 10 monclient: get_auth_request con 0x7f5ffc03df30 auth_method 0 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 10 monclient(hunting): handle_auth_done global_id 104155 payload 824 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 1 -- 172.21.15.98:0/1880514843 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60180a85c0 con 0x7f601813f9c0 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 client.94376.objecter ms_handle_connect 0x7f601813f9c0 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 client.94376.objecter resend_mon_ops 2024-09-11T03:08:02.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601d708640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601813f9c0 0x7f6018136fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60140119c0 tx=0x7f6014011e80 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 client.94376.objecter ms_handle_connect 0x7f5ffc03df30 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6014019070 con 0x7f601813f9c0 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f601cf07640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5ffc03df30 0x7f5ffc0403f0 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f601813c7f0 tx=0x7f6008009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: dump: 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.700 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6014004300 con 0x7f601813f9c0 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601813f9c0 msgr2=0x7f6018136fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601813f9c0 0x7f6018136fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60140119c0 tx=0x7f6014011e80 comp rx=0 tx=0).stop 2024-09-11T03:08:02.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601813f9c0 msgr2=0x7f6018136fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.675+0000 7f600dffb640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601813f9c0 0x7f6018136fe0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60140119c0 tx=0x7f6014011e80 comp rx=0 tx=0).stop 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601f96c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601f96c640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601802d450 0x7f601802d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601f96c640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f601806ca10 con 0x7f601802d450 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601802d450 0x7f601802d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient: get_auth_request con 0x7f601802d450 auth_method 0 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 10 monclient(hunting): handle_auth_done global_id 104164 payload 824 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 1 -- 172.21.15.98:0/1880514843 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f601806f390 con 0x7f601802d450 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f601d708640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601802d450 0x7f601802d860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60140119c0 tx=0x7f60140046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 10 client.94376.objecter ms_handle_connect 0x7f601802d450 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 10 client.94376.objecter resend_mon_ops 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6014019040 con 0x7f601802d450 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 10 monclient: dump: 2024-09-11T03:08:02.704 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.705 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 <== mon.7 v2:172.21.15.190:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9267 (secure 0 0 0) 0x7f601401e450 con 0x7f601802d450 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601802d450 msgr2=0x7f601802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601802d450 0x7f601802d860 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60140119c0 tx=0x7f60140046d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601802d450 msgr2=0x7f601802d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-11T03:08:02.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.683+0000 7f600dffb640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f601802d450 0x7f601802d860 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60140119c0 tx=0x7f60140046d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "peon", 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: "election_epoch": 122, 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 29127, 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum": [ 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: 0, 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: 1, 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: 2, 2024-09-11T03:08:02.707 INFO:teuthology.orchestra.run.smithi098.stdout: 3, 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: 4, 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: 5, 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: 6, 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: 7, 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: 8 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_age": 3, 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: "required_con": "2450521708007866372", 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: "required_mon": [ 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.708 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_con": "4540719139924082687", 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "quorum_mon": [ 2024-09-11T03:08:02.709 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "outside_quorum": [], 2024-09-11T03:08:02.710 INFO:teuthology.orchestra.run.smithi098.stdout: "extra_probe_peers": [], 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_provider": [], 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "monmap": { 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "fsid": "797ac5bf-8bfb-4032-bff5-2e861cee7479", 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "modified": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "created": "2024-09-11T02:49:17.084567Z", 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release": 19, 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min_mon_release_name": "squid", 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "election_strategy": 3, 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "disallowed_leaders: ": "", 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false, 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tiebreaker_mon": "", 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "removed_ranks: ": "", 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "features": { 2024-09-11T03:08:02.711 INFO:teuthology.orchestra.run.smithi098.stdout: "persistent": [ 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "kraken", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "luminous", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "mimic", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap-prune", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "nautilus", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "elector-pinging", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "reef", 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "squid" 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.712 INFO:teuthology.orchestra.run.smithi098.stdout: "optional": [] 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "mons": [ 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 0, 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "a", 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3300", 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.713 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789", 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6789/0", 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.714 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 1, 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "f", 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3300", 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789", 2024-09-11T03:08:02.715 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6789/0", 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 2, 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "b", 2024-09-11T03:08:02.716 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3301", 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790", 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6790/0", 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 3, 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "g", 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3301", 2024-09-11T03:08:02.718 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790", 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6790/0", 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.720 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 4, 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "c", 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3302", 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791", 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6791/0", 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 5, 2024-09-11T03:08:02.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "h", 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3302", 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791", 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6791/0", 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 6, 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "d", 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3303", 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792", 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6792/0", 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.725 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 7, 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "i", 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:3303", 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.726 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792", 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.190:6792/0", 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "rank": 8, 2024-09-11T03:08:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "e", 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addrs": { 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:3304", 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793", 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 0 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "public_addr": "172.21.15.98:6793/0", 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "priority": 0, 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "weight": 0, 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_location": "{}" 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "feature_map": { 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": [ 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.729 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: "client": [ 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: "features": "0x3f03dffffffdffff", 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: "release": "squid", 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 1 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout: "stretch_mode": false 2024-09-11T03:08:02.730 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 10 librados: watch_flush enter 2024-09-11T03:08:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 10 librados: watch_flush exit 2024-09-11T03:08:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 20 client.94376.objecter shutdown clearing up homeless session... 2024-09-11T03:08:02.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 10 client.94376.objecter successfully canceled tick 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5ffc03df30 msgr2=0x7f5ffc0403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f5ffc03df30 0x7f5ffc0403f0 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f601813c7f0 tx=0x7f6008009240 comp rx=0 tx=0).stop 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 10 monclient: shutdown 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6018139e70 msgr2=0x7f601813b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 --2- 172.21.15.98:0/1880514843 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f6018139e70 0x7f601813b430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f60100ab000 tx=0x7f60100195d0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601df09640 1 -- 172.21.15.98:0/1880514843 reap_dead start 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601df09640 1 -- 172.21.15.98:0/1880514843 reap_dead start 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 shutdown_connections 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 >> 172.21.15.98:0/1880514843 conn(0x7f601806ed00 msgr2=0x7f601813fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 shutdown_connections 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 -- 172.21.15.98:0/1880514843 wait complete. 2024-09-11T03:08:02.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.687+0000 7f601f96c640 1 librados: shutdown 2024-09-11T03:08:02.734 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-09-11T03:08:02.734 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-11T03:08:02.734 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.847+0000 7f9074101640 10 monclient: get_monmap_and_config 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.847+0000 7f9074101640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.847+0000 7f9074101640 10 monclient: monmap: 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:02.851730+0000 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:02.851730+0000 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:02.849 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:02.850 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.847+0000 7f9074101640 1 Processor -- start 2024-09-11T03:08:02.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.847+0000 7f9074101640 1 -- start start 2024-09-11T03:08:02.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.847+0000 7f9074101640 10 monclient: init 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient: _add_conns ranks=[5,6,2,3,9,0,7,8,10,1,4] 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): picked mon.noname-h con 0x7f906c06f6c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f906c06faa0 0x7f906c06fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): picked mon.noname-d con 0x7f906c06faa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f906c0703c0 0x7f906c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): picked mon.noname-b con 0x7f906c0703c0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f906c071ff0 con 0x7f906c0703c0 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f906c0a8750 con 0x7f906c06f6c0 2024-09-11T03:08:02.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f906c0716c0 con 0x7f906c06faa0 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient(hunting): authenticate will time out at 2036.732748s 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f906c0703c0 0x7f906c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f906c06faa0 0x7f906c06fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f906c0703c0 0x7f906c1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60434/0 (socket says 172.21.15.98:60434) 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 -- 172.21.15.98:0/4189912413 learned_addr learned my addr 172.21.15.98:0/4189912413 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): get_auth_request con 0x7f906c0703c0 auth_method 0 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 10 monclient(hunting): get_auth_request con 0x7f906c06faa0 auth_method 0 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9071e9d640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9071e9d640 10 monclient(hunting): get_auth_request con 0x7f906c06f6c0 auth_method 0 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9071e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9071e9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9071e9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907169c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient(hunting): handle_auth_done global_id 94385 payload 274 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 -- 172.21.15.98:0/4189912413 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f906c06faa0 msgr2=0x7f906c06fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f906c06faa0 0x7f906c06fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 -- 172.21.15.98:0/4189912413 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 msgr2=0x7f906c072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 monclient: found mon.noname-b 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 -- 172.21.15.98:0/4189912413 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c0a64a0 con 0x7f906c0703c0 2024-09-11T03:08:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9071e9d640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f907269e640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f906c0703c0 0x7f906c1397c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9064009e80 tx=0x7f906401c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2ed0a64c4da190b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 1 -- 172.21.15.98:0/4189912413 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f90640ac020 con 0x7f906c0703c0 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: dump: 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.862 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 1 -- 172.21.15.98:0/4189912413 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9064009110 con 0x7f906c0703c0 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 5 monclient: authenticate success, global_id 94385 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 1 -- 172.21.15.98:0/4189912413 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f90640a6d40 con 0x7f906c0703c0 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9074101640 4 set_mon_vals no callback set 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.851+0000 7f9070e9b640 10 monclient: dump: 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: shutdown 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/4189912413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f906c0703c0 msgr2=0x7f906c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f906c0703c0 0x7f906c1397c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9064009e80 tx=0x7f906401c650 comp rx=0 tx=0).stop 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/4189912413 shutdown_connections 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f906c0703c0 0x7f906c1397c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f906c06faa0 0x7f906c06fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 --2- 172.21.15.98:0/4189912413 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/4189912413 >> 172.21.15.98:0/4189912413 conn(0x7f906c0acac0 msgr2=0x7f906c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/4189912413 shutdown_connections 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/4189912413 wait complete. 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: build_initial_monmap 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: monmap: 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:02.867 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 librados: starting msgr at 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 librados: starting objecter 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 Processor -- start 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- start start 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 librados: setting wanted keys 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 librados: calling monclient init 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: init 2024-09-11T03:08:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _add_conns ranks=[5,8,3,1,0,2,6,7,4] 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c0a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): picked mon.noname-h con 0x7f906c06f6c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f906c06faa0 0x7f906c0a1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): picked mon.noname-e con 0x7f906c06faa0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f906c0703c0 0x7f906c0a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): picked mon.noname-g con 0x7f906c0703c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f906c143840 con 0x7f906c0703c0 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f906c1436c0 con 0x7f906c06f6c0 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f906c1439c0 con 0x7f906c06faa0 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient(hunting): authenticate will time out at 2036.737928s 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f906c06faa0 0x7f906c0a1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f906c06faa0 0x7f906c0a1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49518/0 (socket says 172.21.15.98:49518) 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 -- 172.21.15.98:0/462481266 learned_addr learned my addr 172.21.15.98:0/462481266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): get_auth_request con 0x7f906c06faa0 auth_method 0 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907269e640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f906c0703c0 0x7f906c0a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9071e9d640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c0a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9071e9d640 10 monclient(hunting): get_auth_request con 0x7f906c06f6c0 auth_method 0 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9071e9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9071e9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9071e9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907269e640 10 monclient(hunting): get_auth_request con 0x7f906c0703c0 auth_method 0 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907269e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907269e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907269e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient(hunting): handle_auth_done global_id 25514 payload 824 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 -- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 msgr2=0x7f906c0a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c0a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 -- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f906c0703c0 msgr2=0x7f906c0a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f906c0703c0 0x7f906c0a24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient: _finish_hunting 0 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 monclient: found mon.noname-e 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 -- 172.21.15.98:0/462481266 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906c0aa470 con 0x7f906c06faa0 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9071e9d640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c0a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907269e640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f906c0703c0 0x7f906c0a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f907169c640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f906c06faa0 0x7f906c0a1fa0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e960 tx=0x7f905c00ee20 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 client.?.objecter ms_handle_connect 0x7f906c06faa0 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 1 -- 172.21.15.98:0/462481266 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f905c017070 con 0x7f906c06faa0 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: dump: 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 1 -- 172.21.15.98:0/462481266 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f905c00ce70 con 0x7f906c06faa0 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 5 monclient: authenticate success, global_id 25514 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _renew_subs 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906c0a5270 con 0x7f906c06faa0 2024-09-11T03:08:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 1 -- 172.21.15.98:0/462481266 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f905c0138e0 con 0x7f906c06faa0 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9063fff640 4 set_mon_vals no callback set 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 client.25514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 10 monclient: dump: 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _renew_subs 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f906c140150 con 0x7f906c06faa0 2024-09-11T03:08:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 librados: init done 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f906c13fcb0 con 0x7f906c06faa0 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.855+0000 7f90627fc640 1 -- 172.21.15.98:0/462481266 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f905c013470 con 0x7f906c06faa0 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f90627fc640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f905003df70 0x7f9050040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f90627fc640 1 -- 172.21.15.98:0/462481266 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f905c00f040 con 0x7f906c06faa0 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f90627fc640 10 client.25514.objecter ms_dispatch 0x7f906c000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f90627fc640 3 client.25514.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f90627fc640 3 client.25514.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f90627fc640 20 client.25514.objecter dump_active .. 0 homeless 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f9071e9d640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f905003df70 0x7f9050040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f9071e9d640 10 monclient: get_auth_request con 0x7f905003df70 auth_method 0 2024-09-11T03:08:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f90627fc640 10 client.25514.objecter ms_handle_connect 0x7f905003df70 2024-09-11T03:08:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.859+0000 7f9071e9d640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f905003df70 0x7f9050040430 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 crypto rx=0x7f9068009970 tx=0x7f9068008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.863+0000 7f90627fc640 1 -- 172.21.15.98:0/462481266 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f905c021070 con 0x7f906c06faa0 2024-09-11T03:08:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.863+0000 7f90627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:02.863+0000 7f90627fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:03.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.087+0000 7f9074101640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-11T03:08:03.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.087+0000 7f9074101640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-11T03:08:03.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.087+0000 7f9074101640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:03.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.087+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f906c06fe80 con 0x7f906c06faa0 2024-09-11T03:08:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f90627fc640 1 -- 172.21.15.98:0/462481266 <== mon.8 v2:172.21.15.98:3304/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f906c0aa470 con 0x7f906c06faa0 2024-09-11T03:08:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f90627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-11T03:08:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f90627fc640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=742315963,config=692619927,health=1562255701,kv=3991532845,logm=3719552169}) 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=742315963,config=692619927,health=123281230,kv=3991532845,logm=3031105363}) 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.093 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=3808373997,kv=3994774503,logm=700341093}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=20,config=1,health=11,kv=10,logm=58} crc {auth=593852294,config=1228700967,health=2030541253,kv=3991532845,logm=3671839683}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=20,config=2,health=12,kv=10,logm=56} crc {auth=742315963,config=692619927,health=148817331,kv=940914997,logm=3336723152}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=742315963,config=692619927,health=3799593782,kv=3991532845,logm=138076615}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=1000494959,config=692619927,health=3011496219,kv=3991532845,logm=889677425}) 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.094 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=9,logm=59} crc {auth=742315963,config=692619927,health=385417122,kv=1551318347,logm=3359215752}) 2024-09-11T03:08:03.095 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.095+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=66036512,config=692619927,health=1596444868,kv=3991532845,logm=1572718690}) 2024-09-11T03:08:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 10 librados: watch_flush enter 2024-09-11T03:08:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 10 librados: watch_flush exit 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 20 client.25514.objecter shutdown clearing up homeless session... 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 10 client.25514.objecter successfully canceled tick 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f905003df70 msgr2=0x7f9050040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f905003df70 0x7f9050040430 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 crypto rx=0x7f9068009970 tx=0x7f9068008040 comp rx=0 tx=0).stop 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 10 monclient: shutdown 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f906c06faa0 msgr2=0x7f906c0a1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f906c06faa0 0x7f906c0a1fa0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e960 tx=0x7f905c00ee20 comp rx=0 tx=0).stop 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 shutdown_connections 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f905003df70 0x7f9050040430 unknown :-1 s=CLOSED pgs=3708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f906c0703c0 0x7f906c0a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:03.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f906c06faa0 0x7f906c0a1fa0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 --2- 172.21.15.98:0/462481266 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f906c06f6c0 0x7f906c0a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 >> 172.21.15.98:0/462481266 conn(0x7f906c0acac0 msgr2=0x7f906c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 shutdown_connections 2024-09-11T03:08:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 -- 172.21.15.98:0/462481266 wait complete. 2024-09-11T03:08:03.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:03.091+0000 7f9074101640 1 librados: shutdown 2024-09-11T03:08:03.098 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.099+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.099+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.099+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102411136}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.099+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.099+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684642235}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1209756496}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2397407494}) 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.099 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3877980678}) 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2345867585}) 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=788709978}) 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738007674}) 2024-09-11T03:08:03.100 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.103+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2808977183}) 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2068751211}) 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476101826}) 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4173555265}) 2024-09-11T03:08:03.105 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4065073244}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3626200932}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3782574125}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=973637236}) 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.106 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=488847212}) 2024-09-11T03:08:03.107 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.107+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=865035306}) 2024-09-11T03:08:03.111 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.111 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480285659}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2242377343}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2643950671}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=933009413}) 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.112 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1659447419}) 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2916935091}) 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2653761114}) 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3783653542}) 2024-09-11T03:08:03.113 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.115+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2346401987}) 2024-09-11T03:08:03.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411786188}) 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512032076}) 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1954784301}) 2024-09-11T03:08:03.117 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=233855366}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2255538118}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3248668457}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3482611509}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854372440}) 2024-09-11T03:08:03.118 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.119+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3545271971}) 2024-09-11T03:08:03.120 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991138338}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966072340}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1576530313}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3120358475}) 2024-09-11T03:08:03.121 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=214377818}) 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2373617336}) 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=89266323}) 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=256776244}) 2024-09-11T03:08:03.122 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.123+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3233758475}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555270200}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=258942845}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3521710421}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.125 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=99458190}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1514282303}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2712939727}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4269922864}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915119910}) 2024-09-11T03:08:03.126 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.127+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2235908711}) 2024-09-11T03:08:03.128 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.128 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176770840}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3123338273}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3613447349}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2826241235}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.129 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=65622077}) 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=105874545}) 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=844772195}) 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838088870}) 2024-09-11T03:08:03.130 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.131+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=837423231}) 2024-09-11T03:08:03.132 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.132 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.132 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186790863}) 2024-09-11T03:08:03.132 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733956104}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3263363064}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1459745067}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2217908801}) 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.133 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.134 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2735984412}) 2024-09-11T03:08:03.134 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.134 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.134 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2327014587}) 2024-09-11T03:08:03.134 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.134 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617681998}) 2024-09-11T03:08:03.134 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.135+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=951880374}) 2024-09-11T03:08:03.136 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809077153}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1490726248}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3854032025}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3055111667}) 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.137 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1324457037}) 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=685606181}) 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1607242249}) 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=655992324}) 2024-09-11T03:08:03.138 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.139+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=212767343}) 2024-09-11T03:08:03.140 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.140 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.140 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535611908}) 2024-09-11T03:08:03.140 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.140 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.140 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945479761}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=611415810}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2790454736}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3215967142}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4266924554}) 2024-09-11T03:08:03.141 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.142 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.142 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=220674540}) 2024-09-11T03:08:03.142 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.142 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45463845}) 2024-09-11T03:08:03.142 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.143+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4009008961}) 2024-09-11T03:08:03.144 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.144 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.144 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4047240888}) 2024-09-11T03:08:03.144 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.144 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043876404}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317129394}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4095719014}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=319570305}) 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.145 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.146 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=589211548}) 2024-09-11T03:08:03.146 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.146 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.146 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3620905801}) 2024-09-11T03:08:03.146 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.146 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319779068}) 2024-09-11T03:08:03.146 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.147+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=654866274}) 2024-09-11T03:08:03.148 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.148 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.148 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177664139}) 2024-09-11T03:08:03.148 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.148 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723900320}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=737414238}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2116379305}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3018842672}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.149 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=71837887}) 2024-09-11T03:08:03.150 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.150 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.150 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1372250540}) 2024-09-11T03:08:03.150 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.150 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3516801897}) 2024-09-11T03:08:03.150 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.151+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3692903565}) 2024-09-11T03:08:03.152 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.152 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337603907}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3779938607}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3714049507}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=281076827}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.153 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1636925580}) 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1557728117}) 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3398081804}) 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935974228}) 2024-09-11T03:08:03.154 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.155+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3199606311}) 2024-09-11T03:08:03.156 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.156 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.156 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897663779}) 2024-09-11T03:08:03.156 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936861337}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=101839345}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4240696236}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1224649552}) 2024-09-11T03:08:03.157 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3081512982}) 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1327923024}) 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818183642}) 2024-09-11T03:08:03.158 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.159+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4146296812}) 2024-09-11T03:08:03.160 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.160 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292647433}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=276010192}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=574103337}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1917150924}) 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.161 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3511451394}) 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1675788561}) 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1547921068}) 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686981811}) 2024-09-11T03:08:03.162 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.163+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2641275157}) 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184482028}) 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750919142}) 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385493359}) 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.164 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=720234118}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=213545251}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1688509691}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4140716304}) 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.165 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1497939028}) 2024-09-11T03:08:03.166 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.167+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3998707967}) 2024-09-11T03:08:03.167 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.167 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372822705}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532291529}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3407796286}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=923882246}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.168 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1686557797}) 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4220934222}) 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1863437965}) 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3800228028}) 2024-09-11T03:08:03.169 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.171+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1629264064}) 2024-09-11T03:08:03.171 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.171 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.171 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152523822}) 2024-09-11T03:08:03.171 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1844381049}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3598130381}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2192450725}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2048779475}) 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.172 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.173 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1786356862}) 2024-09-11T03:08:03.173 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.173 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.173 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3912332480}) 2024-09-11T03:08:03.173 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.173 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140781244}) 2024-09-11T03:08:03.173 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.175+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4184152280}) 2024-09-11T03:08:03.175 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.175 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.175 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415014258}) 2024-09-11T03:08:03.175 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.175 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.175 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210238686}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=709437146}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2135479157}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1444069944}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1708819331}) 2024-09-11T03:08:03.176 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.177 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.177 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1166867405}) 2024-09-11T03:08:03.177 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.177 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607940911}) 2024-09-11T03:08:03.177 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.179+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=160703224}) 2024-09-11T03:08:03.178 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.178 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.178 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112242664}) 2024-09-11T03:08:03.178 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.178 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3981886892}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=539592682}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=581040211}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3652915615}) 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.179 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.180 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2127032174}) 2024-09-11T03:08:03.180 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.180 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.180 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1286663825}) 2024-09-11T03:08:03.180 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.180 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728960835}) 2024-09-11T03:08:03.180 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2400739946}) 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4076268620}) 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4099533081}) 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.182 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=988259921}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.183+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3311379716}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2658904446}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3227902699}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966914533}) 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.183 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764977922}) 2024-09-11T03:08:03.184 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=858600605}) 2024-09-11T03:08:03.186 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.186 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.186 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473272969}) 2024-09-11T03:08:03.186 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008863908}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2062135265}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1420891695}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1343323023}) 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.187 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.188 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1982074473}) 2024-09-11T03:08:03.188 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.188 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.188 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4092280810}) 2024-09-11T03:08:03.188 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.188 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.187+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229293814}) 2024-09-11T03:08:03.188 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3702346283}) 2024-09-11T03:08:03.189 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489416234}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484328470}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1661974478}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3816368414}) 2024-09-11T03:08:03.190 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3982165292}) 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2483222038}) 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3154686108}) 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2023802512}) 2024-09-11T03:08:03.191 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.191+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3372319036}) 2024-09-11T03:08:03.193 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021803004}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3416822542}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=471570451}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3839968295}) 2024-09-11T03:08:03.194 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1940826476}) 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3922727637}) 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1808684873}) 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2873267737}) 2024-09-11T03:08:03.195 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.195+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2576509678}) 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617239675}) 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287577573}) 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4220278161}) 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.200 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2864586377}) 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=721239530}) 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1684168065}) 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.201 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3403508314}) 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294198757}) 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.199+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1452246253}) 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084165413}) 2024-09-11T03:08:03.204 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626566920}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=400109262}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=729333305}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1607978496}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.205 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=601743111}) 2024-09-11T03:08:03.206 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.206 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.206 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=319669846}) 2024-09-11T03:08:03.206 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.206 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354266748}) 2024-09-11T03:08:03.206 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.203+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=714611397}) 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558810034}) 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2221573180}) 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.212 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1074342870}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2191191852}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2513208344}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1358677180}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2912128164}) 2024-09-11T03:08:03.213 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568292790}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.207+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3495896485}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191139665}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554005389}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1108781827}) 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.214 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3445835184}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=78432099}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3613405802}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1079168465}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248885416}) 2024-09-11T03:08:03.215 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3788995213}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222913560}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039468467}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2090216092}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3140465555}) 2024-09-11T03:08:03.216 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=607642650}) 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=648072750}) 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.211+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2555530624}) 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983328336}) 2024-09-11T03:08:03.217 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2239995040}) 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045048013}) 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461729824}) 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.227 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=918706634}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1408224884}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2218464040}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=74213659}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.228 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=232909147}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101755583}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.215+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=629373670}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656856773}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1755582236}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=749044306}) 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.229 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1045106848}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3234441316}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2541711644}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1391341405}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262791499}) 2024-09-11T03:08:03.230 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.219+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2479137649}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818977866}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040073782}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3316343206}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3136308192}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.231 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1463965339}) 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3757590764}) 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2455464583}) 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177657854}) 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.223+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1819955470}) 2024-09-11T03:08:03.232 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67258924}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575897050}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1719116016}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643526904}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.233 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=716960233}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1841086086}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3210323443}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642973618}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1476273547}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930750907}) 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.234 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.227+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=452842906}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1002435201}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1735403563}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3632904334}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.235 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2261706248}) 2024-09-11T03:08:03.236 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.236 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.236 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1370916941}) 2024-09-11T03:08:03.236 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.236 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451106951}) 2024-09-11T03:08:03.236 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=592809559}) 2024-09-11T03:08:03.251 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.251 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.251 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973854134}) 2024-09-11T03:08:03.251 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.251 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993851218}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=664851180}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4108002947}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3976075881}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.252 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1643066125}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1703665034}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042606098}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.231+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=853239044}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3323923946}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308184965}) 2024-09-11T03:08:03.253 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2525716264}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3986269453}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=46036287}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1778464020}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.254 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=699297963}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588409532}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.235+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=112470070}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436859}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899143310}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3671286724}) 2024-09-11T03:08:03.255 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3694423893}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3439969410}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2158438865}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4208424926}) 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.256 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509080340}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1796441361}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749883326}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=60134380}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.239+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2924193202}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.257 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2795250677}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1123261987}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1355907892}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3787364282}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942674771}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3425328253}) 2024-09-11T03:08:03.258 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083255254}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210670834}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1890800219}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1949082085}) 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.259 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2342946837}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1177283667}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3431582747}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403656589}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.243+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3922330853}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.260 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1432529334}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1294896261}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3745312539}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3793586894}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.261 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1964429617}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=271023639}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3828485493}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1238090055}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.247+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2601840722}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618169070}) 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.262 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483164026}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868887307}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4115269833}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2260699647}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.263 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=622734504}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=252349372}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934246111}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3493002893}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998420843}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2031853990}) 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.264 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=994146683}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2773760645}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2563760231}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=874098207}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.251+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.265 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1533364356}) 2024-09-11T03:08:03.266 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.266 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=798352282}) 2024-09-11T03:08:03.266 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3352469026}) 2024-09-11T03:08:03.289 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1623141040}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711524642}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1742049731}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1435643595}) 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.290 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=867426201}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1534458410}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=982338624}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125781809}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.255+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850722081}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.291 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877520061}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542006433}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=59989543}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3707724530}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=216422588}) 2024-09-11T03:08:03.292 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2998133151}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1531463270}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2295877534}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.259+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=194514856}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346719677}) 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.293 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087113394}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3761417256}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4251780453}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4254855696}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.294 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1349994270}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2380993569}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805358552}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.263+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2082010824}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305277360}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556292683}) 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.295 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1762991649}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2572456934}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=766707209}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3506296382}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.296 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=914497841}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754644164}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.267+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1120119927}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4052936491}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3732540928}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1400883029}) 2024-09-11T03:08:03.297 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=692221109}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3659490575}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=243766427}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3592411342}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372673859}) 2024-09-11T03:08:03.298 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3271332848}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581779197}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742514725}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4270110737}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1136800083}) 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.299 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2883749816}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4027881433}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311193807}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761698914}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.271+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=487304287}) 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.300 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4108772120}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293200315}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3734210355}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=491082999}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.301 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=329521316}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3811332242}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3075886628}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696149766}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2772609439}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555052562}) 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.302 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304145782}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1466649313}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2278073090}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.275+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1702643023}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4115170637}) 2024-09-11T03:08:03.303 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=661163464}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1608076161}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1572215159}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172499070}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954985266}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.304 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3008236770}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=102730727}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=901716621}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2657446541}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2179449695}) 2024-09-11T03:08:03.305 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3764215401}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.279+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3089626279}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762393062}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340041068}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1305495812}) 2024-09-11T03:08:03.306 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4165533579}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1023682750}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3368835737}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2816211300}) 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.307 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1987254787}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1731965648}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977427138}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4287046791}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=630178446}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.308 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1305520932}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2971017203}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2820685105}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3912559724}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258906978}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.283+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3914292700}) 2024-09-11T03:08:03.309 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3091541734}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625911872}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=206981384}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1857303328}) 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.310 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2371432044}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2087336838}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3817417333}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002977878}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=93783809}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.311 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3053744560}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414346842}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=561443637}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1653141756}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=782266495}) 2024-09-11T03:08:03.312 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.287+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1728721880}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3845543955}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127211287}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3153669101}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2994323737}) 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.313 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949809034}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3100913037}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3582977033}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1139556973}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.314 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4182955307}) 2024-09-11T03:08:03.315 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.315 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.315 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4003698902}) 2024-09-11T03:08:03.315 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.315 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711519213}) 2024-09-11T03:08:03.315 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.291+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1475452873}) 2024-09-11T03:08:03.479 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3533824828}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110037439}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3361852500}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1428877750}) 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.480 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3981701476}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1405953565}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2713083489}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495279227}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3287591165}) 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.481 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462693544}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216363406}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1247075351}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614481455}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.482 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1954945098}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1678055282}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1527578499}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2894264308}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.295+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3021497692}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093930841}) 2024-09-11T03:08:03.483 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2540869883}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3847494337}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1312193740}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3580731198}) 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.484 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2004259576}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=236938705}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198412823}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407041667}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2038059723}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.485 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3110338939}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2979429619}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2541949653}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3529201301}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.486 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=826589964}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1040714980}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=340556597}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.299+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2064506729}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196480446}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3950035705}) 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.487 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=231868388}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1191997785}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3970128524}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2327473033}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.488 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3463431952}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501178538}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2950115253}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097684882}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1271338943}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=166511741}) 2024-09-11T03:08:03.489 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250270599}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4202359117}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=442612346}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2045638701}) 2024-09-11T03:08:03.490 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.303+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021042337}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=537523039}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087905585}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=457329738}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3607234197}) 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.491 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2123854203}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1930655487}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2407045703}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1354498631}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3411074812}) 2024-09-11T03:08:03.492 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.307+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2849726134}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661089836}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=754954697}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2884678056}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=232464319}) 2024-09-11T03:08:03.493 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2852178031}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1063832814}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=797751623}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1721756941}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3677532300}) 2024-09-11T03:08:03.494 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408610087}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1313047141}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=980282815}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1448770714}) 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.495 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3671442979}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1951049834}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4207408991}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449585369}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.311+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3535152293}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.496 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1933449522}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676608289}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=157372638}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3939317079}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3908350521}) 2024-09-11T03:08:03.497 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3810459876}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3552831893}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924110886}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=733884713}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830320343}) 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.498 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=193753940}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=69719691}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2721316729}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1690442407}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=34945862}) 2024-09-11T03:08:03.499 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=468584793}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330292227}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.315+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3728451681}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837808009}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3365509326}) 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.500 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3715236321}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=995913611}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2155320533}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2933840639}) 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.501 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=235784780}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155963806}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.319+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3229088597}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=750168766}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2810572230}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.502 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3107876141}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=58669575}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3094152982}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2016819443}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1069224088}) 2024-09-11T03:08:03.503 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397743344}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2800044968}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151628717}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3450877185}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=128616489}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.504 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4118075607}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4176037211}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1016167048}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2451313988}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68798121}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.323+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1656148860}) 2024-09-11T03:08:03.505 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444414077}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227759173}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=168622348}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=136176606}) 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.506 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3398750890}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1180146436}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2455724005}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741069720}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3392212853}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.507 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807706931}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2060451449}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3105584308}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2945458451}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.508 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4150628437}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3413012509}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2415559145}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262600920}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.327+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3312253137}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333411284}) 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.509 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967220338}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1196309475}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3751821068}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3233831159}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.510 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3207618629}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1751259146}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273531882}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.331+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4270597802}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019132921}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740941840}) 2024-09-11T03:08:03.511 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4268934526}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3507279001}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3621267464}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1616271453}) 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.512 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1171770962}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=959240659}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4063038482}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947322788}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298421492}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.513 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=361603141}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=450411083}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3130627549}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=899456859}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1453999791}) 2024-09-11T03:08:03.514 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1755511448}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.335+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=44170825}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193275627}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613624057}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4078286543}) 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.515 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2023673827}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3398805143}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2592081758}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3625389772}) 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.516 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491092248}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=593379393}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165385899}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226767823}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3956232966}) 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.517 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3495945802}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=312801243}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4278908086}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2910897270}) 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.518 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580147510}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.339+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1608619140}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885900283}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=276596799}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=813759285}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.519 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=288596767}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3425434255}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=978487578}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2758186692}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1845923826}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=402501652}) 2024-09-11T03:08:03.520 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479712135}) 2024-09-11T03:08:03.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.343+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047305758}) 2024-09-11T03:08:03.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.521 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300607032}) 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1548581098}) 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2663707090}) 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.561 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2293355154}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2180904343}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781540497}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1348040248}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146200125}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.562 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808185229}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3022134111}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2239194682}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=864364970}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1339507824}) 2024-09-11T03:08:03.563 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3426091071}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064252309}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.347+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1271110200}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209863324}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527919739}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.564 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=530321835}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2679663992}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3551926709}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=889241405}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.565 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1182801254}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601562224}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.351+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2578853943}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573154084}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968414625}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=927875252}) 2024-09-11T03:08:03.566 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1699510468}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1807540788}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1197666660}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2275699213}) 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.567 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118886145}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2918154059}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685172671}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=489673608}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1670459953}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.568 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=274256150}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2244077481}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=922570233}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1863213467}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331573593}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.355+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1593709279}) 2024-09-11T03:08:03.569 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132782108}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943913380}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2374045118}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2341771425}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.570 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3633988994}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2240444271}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1766225446}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673084886}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2115911385}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.571 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392218592}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444587139}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.359+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2866610227}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=569224698}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=558897186}) 2024-09-11T03:08:03.572 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1907784141}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4203130828}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2407781152}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3582740308}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026715456}) 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.573 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206370577}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4108287101}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2328951502}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=64090177}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.574 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3492180030}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1490778489}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157156623}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2808586866}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.363+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1740728139}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420261645}) 2024-09-11T03:08:03.575 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4270360249}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3298603517}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2369190653}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1404603024}) 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.576 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=216308981}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117717272}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=435570302}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188900402}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024833617}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.577 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3114566776}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1898448064}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1124662571}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1104498602}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3893946765}) 2024-09-11T03:08:03.578 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758513999}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.367+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508743397}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218399167}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2031778402}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=524873371}) 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.579 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1690352064}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1822847414}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3326203789}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2676838917}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122008771}) 2024-09-11T03:08:03.580 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2520049856}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499087923}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698637569}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257661509}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2330516438}) 2024-09-11T03:08:03.581 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2700349257}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=958708644}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3207649400}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=214510773}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.371+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1240091302}) 2024-09-11T03:08:03.582 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678212278}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204422516}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3567861489}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1830351903}) 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.583 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1133896802}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=938663525}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1236822479}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813701745}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.375+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1533415086}) 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.584 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832757992}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=340411544}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=43926308}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=365918642}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.585 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4063192899}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=743626496}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2197089769}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766113210}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1004697248}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.586 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4205581344}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=73485780}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1612838046}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1034519184}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4005817413}) 2024-09-11T03:08:03.587 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3406948517}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3993419934}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817130208}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.379+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3428652561}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1957357846}) 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.588 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3455465649}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4030685253}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=256525254}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=385301443}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.589 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2035263337}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1944395509}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355053793}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.383+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3895734634}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914712682}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718522534}) 2024-09-11T03:08:03.590 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1960052258}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=72953401}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1123014790}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2630357109}) 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.591 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2023504759}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570240136}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2837859654}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4291493849}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752813845}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.592 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=436748480}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1951655135}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=170589553}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1953117151}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1956740568}) 2024-09-11T03:08:03.593 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670075190}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.387+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1440482152}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499832390}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732732778}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2123045625}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.594 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052821349}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3954052535}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3748775488}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=88436633}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254131045}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3106442277}) 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.595 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760688712}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.391+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258058560}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=79849716}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3697171020}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.596 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=626629773}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2043393270}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2833732525}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299801655}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2552957188}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.597 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679837857}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906798813}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1939903047}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1644668036}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=989683559}) 2024-09-11T03:08:03.598 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1659101644}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1014503618}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978410992}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.395+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3987610339}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=767843010}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.599 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215639427}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1763615643}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1098132204}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4246613342}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1887170186}) 2024-09-11T03:08:03.600 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1989155988}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327761658}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3857411853}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990282767}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132303199}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.601 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3438354349}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=286226663}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3073521438}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2009718370}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3259226865}) 2024-09-11T03:08:03.602 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648541938}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.399+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=785432815}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240541602}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089447198}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3097182770}) 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.603 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=739909817}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3991293207}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2843649221}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=896735433}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670125522}) 2024-09-11T03:08:03.604 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1749588242}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147598216}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=725621431}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=540627374}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.403+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2183140493}) 2024-09-11T03:08:03.605 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1971849131}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1595624658}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4168553822}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186941377}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2809268316}) 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.606 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568983942}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586800468}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=120231890}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4125961793}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2241464990}) 2024-09-11T03:08:03.607 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1506220407}) 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1601997336}) 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792771061}) 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.407+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4152279204}) 2024-09-11T03:08:03.608 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955121232}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3512142795}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=308960705}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4256015633}) 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.609 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.610 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=728576369}) 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2353708605}) 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1758744420}) 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616556655}) 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1034601229}) 2024-09-11T03:08:03.612 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322155253}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1216099873}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2746981917}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=529483562}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.613 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2626009403}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2503009270}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3595948709}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2624103499}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.411+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=858504097}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181581342}) 2024-09-11T03:08:03.614 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711908930}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2258690031}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=354116675}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2463950685}) 2024-09-11T03:08:03.615 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1866023287}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1187898783}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246904398}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=92203378}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565185200}) 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.616 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3571218502}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267548455}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2630677728}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2134295976}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.415+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.617 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3448113885}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1149452158}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573077919}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3279100730}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=825010177}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2779596195}) 2024-09-11T03:08:03.618 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1897298351}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=907474224}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1649535294}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3169206928}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.619 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=651150572}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422448698}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2977667727}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839503502}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015333375}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3234771277}) 2024-09-11T03:08:03.620 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4048431509}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3347977939}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3373365855}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.419+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=258594538}) 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.621 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531265803}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1264526632}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=732771372}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=223867225}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2078902407}) 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.622 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4123176125}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1257586055}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3550714647}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=962611368}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793129425}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3747962973}) 2024-09-11T03:08:03.623 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911283935}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4286636799}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1897413211}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2132633670}) 2024-09-11T03:08:03.624 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.423+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=816534799}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4094720413}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1968231386}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273113327}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3982508914}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.625 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3445958260}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188912599}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806760874}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2050636088}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3213873552}) 2024-09-11T03:08:03.626 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3868970968}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=579127698}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370908323}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.427+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4113427821}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162484063}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.627 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588147350}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3539991439}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=534847750}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1318698462}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=944688949}) 2024-09-11T03:08:03.628 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2646034077}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698644428}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=635259226}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536527287}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2085379866}) 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.629 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=836796761}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1006731796}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2972457082}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=329651695}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.630 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2884165154}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742153853}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1109900295}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153544278}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=827306173}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.431+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2364075638}) 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.631 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3562785705}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=116719302}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1322033908}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3776009864}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2755953402}) 2024-09-11T03:08:03.632 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=574566979}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934913558}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515221828}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3489467220}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1218052236}) 2024-09-11T03:08:03.633 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2661908326}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1237424599}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3397231721}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3126472240}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2194833170}) 2024-09-11T03:08:03.634 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3877773028}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444470184}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1682257174}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2463391252}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.635 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=679134241}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=177581886}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4248191881}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=120292224}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.435+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2487783655}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.636 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205291942}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787206352}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=82101807}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=939843702}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1871014402}) 2024-09-11T03:08:03.637 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3475836325}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4189781021}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444460692}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1143017333}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950547257}) 2024-09-11T03:08:03.638 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=355149955}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1908813298}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2461131241}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=75333248}) 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.639 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3147130661}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1363201038}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123251726}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.439+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3022298374}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249655262}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3031432853}) 2024-09-11T03:08:03.640 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2474036675}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3189740464}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1713996820}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=960416690}) 2024-09-11T03:08:03.641 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3166706374}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781256342}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3444322947}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=808936398}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107176790}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.642 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1568020083}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2084269991}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2567363441}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2579747260}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.643 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=746087876}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540764670}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.443+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1824184499}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1052670053}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2277844383}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3437202437}) 2024-09-11T03:08:03.644 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=491370265}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=425999081}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=217462790}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=571782173}) 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.645 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=107611490}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2345177310}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=634729423}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1315217224}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1332596963}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.646 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=774730223}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3437596319}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=28622677}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=928980608}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803608564}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.447+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1329899047}) 2024-09-11T03:08:03.647 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030038529}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768794449}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1430409812}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=710336224}) 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.648 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3663273997}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2136496307}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3669027524}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723213851}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2596385806}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.649 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176774467}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=879635445}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3264296646}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2563292314}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2860622633}) 2024-09-11T03:08:03.650 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=248550187}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=198995553}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263458620}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.451+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=116691894}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340972384}) 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.651 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=188097123}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1743125400}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1828869915}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2738156432}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.652 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3253320486}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2811446251}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239457791}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=401536998}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1862328235}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.653 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2511345983}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3852228831}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1238810261}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=934399575}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1844344559}) 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.654 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2884555408}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518927600}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.455+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4075478456}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884901582}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3856305080}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1511848275}) 2024-09-11T03:08:03.655 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=857557456}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3258836284}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=318753365}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2134553395}) 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.656 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=803296661}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3178888605}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028659028}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560144307}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1552842368}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.657 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=942094485}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2735934283}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2961496388}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=56063619}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1168744036}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.459+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=291647850}) 2024-09-11T03:08:03.658 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1628777792}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2551794096}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3606043468}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2342081860}) 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.659 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=960133511}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=962677960}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4290918420}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916252931}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2003715008}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=1} crc {logm=645557426,mdsmap=92030302,mgr=3539466285,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3389078134,mon_config_key=2106052021,monmap=1217224211}) 2024-09-11T03:08:03.660 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9} crc {logm=2182871778,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3389078134,mon_config_key=2833664290}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=70,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7} crc {logm=4089515235,mdsmap=92030302,mgr=492734241,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3389078134,mon_config_key=1532443479}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8} crc {logm=1500178934,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3389078134,mon_config_key=3299462871}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11} crc {logm=866070335,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=4113798025,mon_config_key=2106052021}) 2024-09-11T03:08:03.661 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=4161664370,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730519,mgr_metadata=3513241537,mgrstat=3866430353,mon_config_key=1000899325}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11} crc {logm=855563158,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3389078134,mon_config_key=2997118629}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,nvmeofgw=1} crc {logm=3120977474,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgrstat=1021870221,mon_config_key=2874876547,monmap=2495409540,nvmeofgw=773687688}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.463+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8} crc {logm=1969625619,mdsmap=3491474967,mgr=492734240,mgr_command_descs=2308730518,mgr_metadata=3513241537,mgrstat=3389078134,mon_config_key=164477497}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=51} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=894316331,osdmap=375242011}) 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.662 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {nvmeofgw=1,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=50} crc {nvmeofgw=1228700967,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=2287811531,osdmap=2035934399}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=50} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=603024958,osdmap=1070097665}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=39,osdmap=53} crc {nvmeofgw=3140514009,osd_metadata=4278568289,osd_pg_creating=3096585049,osd_snap=164605997,osdmap=4232337436}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_snap=41,osdmap=51} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_snap=958511064,osdmap=3822368876}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=42,osdmap=50} crc {nvmeofgw=3140514009,osd_metadata=1737952455,osd_pg_creating=3096585049,osd_snap=1692699601,osdmap=1814399588}) 2024-09-11T03:08:03.663 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=2577444928,osdmap=4125524919}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=49} crc {nvmeofgw=3140514009,osd_metadata=1477139452,osd_pg_creating=3096585049,osd_snap=1863612498,osdmap=2188558707}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {nvmeofgw=2,osd_metadata=4,osd_snap=39,osdmap=55} crc {nvmeofgw=3140514009,osd_metadata=2334082449,osd_snap=3968404291,osdmap=779253528}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.467+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=101465362}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1404920436}) 2024-09-11T03:08:03.664 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2691464116}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2824701240}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2807771119}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2320699191}) 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.665 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=529097366}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3653627350}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3509459300}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1459894326}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4029778212}) 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.666 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3244875848}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2810430825}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3937399127}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1175864798}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1371732630}) 2024-09-11T03:08:03.667 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3432565183}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.471+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=918014972}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1770288560}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2562322476}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3916510783}) 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.668 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3088220280}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2926544181}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2981599261}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4001903770}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=81453602}) 2024-09-11T03:08:03.669 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2189420567}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2153603884}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4240568585}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1625602621}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1230105856}) 2024-09-11T03:08:03.670 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=929933682}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2642552377}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2114173549}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2151794507}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.475+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2157955971}) 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.671 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2724295745}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=328387238}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=30420980}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=461012436}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.672 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=522461914}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2545708029}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=427759929}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2531630848}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4110582213}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1280859100}) 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.673 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2247125619}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2633778015}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3767858621}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1167238458}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.674 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3297472936}) 2024-09-11T03:08:03.675 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.675 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.675 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=398727316}) 2024-09-11T03:08:03.675 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.675 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1121230048}) 2024-09-11T03:08:03.675 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.479+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3291928216}) 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=830441386}) 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=349923543}) 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1125148242}) 2024-09-11T03:08:03.684 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=727186865}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=598806974}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2134217485}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1852101846}) 2024-09-11T03:08:03.685 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3706047009}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.483+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2966991609}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1892821220}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3199733632}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1531768326}) 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.686 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2986732648}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4218404376}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3762576973}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1582555301}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2033341837}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.487+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2369746457}) 2024-09-11T03:08:03.687 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3937654197}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1157472489}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1569084076}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3672392509}) 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.688 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1423867965}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3622339244}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2055614352}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4223735537}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.491+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2675385722}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.689 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=700833213}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=65} crc {osdmap=1008158857}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=66} crc {osdmap=781598497}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=64} crc {osdmap=2583405275}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=64} crc {osdmap=2208032432}) 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.690 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.691 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=65} crc {osdmap=844027330}) 2024-09-11T03:08:03.691 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.691 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.691 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=63} crc {osdmap=3184612271}) 2024-09-11T03:08:03.691 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-11T03:08:03.691 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1912853944}) 2024-09-11T03:08:03.691 INFO:tasks.ceph.mon.f.smithi190.stderr:2024-09-11T03:08:03.507+0000 7f3f159ec640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=62} crc {osdmap=16995183}) 2024-09-11T03:08:04.114 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-09-11T03:08:04.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient: get_monmap_and_config 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient: build_initial_monmap 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient: monmap: 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:04.258980+0000 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:04.258980+0000 2024-09-11T03:08:04.256 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:04.257 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 Processor -- start 2024-09-11T03:08:04.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 -- start start 2024-09-11T03:08:04.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient: init 2024-09-11T03:08:04.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:04.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient: _add_conns ranks=[5,6,3,4,10,7,9,8,2,1,0] 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f141c09b8c0 0x7f141c09bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): picked mon.noname-h con 0x7f141c09b8c0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f141c0a1540 0x7f141c0a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): picked mon.noname-d con 0x7f141c0a1540 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0a3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): picked mon.noname-g con 0x7f141c0a3730 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f141c0a4b20 con 0x7f141c0a3730 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f141c0a4cf0 con 0x7f141c09b8c0 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f141c09fa20 con 0x7f141c0a1540 2024-09-11T03:08:04.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f1421ca1640 10 monclient(hunting): authenticate will time out at 2038.139093s 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f141c0a1540 0x7f141c0a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f141c0a1540 0x7f141c0a31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54508/0 (socket says 172.21.15.98:54508) 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 -- 172.21.15.98:0/3198532511 learned_addr learned my addr 172.21.15.98:0/3198532511 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): get_auth_request con 0x7f141c0a1540 auth_method 0 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141bfff640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0a3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141b7fe640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f141c09b8c0 0x7f141c09bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141bfff640 10 monclient(hunting): get_auth_request con 0x7f141c0a3730 auth_method 0 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient(hunting): handle_auth_done global_id 84732 payload 274 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 -- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f141c09b8c0 msgr2=0x7f141c09bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f141c09b8c0 0x7f141c09bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 -- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 msgr2=0x7f141c0a3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0a3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient: _finish_hunting 0 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 monclient: found mon.noname-d 2024-09-11T03:08:04.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141affd640 1 -- 172.21.15.98:0/3198532511 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c0a0880 con 0x7f141c0a1540 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141b7fe640 10 monclient: get_auth_request con 0x7f141c09b8c0 auth_method 0 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.255+0000 7f141b7fe640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f141c09b8c0 0x7f141c09bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141affd640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f141c0a1540 0x7f141c0a31f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f140c009eb0 tx=0x7f140c01c8f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=46b62bd029156941 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 1 -- 172.21.15.98:0/3198532511 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f140c0090d0 con 0x7f141c0a1540 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: dump: 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.270 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:04.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 1 -- 172.21.15.98:0/3198532511 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140c0a6950 con 0x7f141c0a1540 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 5 monclient: authenticate success, global_id 84732 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 1 -- 172.21.15.98:0/3198532511 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f140c0af480 con 0x7f141c0a1540 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 4 set_mon_vals no callback set 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f141a7fc640 10 monclient: dump: 2024-09-11T03:08:04.272 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.273 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 10 monclient: shutdown 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 -- 172.21.15.98:0/3198532511 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f141c0a1540 msgr2=0x7f141c0a31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f141c0a1540 0x7f141c0a31f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f140c009eb0 tx=0x7f140c01c8f0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 -- 172.21.15.98:0/3198532511 shutdown_connections 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0a3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f141c0a1540 0x7f141c0a31f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 --2- 172.21.15.98:0/3198532511 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f141c09b8c0 0x7f141c09bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 -- 172.21.15.98:0/3198532511 >> 172.21.15.98:0/3198532511 conn(0x7f141c0a86f0 msgr2=0x7f141c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 -- 172.21.15.98:0/3198532511 shutdown_connections 2024-09-11T03:08:04.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 -- 172.21.15.98:0/3198532511 wait complete. 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 10 monclient: build_initial_monmap 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 10 monclient: monmap: 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:04.275 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 librados: starting msgr at 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.259+0000 7f1421ca1640 1 librados: starting objecter 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 Processor -- start 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 -- start start 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 librados: setting wanted keys 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 librados: calling monclient init 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: init 2024-09-11T03:08:04.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _add_conns ranks=[7,4,3,2,0,8,1,6,5] 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f141c09b8c0 0x7f141c138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): picked mon.noname-i con 0x7f141c09b8c0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f141c0a1540 0x7f141c138f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): picked mon.noname-c con 0x7f141c0a1540 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): picked mon.noname-g con 0x7f141c0a3730 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f141c143080 con 0x7f141c0a3730 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f141c142f00 con 0x7f141c0a1540 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f141c143200 con 0x7f141c09b8c0 2024-09-11T03:08:04.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient(hunting): authenticate will time out at 2038.144605s 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f141c0a1540 0x7f141c138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f141c0a1540 0x7f141c138f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57794/0 (socket says 172.21.15.98:57794) 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 -- 172.21.15.98:0/2081750120 learned_addr learned my addr 172.21.15.98:0/2081750120 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): get_auth_request con 0x7f141c0a1540 auth_method 0 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f141c09b8c0 0x7f141c138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 10 monclient(hunting): get_auth_request con 0x7f141c0a3730 auth_method 0 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 10 monclient(hunting): get_auth_request con 0x7f141c09b8c0 auth_method 0 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient(hunting): handle_auth_done global_id 24934 payload 824 2024-09-11T03:08:04.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 -- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f141c09b8c0 msgr2=0x7f141c138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f141c09b8c0 0x7f141c138a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 -- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 msgr2=0x7f141c0707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient: _finish_hunting 0 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 monclient: found mon.noname-c 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 -- 172.21.15.98:0/2081750120 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c09acd0 con 0x7f141c0a1540 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141affd640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f141c0a1540 0x7f141c138f60 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f140c01c720 tx=0x7f140c019690 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 client.?.objecter ms_handle_connect 0x7f141c0a1540 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 1 -- 172.21.15.98:0/2081750120 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f140c0a6770 con 0x7f141c0a1540 2024-09-11T03:08:04.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: dump: 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.281 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: _finish_auth 0 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 1 -- 172.21.15.98:0/2081750120 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140c0039a0 con 0x7f141c0a1540 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 1 -- 172.21.15.98:0/2081750120 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f140c0a6b00 con 0x7f141c0a1540 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 5 monclient: authenticate success, global_id 24934 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:04.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 monclient: dump: 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.283 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1419ffb640 4 set_mon_vals no callback set 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _renew_subs 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141c073f50 con 0x7f141c0a1540 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 client.24934.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _renew_subs 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f141c0a5700 con 0x7f141c0a1540 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 librados: init done 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141c13ca90 con 0x7f141c0a1540 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 1 -- 172.21.15.98:0/2081750120 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f140c0c2060 con 0x7f141c0a1540 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f140803df40 0x7f1408040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 1 -- 172.21.15.98:0/2081750120 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f140c0b6070 con 0x7f141c0a1540 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 client.24934.objecter ms_dispatch 0x7f141c000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 3 client.24934.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 3 client.24934.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 20 client.24934.objecter dump_active .. 0 homeless 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f140803df40 0x7f1408040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 10 monclient: get_auth_request con 0x7f140803df40 auth_method 0 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f141b7fe640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f140803df40 0x7f1408040400 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7f141000a9a0 tx=0x7f1410008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.263+0000 7f1403fff640 10 client.24934.objecter ms_handle_connect 0x7f140803df40 2024-09-11T03:08:04.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.267+0000 7f1403fff640 1 -- 172.21.15.98:0/2081750120 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f140c016600 con 0x7f141c0a1540 2024-09-11T03:08:04.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.267+0000 7f1403fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.267+0000 7f1403fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:04.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.503+0000 7f1421ca1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-11T03:08:04.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.503+0000 7f1421ca1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-11T03:08:04.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.503+0000 7f1421ca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:04.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.503+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f141c09bca0 con 0x7f141c0a1540 2024-09-11T03:08:04.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.503+0000 7f1403fff640 1 -- 172.21.15.98:0/2081750120 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f140c0aedd0 con 0x7f141c0a1540 2024-09-11T03:08:04.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.503+0000 7f1403fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-11T03:08:04.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.503+0000 7f1403fff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:04.506 INFO:teuthology.orchestra.run.smithi098.stdout:{"election_epoch":122,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"f","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","modified":"2024-09-11T02:49:17.084567Z","created":"2024-09-11T02:49:17.084567Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3301","nonce":0},{"type":"v1","addr":"172.21.15.190:6790","nonce":0}]},"addr":"172.21.15.190:6790/0","public_addr":"172.21.15.190:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3302","nonce":0},{"type":"v1","addr":"172.21.15.98:6791","nonce":0}]},"addr":"172.21.15.98:6791/0","public_addr":"172.21.15.98:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3302","nonce":0},{"type":"v1","addr":"172.21.15.190:6791","nonce":0}]},"addr":"172.21.15.190:6791/0","public_addr":"172.21.15.190:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3303","nonce":0},{"type":"v1","addr":"172.21.15.98:6792","nonce":0}]},"addr":"172.21.15.98:6792/0","public_addr":"172.21.15.98:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3303","nonce":0},{"type":"v1","addr":"172.21.15.190:6792","nonce":0}]},"addr":"172.21.15.190:6792/0","public_addr":"172.21.15.190:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3304","nonce":0},{"type":"v1","addr":"172.21.15.98:6793","nonce":0}]},"addr":"172.21.15.98:6793/0","public_addr":"172.21.15.98:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 10 librados: watch_flush enter 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 10 librados: watch_flush exit 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 20 client.24934.objecter shutdown clearing up homeless session... 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 10 client.24934.objecter successfully canceled tick 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f140803df40 msgr2=0x7f1408040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f140803df40 0x7f1408040400 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7f141000a9a0 tx=0x7f1410008040 comp rx=0 tx=0).stop 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 10 monclient: shutdown 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f141c0a1540 msgr2=0x7f141c138f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f141c0a1540 0x7f141c138f60 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f140c01c720 tx=0x7f140c019690 comp rx=0 tx=0).stop 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 shutdown_connections 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f140803df40 0x7f1408040400 unknown :-1 s=CLOSED pgs=3709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f141c0a1540 0x7f141c138f60 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f141c0a3730 0x7f141c0707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 --2- 172.21.15.98:0/2081750120 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f141c09b8c0 0x7f141c138a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 >> 172.21.15.98:0/2081750120 conn(0x7f141c0a86f0 msgr2=0x7f141c13cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 shutdown_connections 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 -- 172.21.15.98:0/2081750120 wait complete. 2024-09-11T03:08:04.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.507+0000 7f1421ca1640 1 librados: shutdown 2024-09-11T03:08:04.524 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-09-11T03:08:04.524 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T03:08:04.533 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T03:08:04.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.643+0000 7fef47765640 10 monclient: get_monmap_and_config 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.643+0000 7fef47765640 10 monclient: build_initial_monmap 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.643+0000 7fef47765640 10 monclient: monmap: 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:04.647990+0000 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:04.647990+0000 2024-09-11T03:08:04.645 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:04.646 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.643+0000 7fef47765640 1 Processor -- start 2024-09-11T03:08:04.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.643+0000 7fef47765640 1 -- start start 2024-09-11T03:08:04.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.643+0000 7fef47765640 10 monclient: init 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient: _add_conns ranks=[9,0,7,1,4,6,2,3,10,5,8] 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef400a58c0 0x7fef400a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient(hunting): picked mon.noname-a con 0x7fef400a58c0 addr v2:172.21.15.98:3300/0 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef4006fa90 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fef400a61e0 0x7fef40139da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient(hunting): picked mon.noname-i con 0x7fef400a61e0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef4006f7b0 con 0x7fef4006fa90 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fef40070ca0 con 0x7fef400a58c0 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fef400a7870 con 0x7fef400a61e0 2024-09-11T03:08:04.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient(hunting): authenticate will time out at 2038.528233s 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef400a58c0 0x7fef400a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef44d00640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fef4006fa90 0x7fef40139690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39694/0 (socket says 172.21.15.98:39694) 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fef400a58c0 0x7fef400a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54036/0 (socket says 172.21.15.98:54036) 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef44d00640 1 -- 172.21.15.98:0/3983330221 learned_addr learned my addr 172.21.15.98:0/3983330221 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): get_auth_request con 0x7fef400a58c0 auth_method 0 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient(hunting): handle_auth_done global_id 104184 payload 274 2024-09-11T03:08:04.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45d02640 1 --2- 172.21.15.98:0/3983330221 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fef400a61e0 0x7fef40139da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 -- 172.21.15.98:0/3983330221 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fef400a61e0 msgr2=0x7fef40139da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 --2- 172.21.15.98:0/3983330221 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fef400a61e0 0x7fef40139da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 -- 172.21.15.98:0/3983330221 >> v1:172.21.15.98:6789/0 conn(0x7fef4006fa90 legacy=0x7fef40139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient: _finish_hunting 0 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 monclient: found mon.noname-a 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 -- 172.21.15.98:0/3983330221 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef400706e0 con 0x7fef400a58c0 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 1 -- 172.21.15.98:0/3983330221 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2290584471 0 0) 0x7fef4006f7b0 con 0x7fef4006fa90 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45d02640 10 monclient: get_auth_request con 0x7fef400a61e0 auth_method 0 2024-09-11T03:08:04.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45d02640 1 --2- 172.21.15.98:0/3983330221 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fef400a61e0 0x7fef40139da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef45501640 1 --2- 172.21.15.98:0/3983330221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a58c0 0x7fef400a5ca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fef300097c0 tx=0x7fef3001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9aa2cd6405ffd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 1 -- 172.21.15.98:0/3983330221 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fef300b0070 con 0x7fef400a58c0 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: dump: 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.654 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: _finish_auth 0 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 5 monclient: authenticate success, global_id 104184 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 1 -- 172.21.15.98:0/3983330221 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef300aa810 con 0x7fef400a58c0 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:04.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 1 -- 172.21.15.98:0/3983330221 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fef300aae30 con 0x7fef400a58c0 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 4 set_mon_vals no callback set 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef37fff640 10 monclient: dump: 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 10 monclient: shutdown 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- 172.21.15.98:0/3983330221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a58c0 msgr2=0x7fef400a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 --2- 172.21.15.98:0/3983330221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a58c0 0x7fef400a5ca0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fef300097c0 tx=0x7fef3001c330 comp rx=0 tx=0).stop 2024-09-11T03:08:04.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- 172.21.15.98:0/3983330221 shutdown_connections 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 --2- 172.21.15.98:0/3983330221 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fef400a61e0 0x7fef40139da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 --2- 172.21.15.98:0/3983330221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a58c0 0x7fef400a5ca0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- 172.21.15.98:0/3983330221 >> 172.21.15.98:0/3983330221 conn(0x7fef400aa4b0 msgr2=0x7fef400a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- 172.21.15.98:0/3983330221 shutdown_connections 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.647+0000 7fef47765640 1 -- 172.21.15.98:0/3983330221 wait complete. 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: build_initial_monmap 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: monmap: 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:04.658 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 librados: starting msgr at 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 librados: starting objecter 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 Processor -- start 2024-09-11T03:08:04.659 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 -- start start 2024-09-11T03:08:04.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 librados: setting wanted keys 2024-09-11T03:08:04.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 librados: calling monclient init 2024-09-11T03:08:04.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: init 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _add_conns ranks=[4,3,0,6,1,8,7,5,2] 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fef4006fa90 0x7fef40120500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): picked mon.noname-c con 0x7fef4006fa90 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef400a58c0 0x7fef40120a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): picked mon.noname-g con 0x7fef400a58c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a61e0 0x7fef40120f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): picked mon.noname-a con 0x7fef400a61e0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef40071c70 con 0x7fef400a61e0 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fef40071af0 con 0x7fef400a58c0 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fef4013a2e0 con 0x7fef4006fa90 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient(hunting): authenticate will time out at 2038.533108s 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a61e0 0x7fef40120f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a61e0 0x7fef40120f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54044/0 (socket says 172.21.15.98:54044) 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 -- 172.21.15.98:0/6604227 learned_addr learned my addr 172.21.15.98:0/6604227 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): get_auth_request con 0x7fef400a61e0 auth_method 0 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fef4006fa90 0x7fef40120500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient(hunting): get_auth_request con 0x7fef4006fa90 auth_method 0 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef44d00640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef400a58c0 0x7fef40120a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef44d00640 10 monclient(hunting): get_auth_request con 0x7fef400a58c0 auth_method 0 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef44d00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef44d00640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef44d00640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient(hunting): handle_auth_done global_id 104202 payload 824 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 -- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fef4006fa90 msgr2=0x7fef40120500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fef4006fa90 0x7fef40120500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 -- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef400a58c0 msgr2=0x7fef40120a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef400a58c0 0x7fef40120a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient: _finish_hunting 0 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 monclient: found mon.noname-a 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 -- 172.21.15.98:0/6604227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef400a3860 con 0x7fef400a61e0 2024-09-11T03:08:04.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45d02640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a61e0 0x7fef40120f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef3800cd80 tx=0x7fef3800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 client.?.objecter ms_handle_connect 0x7fef400a61e0 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 1 -- 172.21.15.98:0/6604227 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fef3800f8f0 con 0x7fef400a61e0 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: dump: 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.669 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: _finish_auth 0 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 5 monclient: authenticate success, global_id 104202 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _renew_subs 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 1 -- 172.21.15.98:0/6604227 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3800fad0 con 0x7fef400a61e0 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef401233e0 con 0x7fef400a61e0 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 1 -- 172.21.15.98:0/6604227 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fef3800fc80 con 0x7fef400a61e0 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 client.104202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef377fe640 4 set_mon_vals no callback set 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 monclient: dump: 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.671 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _renew_subs 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef400a3db0 con 0x7fef400a61e0 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 librados: init done 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 1 -- 172.21.15.98:0/6604227 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fef38020030 con 0x7fef400a61e0 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.672 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef40072ae0 con 0x7fef400a61e0 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fef2403df70 0x7fef24040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 1 -- 172.21.15.98:0/6604227 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fef3804fe40 con 0x7fef400a61e0 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 10 client.104202.objecter ms_dispatch 0x7fef40000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 3 client.104202.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 3 client.104202.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef35ffb640 20 client.104202.objecter dump_active .. 0 homeless 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fef2403df70 0x7fef24040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.651+0000 7fef45501640 10 monclient: get_auth_request con 0x7fef2403df70 auth_method 0 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.655+0000 7fef45501640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fef2403df70 0x7fef24040430 secure :-1 s=READY pgs=3710 cs=0 l=1 rev1=1 crypto rx=0x7fef300ad000 tx=0x7fef300a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.655+0000 7fef35ffb640 10 client.104202.objecter ms_handle_connect 0x7fef2403df70 2024-09-11T03:08:04.673 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.655+0000 7fef35ffb640 1 -- 172.21.15.98:0/6604227 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fef400a3860 con 0x7fef400a61e0 2024-09-11T03:08:04.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.655+0000 7fef35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.655+0000 7fef35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:04.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.823+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef400a5ca0 con 0x7fef2403df70 2024-09-11T03:08:04.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef35ffb640 1 -- 172.21.15.98:0/6604227 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114038 (secure 0 0 0) 0x7fef400a5ca0 con 0x7fef2403df70 2024-09-11T03:08:04.831 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T03:08:04.832 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T03:08:04.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 10 librados: watch_flush enter 2024-09-11T03:08:04.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 10 librados: watch_flush exit 2024-09-11T03:08:04.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 20 client.104202.objecter shutdown clearing up homeless session... 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 10 client.104202.objecter successfully canceled tick 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fef2403df70 msgr2=0x7fef24040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fef2403df70 0x7fef24040430 secure :-1 s=READY pgs=3710 cs=0 l=1 rev1=1 crypto rx=0x7fef300ad000 tx=0x7fef300a6000 comp rx=0 tx=0).stop 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 10 monclient: shutdown 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a61e0 msgr2=0x7fef40120f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a61e0 0x7fef40120f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fef3800cd80 tx=0x7fef3800a650 comp rx=0 tx=0).stop 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 shutdown_connections 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fef2403df70 0x7fef24040430 unknown :-1 s=CLOSED pgs=3710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef400a61e0 0x7fef40120f80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fef400a58c0 0x7fef40120a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 --2- 172.21.15.98:0/6604227 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fef4006fa90 0x7fef40120500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 >> 172.21.15.98:0/6604227 conn(0x7fef400aa4b0 msgr2=0x7fef400a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 shutdown_connections 2024-09-11T03:08:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 -- 172.21.15.98:0/6604227 wait complete. 2024-09-11T03:08:04.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.831+0000 7fef47765640 1 librados: shutdown 2024-09-11T03:08:04.849 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1239,"stamp":"2024-09-11T03:08:04.134359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":451712,"kb_used_data":61920,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565779328,"statfs":{"total":579820584960,"available":579358031872,"internally_reserved":0,"allocated":63406080,"data_stored":35633462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":399039138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003885"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925479+0000","last_change":"2024-09-11T02:51:02.707390+0000","last_active":"2024-09-11T03:07:32.925479+0000","last_peered":"2024-09-11T03:07:32.925479+0000","last_clean":"2024-09-11T03:07:32.925479+0000","last_became_active":"2024-09-11T02:51:02.704466+0000","last_became_peered":"2024-09-11T02:51:02.704466+0000","last_unstale":"2024-09-11T03:07:32.925479+0000","last_undegraded":"2024-09-11T03:07:32.925479+0000","last_fullsized":"2024-09-11T03:07:32.925479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:51:38.363080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939381+0000","last_change":"2024-09-11T02:51:02.751947+0000","last_active":"2024-09-11T03:07:33.939381+0000","last_peered":"2024-09-11T03:07:33.939381+0000","last_clean":"2024-09-11T03:07:33.939381+0000","last_became_active":"2024-09-11T02:51:02.747449+0000","last_became_peered":"2024-09-11T02:51:02.747449+0000","last_unstale":"2024-09-11T03:07:33.939381+0000","last_undegraded":"2024-09-11T03:07:33.939381+0000","last_fullsized":"2024-09-11T03:07:33.939381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:34.946819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.923807+0000","last_change":"2024-09-11T02:51:02.757155+0000","last_active":"2024-09-11T03:07:32.923807+0000","last_peered":"2024-09-11T03:07:32.923807+0000","last_clean":"2024-09-11T03:07:32.923807+0000","last_became_active":"2024-09-11T02:51:02.746704+0000","last_became_peered":"2024-09-11T02:51:02.746704+0000","last_unstale":"2024-09-11T03:07:32.923807+0000","last_undegraded":"2024-09-11T03:07:32.923807+0000","last_fullsized":"2024-09-11T03:07:32.923807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:08:27.349903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938159+0000","last_change":"2024-09-11T02:51:02.703570+0000","last_active":"2024-09-11T03:07:33.938159+0000","last_peered":"2024-09-11T03:07:33.938159+0000","last_clean":"2024-09-11T03:07:33.938159+0000","last_became_active":"2024-09-11T02:51:02.695884+0000","last_became_peered":"2024-09-11T02:51:02.695884+0000","last_unstale":"2024-09-11T03:07:33.938159+0000","last_undegraded":"2024-09-11T03:07:33.938159+0000","last_fullsized":"2024-09-11T03:07:33.938159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:24.055472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930411+0000","last_change":"2024-09-11T02:51:02.604922+0000","last_active":"2024-09-11T03:07:32.930411+0000","last_peered":"2024-09-11T03:07:32.930411+0000","last_clean":"2024-09-11T03:07:32.930411+0000","last_became_active":"2024-09-11T02:51:02.604043+0000","last_became_peered":"2024-09-11T02:51:02.604043+0000","last_unstale":"2024-09-11T03:07:32.930411+0000","last_undegraded":"2024-09-11T03:07:32.930411+0000","last_fullsized":"2024-09-11T03:07:32.930411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:55.319066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932296+0000","last_change":"2024-09-11T02:51:02.574684+0000","last_active":"2024-09-11T03:07:33.932296+0000","last_peered":"2024-09-11T03:07:33.932296+0000","last_clean":"2024-09-11T03:07:33.932296+0000","last_became_active":"2024-09-11T02:51:02.574238+0000","last_became_peered":"2024-09-11T02:51:02.574238+0000","last_unstale":"2024-09-11T03:07:33.932296+0000","last_undegraded":"2024-09-11T03:07:33.932296+0000","last_fullsized":"2024-09-11T03:07:33.932296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:10:59.496994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938047+0000","last_change":"2024-09-11T02:51:02.752301+0000","last_active":"2024-09-11T03:07:33.938047+0000","last_peered":"2024-09-11T03:07:33.938047+0000","last_clean":"2024-09-11T03:07:33.938047+0000","last_became_active":"2024-09-11T02:51:02.745188+0000","last_became_peered":"2024-09-11T02:51:02.745188+0000","last_unstale":"2024-09-11T03:07:33.938047+0000","last_undegraded":"2024-09-11T03:07:33.938047+0000","last_fullsized":"2024-09-11T03:07:33.938047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:05:51.425341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924911+0000","last_change":"2024-09-11T02:51:02.583697+0000","last_active":"2024-09-11T03:07:32.924911+0000","last_peered":"2024-09-11T03:07:32.924911+0000","last_clean":"2024-09-11T03:07:32.924911+0000","last_became_active":"2024-09-11T02:51:02.578172+0000","last_became_peered":"2024-09-11T02:51:02.578172+0000","last_unstale":"2024-09-11T03:07:32.924911+0000","last_undegraded":"2024-09-11T03:07:32.924911+0000","last_fullsized":"2024-09-11T03:07:32.924911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:55:49.570822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939024+0000","last_change":"2024-09-11T02:51:02.671125+0000","last_active":"2024-09-11T03:07:33.939024+0000","last_peered":"2024-09-11T03:07:33.939024+0000","last_clean":"2024-09-11T03:07:33.939024+0000","last_became_active":"2024-09-11T02:51:02.666973+0000","last_became_peered":"2024-09-11T02:51:02.666973+0000","last_unstale":"2024-09-11T03:07:33.939024+0000","last_undegraded":"2024-09-11T03:07:33.939024+0000","last_fullsized":"2024-09-11T03:07:33.939024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:57.810050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925926+0000","last_change":"2024-09-11T02:51:02.746877+0000","last_active":"2024-09-11T03:07:32.925926+0000","last_peered":"2024-09-11T03:07:32.925926+0000","last_clean":"2024-09-11T03:07:32.925926+0000","last_became_active":"2024-09-11T02:51:02.737993+0000","last_became_peered":"2024-09-11T02:51:02.737993+0000","last_unstale":"2024-09-11T03:07:32.925926+0000","last_undegraded":"2024-09-11T03:07:32.925926+0000","last_fullsized":"2024-09-11T03:07:32.925926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:49.770401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932737+0000","last_change":"2024-09-11T02:51:02.692125+0000","last_active":"2024-09-11T03:07:33.932737+0000","last_peered":"2024-09-11T03:07:33.932737+0000","last_clean":"2024-09-11T03:07:33.932737+0000","last_became_active":"2024-09-11T02:51:02.671301+0000","last_became_peered":"2024-09-11T02:51:02.671301+0000","last_unstale":"2024-09-11T03:07:33.932737+0000","last_undegraded":"2024-09-11T03:07:33.932737+0000","last_fullsized":"2024-09-11T03:07:33.932737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:39:47.378191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930877+0000","last_change":"2024-09-11T02:51:02.713974+0000","last_active":"2024-09-11T03:07:32.930877+0000","last_peered":"2024-09-11T03:07:32.930877+0000","last_clean":"2024-09-11T03:07:32.930877+0000","last_became_active":"2024-09-11T02:51:02.712319+0000","last_became_peered":"2024-09-11T02:51:02.712319+0000","last_unstale":"2024-09-11T03:07:32.930877+0000","last_undegraded":"2024-09-11T03:07:32.930877+0000","last_fullsized":"2024-09-11T03:07:32.930877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:12.353704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1353,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930489+0000","last_change":"2024-09-11T02:51:02.678057+0000","last_active":"2024-09-11T03:07:32.930489+0000","last_peered":"2024-09-11T03:07:32.930489+0000","last_clean":"2024-09-11T03:07:32.930489+0000","last_became_active":"2024-09-11T02:51:02.672559+0000","last_became_peered":"2024-09-11T02:51:02.672559+0000","last_unstale":"2024-09-11T03:07:32.930489+0000","last_undegraded":"2024-09-11T03:07:32.930489+0000","last_fullsized":"2024-09-11T03:07:32.930489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:41.716691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1468,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937864+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T03:07:33.937864+0000","last_peered":"2024-09-11T03:07:33.937864+0000","last_clean":"2024-09-11T03:07:33.937864+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T03:07:33.937864+0000","last_undegraded":"2024-09-11T03:07:33.937864+0000","last_fullsized":"2024-09-11T03:07:33.937864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1371,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930535+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T03:07:32.930535+0000","last_peered":"2024-09-11T03:07:32.930535+0000","last_clean":"2024-09-11T03:07:32.930535+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T03:07:32.930535+0000","last_undegraded":"2024-09-11T03:07:32.930535+0000","last_fullsized":"2024-09-11T03:07:32.930535+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932586+0000","last_change":"2024-09-11T02:51:02.712255+0000","last_active":"2024-09-11T03:07:33.932586+0000","last_peered":"2024-09-11T03:07:33.932586+0000","last_clean":"2024-09-11T03:07:33.932586+0000","last_became_active":"2024-09-11T02:51:02.709268+0000","last_became_peered":"2024-09-11T02:51:02.709268+0000","last_unstale":"2024-09-11T03:07:33.932586+0000","last_undegraded":"2024-09-11T03:07:33.932586+0000","last_fullsized":"2024-09-11T03:07:33.932586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:41.994663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937955+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T03:07:33.937955+0000","last_peered":"2024-09-11T03:07:33.937955+0000","last_clean":"2024-09-11T03:07:33.937955+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T03:07:33.937955+0000","last_undegraded":"2024-09-11T03:07:33.937955+0000","last_fullsized":"2024-09-11T03:07:33.937955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1355,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.931087+0000","last_change":"2024-09-11T02:51:02.592214+0000","last_active":"2024-09-11T03:07:32.931087+0000","last_peered":"2024-09-11T03:07:32.931087+0000","last_clean":"2024-09-11T03:07:32.931087+0000","last_became_active":"2024-09-11T02:51:02.587630+0000","last_became_peered":"2024-09-11T02:51:02.587630+0000","last_unstale":"2024-09-11T03:07:32.931087+0000","last_undegraded":"2024-09-11T03:07:32.931087+0000","last_fullsized":"2024-09-11T03:07:32.931087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:26:26.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924408+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T03:07:32.924408+0000","last_peered":"2024-09-11T03:07:32.924408+0000","last_clean":"2024-09-11T03:07:32.924408+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T03:07:32.924408+0000","last_undegraded":"2024-09-11T03:07:32.924408+0000","last_fullsized":"2024-09-11T03:07:32.924408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932714+0000","last_change":"2024-09-11T02:51:02.686284+0000","last_active":"2024-09-11T03:07:33.932714+0000","last_peered":"2024-09-11T03:07:33.932714+0000","last_clean":"2024-09-11T03:07:33.932714+0000","last_became_active":"2024-09-11T02:51:02.679438+0000","last_became_peered":"2024-09-11T02:51:02.679438+0000","last_unstale":"2024-09-11T03:07:33.932714+0000","last_undegraded":"2024-09-11T03:07:33.932714+0000","last_fullsized":"2024-09-11T03:07:33.932714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:30:09.310176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930275+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T03:07:32.930275+0000","last_peered":"2024-09-11T03:07:32.930275+0000","last_clean":"2024-09-11T03:07:32.930275+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T03:07:32.930275+0000","last_undegraded":"2024-09-11T03:07:32.930275+0000","last_fullsized":"2024-09-11T03:07:32.930275+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1363,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924938+0000","last_change":"2024-09-11T02:51:02.707327+0000","last_active":"2024-09-11T03:07:32.924938+0000","last_peered":"2024-09-11T03:07:32.924938+0000","last_clean":"2024-09-11T03:07:32.924938+0000","last_became_active":"2024-09-11T02:51:02.655843+0000","last_became_peered":"2024-09-11T02:51:02.655843+0000","last_unstale":"2024-09-11T03:07:32.924938+0000","last_undegraded":"2024-09-11T03:07:32.924938+0000","last_fullsized":"2024-09-11T03:07:32.924938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:24.067746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932558+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T03:07:33.932558+0000","last_peered":"2024-09-11T03:07:33.932558+0000","last_clean":"2024-09-11T03:07:33.932558+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T03:07:33.932558+0000","last_undegraded":"2024-09-11T03:07:33.932558+0000","last_fullsized":"2024-09-11T03:07:33.932558+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1365,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932452+0000","last_change":"2024-09-11T02:51:02.618073+0000","last_active":"2024-09-11T03:07:33.932452+0000","last_peered":"2024-09-11T03:07:33.932452+0000","last_clean":"2024-09-11T03:07:33.932452+0000","last_became_active":"2024-09-11T02:51:02.586208+0000","last_became_peered":"2024-09-11T02:51:02.586208+0000","last_unstale":"2024-09-11T03:07:33.932452+0000","last_undegraded":"2024-09-11T03:07:33.932452+0000","last_fullsized":"2024-09-11T03:07:33.932452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:45:11.456188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938063+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T03:07:33.938063+0000","last_peered":"2024-09-11T03:07:33.938063+0000","last_clean":"2024-09-11T03:07:33.938063+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T03:07:33.938063+0000","last_undegraded":"2024-09-11T03:07:33.938063+0000","last_fullsized":"2024-09-11T03:07:33.938063+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938507+0000","last_change":"2024-09-11T02:51:02.645107+0000","last_active":"2024-09-11T03:07:33.938507+0000","last_peered":"2024-09-11T03:07:33.938507+0000","last_clean":"2024-09-11T03:07:33.938507+0000","last_became_active":"2024-09-11T02:51:02.642031+0000","last_became_peered":"2024-09-11T02:51:02.642031+0000","last_unstale":"2024-09-11T03:07:33.938507+0000","last_undegraded":"2024-09-11T03:07:33.938507+0000","last_fullsized":"2024-09-11T03:07:33.938507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:05.310788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932866+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T03:07:33.932866+0000","last_peered":"2024-09-11T03:07:33.932866+0000","last_clean":"2024-09-11T03:07:33.932866+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T03:07:33.932866+0000","last_undegraded":"2024-09-11T03:07:33.932866+0000","last_fullsized":"2024-09-11T03:07:33.932866+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938183+0000","last_change":"2024-09-11T02:51:02.753940+0000","last_active":"2024-09-11T03:07:33.938183+0000","last_peered":"2024-09-11T03:07:33.938183+0000","last_clean":"2024-09-11T03:07:33.938183+0000","last_became_active":"2024-09-11T02:51:02.748048+0000","last_became_peered":"2024-09-11T02:51:02.748048+0000","last_unstale":"2024-09-11T03:07:33.938183+0000","last_undegraded":"2024-09-11T03:07:33.938183+0000","last_fullsized":"2024-09-11T03:07:33.938183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:44.448053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939090+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T03:07:33.939090+0000","last_peered":"2024-09-11T03:07:33.939090+0000","last_clean":"2024-09-11T03:07:33.939090+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T03:07:33.939090+0000","last_undegraded":"2024-09-11T03:07:33.939090+0000","last_fullsized":"2024-09-11T03:07:33.939090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924172+0000","last_change":"2024-09-11T02:51:02.654035+0000","last_active":"2024-09-11T03:07:32.924172+0000","last_peered":"2024-09-11T03:07:32.924172+0000","last_clean":"2024-09-11T03:07:32.924172+0000","last_became_active":"2024-09-11T02:51:02.653755+0000","last_became_peered":"2024-09-11T02:51:02.653755+0000","last_unstale":"2024-09-11T03:07:32.924172+0000","last_undegraded":"2024-09-11T03:07:32.924172+0000","last_fullsized":"2024-09-11T03:07:32.924172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:31.581016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925152+0000","last_change":"2024-09-11T02:51:02.615081+0000","last_active":"2024-09-11T03:07:32.925152+0000","last_peered":"2024-09-11T03:07:32.925152+0000","last_clean":"2024-09-11T03:07:32.925152+0000","last_became_active":"2024-09-11T02:51:02.594230+0000","last_became_peered":"2024-09-11T02:51:02.594230+0000","last_unstale":"2024-09-11T03:07:32.925152+0000","last_undegraded":"2024-09-11T03:07:32.925152+0000","last_fullsized":"2024-09-11T03:07:32.925152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:18.916193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925259+0000","last_change":"2024-09-11T02:51:02.736915+0000","last_active":"2024-09-11T03:07:32.925259+0000","last_peered":"2024-09-11T03:07:32.925259+0000","last_clean":"2024-09-11T03:07:32.925259+0000","last_became_active":"2024-09-11T02:51:02.736590+0000","last_became_peered":"2024-09-11T02:51:02.736590+0000","last_unstale":"2024-09-11T03:07:32.925259+0000","last_undegraded":"2024-09-11T03:07:32.925259+0000","last_fullsized":"2024-09-11T03:07:32.925259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:38.031885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930585+0000","last_change":"2024-09-11T02:51:02.678968+0000","last_active":"2024-09-11T03:07:32.930585+0000","last_peered":"2024-09-11T03:07:32.930585+0000","last_clean":"2024-09-11T03:07:32.930585+0000","last_became_active":"2024-09-11T02:51:02.678439+0000","last_became_peered":"2024-09-11T02:51:02.678439+0000","last_unstale":"2024-09-11T03:07:32.930585+0000","last_undegraded":"2024-09-11T03:07:32.930585+0000","last_fullsized":"2024-09-11T03:07:32.930585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:41:16.872342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1350,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932552+0000","last_change":"2024-09-11T02:51:02.705476+0000","last_active":"2024-09-11T03:07:33.932552+0000","last_peered":"2024-09-11T03:07:33.932552+0000","last_clean":"2024-09-11T03:07:33.932552+0000","last_became_active":"2024-09-11T02:51:02.700809+0000","last_became_peered":"2024-09-11T02:51:02.700809+0000","last_unstale":"2024-09-11T03:07:33.932552+0000","last_undegraded":"2024-09-11T03:07:33.932552+0000","last_fullsized":"2024-09-11T03:07:33.932552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:56.654998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938812+0000","last_change":"2024-09-11T02:51:02.698839+0000","last_active":"2024-09-11T03:07:33.938812+0000","last_peered":"2024-09-11T03:07:33.938812+0000","last_clean":"2024-09-11T03:07:33.938812+0000","last_became_active":"2024-09-11T02:51:02.695758+0000","last_became_peered":"2024-09-11T02:51:02.695758+0000","last_unstale":"2024-09-11T03:07:33.938812+0000","last_undegraded":"2024-09-11T03:07:33.938812+0000","last_fullsized":"2024-09-11T03:07:33.938812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:00.606584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1357,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925801+0000","last_change":"2024-09-11T02:51:02.715299+0000","last_active":"2024-09-11T03:07:32.925801+0000","last_peered":"2024-09-11T03:07:32.925801+0000","last_clean":"2024-09-11T03:07:32.925801+0000","last_became_active":"2024-09-11T02:51:02.715041+0000","last_became_peered":"2024-09-11T02:51:02.715041+0000","last_unstale":"2024-09-11T03:07:32.925801+0000","last_undegraded":"2024-09-11T03:07:32.925801+0000","last_fullsized":"2024-09-11T03:07:32.925801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.561287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939022+0000","last_change":"2024-09-11T02:51:02.582470+0000","last_active":"2024-09-11T03:07:33.939022+0000","last_peered":"2024-09-11T03:07:33.939022+0000","last_clean":"2024-09-11T03:07:33.939022+0000","last_became_active":"2024-09-11T02:51:02.581916+0000","last_became_peered":"2024-09-11T02:51:02.581916+0000","last_unstale":"2024-09-11T03:07:33.939022+0000","last_undegraded":"2024-09-11T03:07:33.939022+0000","last_fullsized":"2024-09-11T03:07:33.939022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:11.711933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930286+0000","last_change":"2024-09-11T02:51:02.708269+0000","last_active":"2024-09-11T03:07:32.930286+0000","last_peered":"2024-09-11T03:07:32.930286+0000","last_clean":"2024-09-11T03:07:32.930286+0000","last_became_active":"2024-09-11T02:51:02.708034+0000","last_became_peered":"2024-09-11T02:51:02.708034+0000","last_unstale":"2024-09-11T03:07:32.930286+0000","last_undegraded":"2024-09-11T03:07:32.930286+0000","last_fullsized":"2024-09-11T03:07:32.930286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:19.177182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924808+0000","last_change":"2024-09-11T02:51:02.711915+0000","last_active":"2024-09-11T03:07:32.924808+0000","last_peered":"2024-09-11T03:07:32.924808+0000","last_clean":"2024-09-11T03:07:32.924808+0000","last_became_active":"2024-09-11T02:51:02.684472+0000","last_became_peered":"2024-09-11T02:51:02.684472+0000","last_unstale":"2024-09-11T03:07:32.924808+0000","last_undegraded":"2024-09-11T03:07:32.924808+0000","last_fullsized":"2024-09-11T03:07:32.924808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:56.385712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924685+0000","last_change":"2024-09-11T02:51:02.637360+0000","last_active":"2024-09-11T03:07:32.924685+0000","last_peered":"2024-09-11T03:07:32.924685+0000","last_clean":"2024-09-11T03:07:32.924685+0000","last_became_active":"2024-09-11T02:51:02.603633+0000","last_became_peered":"2024-09-11T02:51:02.603633+0000","last_unstale":"2024-09-11T03:07:32.924685+0000","last_undegraded":"2024-09-11T03:07:32.924685+0000","last_fullsized":"2024-09-11T03:07:32.924685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:25:57.435918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1358,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937869+0000","last_change":"2024-09-11T02:51:02.659574+0000","last_active":"2024-09-11T03:07:33.937869+0000","last_peered":"2024-09-11T03:07:33.937869+0000","last_clean":"2024-09-11T03:07:33.937869+0000","last_became_active":"2024-09-11T02:51:02.659063+0000","last_became_peered":"2024-09-11T02:51:02.659063+0000","last_unstale":"2024-09-11T03:07:33.937869+0000","last_undegraded":"2024-09-11T03:07:33.937869+0000","last_fullsized":"2024-09-11T03:07:33.937869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:28:59.462620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607774,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77408,"kb_used_data":10128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294432,"statfs":{"total":96636764160,"available":96557498368,"internally_reserved":0,"allocated":10371072,"data_stored":5742132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607772,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77404,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294436,"statfs":{"total":96636764160,"available":96557502464,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607775,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77988,"kb_used_data":10708,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293852,"statfs":{"total":96636764160,"available":96556904448,"internally_reserved":0,"allocated":10964992,"data_stored":6332486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607775,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85724,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286116,"statfs":{"total":96636764160,"available":96548982784,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77412,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294428,"statfs":{"total":96636764160,"available":96557494272,"internally_reserved":0,"allocated":10375168,"data_stored":5742137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55776,"kb_used_data":10704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":10960896,"data_stored":6332481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:08:04.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T03:08:04.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient: get_monmap_and_config 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient: build_initial_monmap 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient: monmap: 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:04.970596+0000 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:04.970596+0000 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:04.968 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:04.969 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 1 Processor -- start 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 1 -- start start 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient: init 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient: _add_conns ranks=[6,10,1,0,5,4,9,2,3,8,7] 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c40a58c0 0x7fe4c40a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient(hunting): picked mon.noname-d con 0x7fe4c40a58c0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fe4c4071d70 0x7fe4c40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient(hunting): picked mon.noname-f con 0x7fe4c4071d70 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe4c4137670 addr v1:172.21.15.190:6789/0 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c40a6720 con 0x7fe4c4137670 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c4073600 con 0x7fe4c4071d70 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe4c4073850 con 0x7fe4c40a58c0 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c9d9d640 10 monclient(hunting): authenticate will time out at 2038.850899s 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c40a58c0 0x7fe4c40a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c40a58c0 0x7fe4c40a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54516/0 (socket says 172.21.15.98:54516) 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 -- 172.21.15.98:0/2183417401 learned_addr learned my addr 172.21.15.98:0/2183417401 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): get_auth_request con 0x7fe4c40a58c0 auth_method 0 2024-09-11T03:08:04.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_done global_id 84741 payload 274 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 -- 172.21.15.98:0/2183417401 >> v2:172.21.15.190:3300/0 conn(0x7fe4c4071d70 msgr2=0x7fe4c40a61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c2ffd640 1 --2- 172.21.15.98:0/2183417401 >> v2:172.21.15.190:3300/0 conn(0x7fe4c4071d70 0x7fe4c40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 --2- 172.21.15.98:0/2183417401 >> v2:172.21.15.190:3300/0 conn(0x7fe4c4071d70 0x7fe4c40a61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 -- 172.21.15.98:0/2183417401 >> v1:172.21.15.190:6789/0 conn(0x7fe4c4137670 legacy=0x7fe4c4137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient: _finish_hunting 0 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 monclient: found mon.noname-d 2024-09-11T03:08:04.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c37fe640 1 -- 172.21.15.98:0/2183417401 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c4073b30 con 0x7fe4c40a58c0 2024-09-11T03:08:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c2ffd640 10 monclient: get_auth_request con 0x7fe4c4071d70 auth_method 0 2024-09-11T03:08:04.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.967+0000 7fe4c2ffd640 1 --2- 172.21.15.98:0/2183417401 >> v2:172.21.15.190:3300/0 conn(0x7fe4c4071d70 0x7fe4c40a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:04.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c37fe640 1 --2- 172.21.15.98:0/2183417401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c40a58c0 0x7fe4c40a5ca0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009870 tx=0x7fe4bc01c240 comp rx=0 tx=0).ready entity=mon.6 client_cookie=d67f98d84c1c6498 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 1 -- 172.21.15.98:0/2183417401 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4bc0b0070 con 0x7fe4c40a58c0 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: dump: 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 1 -- 172.21.15.98:0/2183417401 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc0aa810 con 0x7fe4c40a58c0 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 5 monclient: authenticate success, global_id 84741 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 1 -- 172.21.15.98:0/2183417401 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4bc0a9300 con 0x7fe4c40a58c0 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 4 set_mon_vals no callback set 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c27fc640 10 monclient: dump: 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient: shutdown 2024-09-11T03:08:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/2183417401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c40a58c0 msgr2=0x7fe4c40a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/2183417401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c40a58c0 0x7fe4c40a5ca0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009870 tx=0x7fe4bc01c240 comp rx=0 tx=0).stop 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/2183417401 shutdown_connections 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/2183417401 >> v2:172.21.15.190:3300/0 conn(0x7fe4c4071d70 0x7fe4c40a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/2183417401 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c40a58c0 0x7fe4c40a5ca0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/2183417401 >> 172.21.15.98:0/2183417401 conn(0x7fe4c40aa4b0 msgr2=0x7fe4c40a69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/2183417401 shutdown_connections 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/2183417401 wait complete. 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient: build_initial_monmap 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient: monmap: 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:04.985 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 librados: starting msgr at 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 librados: starting objecter 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 Processor -- start 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- start start 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 librados: setting wanted keys 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 librados: calling monclient init 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient: init 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient: _add_conns ranks=[6,3,2,5,1,0,4,8,7] 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c4071d70 0x7fe4c41cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): picked mon.noname-d con 0x7fe4c4071d70 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c40a58c0 0x7fe4c41cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): picked mon.noname-g con 0x7fe4c40a58c0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4c4137670 0x7fe4c41d0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c4137670 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c41424f0 con 0x7fe4c4137670 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c4142370 con 0x7fe4c40a58c0 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fe4c4142670 con 0x7fe4c4071d70 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.971+0000 7fe4c9d9d640 10 monclient(hunting): authenticate will time out at 2038.855988s 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c4071d70 0x7fe4c41cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4c4137670 0x7fe4c41d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c4071d70 0x7fe4c41cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54522/0 (socket says 172.21.15.98:54522) 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 -- 172.21.15.98:0/3987705179 learned_addr learned my addr 172.21.15.98:0/3987705179 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): get_auth_request con 0x7fe4c4071d70 auth_method 0 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 10 monclient(hunting): get_auth_request con 0x7fe4c4137670 auth_method 0 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c40a58c0 0x7fe4c41cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe4c40a58c0 auth_method 0 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_done global_id 84750 payload 824 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 -- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c40a58c0 msgr2=0x7fe4c41cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c40a58c0 0x7fe4c41cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 -- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4c4137670 msgr2=0x7fe4c41d0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4c4137670 0x7fe4c41d0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient: _finish_hunting 0 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 monclient: found mon.noname-d 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 -- 172.21.15.98:0/3987705179 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c4120f80 con 0x7fe4c4071d70 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c3fff640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4c4137670 0x7fe4c41d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c37fe640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c4071d70 0x7fe4c41cf580 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009870 tx=0x7fe4bc018fe0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 client.?.objecter ms_handle_connect 0x7fe4c4071d70 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 1 -- 172.21.15.98:0/3987705179 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4bc0b9070 con 0x7fe4c4071d70 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: dump: 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: _finish_auth 0 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 1 -- 172.21.15.98:0/3987705179 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc004040 con 0x7fe4c4071d70 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 1 -- 172.21.15.98:0/3987705179 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4bc0b0040 con 0x7fe4c4071d70 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 5 monclient: authenticate success, global_id 84750 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 monclient: _renew_subs 2024-09-11T03:08:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c40a36b0 con 0x7fe4c4071d70 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c17fa640 4 set_mon_vals no callback set 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 monclient: dump: 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 client.84750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 monclient: _renew_subs 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:04.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c413cc50 con 0x7fe4c4071d70 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 1 librados: init done 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c4137ae0 con 0x7fe4c4071d70 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 1 -- 172.21.15.98:0/3987705179 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fe4bc0aab80 con 0x7fe4c4071d70 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe4a403df40 0x7fe4a4040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 1 -- 172.21.15.98:0/3987705179 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fe4bc0e83b0 con 0x7fe4c4071d70 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 client.84750.objecter ms_dispatch 0x7fe4c4000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 3 client.84750.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 3 client.84750.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 20 client.84750.objecter dump_active .. 0 homeless 2024-09-11T03:08:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe4a403df40 0x7fe4a4040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 10 monclient: get_auth_request con 0x7fe4a403df40 auth_method 0 2024-09-11T03:08:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4bbfff640 10 client.84750.objecter ms_handle_connect 0x7fe4a403df40 2024-09-11T03:08:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.975+0000 7fe4c2ffd640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe4a403df40 0x7fe4a4040400 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7fe4c413ca50 tx=0x7fe4ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.979+0000 7fe4bbfff640 1 -- 172.21.15.98:0/3987705179 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fe4c4120f80 con 0x7fe4c4071d70 2024-09-11T03:08:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.979+0000 7fe4bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:04.979+0000 7fe4bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:05.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.147+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4c40a8f40 con 0x7fe4a403df40 2024-09-11T03:08:05.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4bbfff640 1 -- 172.21.15.98:0/3987705179 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114038 (secure 0 0 0) 0x7fe4c40a8f40 con 0x7fe4a403df40 2024-09-11T03:08:05.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T03:08:05.156 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 10 librados: watch_flush enter 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 10 librados: watch_flush exit 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 20 client.84750.objecter shutdown clearing up homeless session... 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 10 client.84750.objecter successfully canceled tick 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe4a403df40 msgr2=0x7fe4a4040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe4a403df40 0x7fe4a4040400 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7fe4c413ca50 tx=0x7fe4ac008040 comp rx=0 tx=0).stop 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 10 monclient: shutdown 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:05.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c4071d70 msgr2=0x7fe4c41cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c4071d70 0x7fe4c41cf580 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009870 tx=0x7fe4bc018fe0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 shutdown_connections 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fe4a403df40 0x7fe4a4040400 unknown :-1 s=CLOSED pgs=3711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fe4c4137670 0x7fe4c41d0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fe4c40a58c0 0x7fe4c41cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 --2- 172.21.15.98:0/3987705179 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fe4c4071d70 0x7fe4c41cf580 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 >> 172.21.15.98:0/3987705179 conn(0x7fe4c40aa4b0 msgr2=0x7fe4c40a1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 shutdown_connections 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 -- 172.21.15.98:0/3987705179 wait complete. 2024-09-11T03:08:05.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.155+0000 7fe4c9d9d640 1 librados: shutdown 2024-09-11T03:08:05.172 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1239,"stamp":"2024-09-11T03:08:04.134359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":451712,"kb_used_data":61920,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565779328,"statfs":{"total":579820584960,"available":579358031872,"internally_reserved":0,"allocated":63406080,"data_stored":35633462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":399039138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003885"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925479+0000","last_change":"2024-09-11T02:51:02.707390+0000","last_active":"2024-09-11T03:07:32.925479+0000","last_peered":"2024-09-11T03:07:32.925479+0000","last_clean":"2024-09-11T03:07:32.925479+0000","last_became_active":"2024-09-11T02:51:02.704466+0000","last_became_peered":"2024-09-11T02:51:02.704466+0000","last_unstale":"2024-09-11T03:07:32.925479+0000","last_undegraded":"2024-09-11T03:07:32.925479+0000","last_fullsized":"2024-09-11T03:07:32.925479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:51:38.363080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939381+0000","last_change":"2024-09-11T02:51:02.751947+0000","last_active":"2024-09-11T03:07:33.939381+0000","last_peered":"2024-09-11T03:07:33.939381+0000","last_clean":"2024-09-11T03:07:33.939381+0000","last_became_active":"2024-09-11T02:51:02.747449+0000","last_became_peered":"2024-09-11T02:51:02.747449+0000","last_unstale":"2024-09-11T03:07:33.939381+0000","last_undegraded":"2024-09-11T03:07:33.939381+0000","last_fullsized":"2024-09-11T03:07:33.939381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:34.946819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.923807+0000","last_change":"2024-09-11T02:51:02.757155+0000","last_active":"2024-09-11T03:07:32.923807+0000","last_peered":"2024-09-11T03:07:32.923807+0000","last_clean":"2024-09-11T03:07:32.923807+0000","last_became_active":"2024-09-11T02:51:02.746704+0000","last_became_peered":"2024-09-11T02:51:02.746704+0000","last_unstale":"2024-09-11T03:07:32.923807+0000","last_undegraded":"2024-09-11T03:07:32.923807+0000","last_fullsized":"2024-09-11T03:07:32.923807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:08:27.349903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938159+0000","last_change":"2024-09-11T02:51:02.703570+0000","last_active":"2024-09-11T03:07:33.938159+0000","last_peered":"2024-09-11T03:07:33.938159+0000","last_clean":"2024-09-11T03:07:33.938159+0000","last_became_active":"2024-09-11T02:51:02.695884+0000","last_became_peered":"2024-09-11T02:51:02.695884+0000","last_unstale":"2024-09-11T03:07:33.938159+0000","last_undegraded":"2024-09-11T03:07:33.938159+0000","last_fullsized":"2024-09-11T03:07:33.938159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:24.055472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930411+0000","last_change":"2024-09-11T02:51:02.604922+0000","last_active":"2024-09-11T03:07:32.930411+0000","last_peered":"2024-09-11T03:07:32.930411+0000","last_clean":"2024-09-11T03:07:32.930411+0000","last_became_active":"2024-09-11T02:51:02.604043+0000","last_became_peered":"2024-09-11T02:51:02.604043+0000","last_unstale":"2024-09-11T03:07:32.930411+0000","last_undegraded":"2024-09-11T03:07:32.930411+0000","last_fullsized":"2024-09-11T03:07:32.930411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:55.319066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932296+0000","last_change":"2024-09-11T02:51:02.574684+0000","last_active":"2024-09-11T03:07:33.932296+0000","last_peered":"2024-09-11T03:07:33.932296+0000","last_clean":"2024-09-11T03:07:33.932296+0000","last_became_active":"2024-09-11T02:51:02.574238+0000","last_became_peered":"2024-09-11T02:51:02.574238+0000","last_unstale":"2024-09-11T03:07:33.932296+0000","last_undegraded":"2024-09-11T03:07:33.932296+0000","last_fullsized":"2024-09-11T03:07:33.932296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:10:59.496994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938047+0000","last_change":"2024-09-11T02:51:02.752301+0000","last_active":"2024-09-11T03:07:33.938047+0000","last_peered":"2024-09-11T03:07:33.938047+0000","last_clean":"2024-09-11T03:07:33.938047+0000","last_became_active":"2024-09-11T02:51:02.745188+0000","last_became_peered":"2024-09-11T02:51:02.745188+0000","last_unstale":"2024-09-11T03:07:33.938047+0000","last_undegraded":"2024-09-11T03:07:33.938047+0000","last_fullsized":"2024-09-11T03:07:33.938047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:05:51.425341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924911+0000","last_change":"2024-09-11T02:51:02.583697+0000","last_active":"2024-09-11T03:07:32.924911+0000","last_peered":"2024-09-11T03:07:32.924911+0000","last_clean":"2024-09-11T03:07:32.924911+0000","last_became_active":"2024-09-11T02:51:02.578172+0000","last_became_peered":"2024-09-11T02:51:02.578172+0000","last_unstale":"2024-09-11T03:07:32.924911+0000","last_undegraded":"2024-09-11T03:07:32.924911+0000","last_fullsized":"2024-09-11T03:07:32.924911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:55:49.570822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939024+0000","last_change":"2024-09-11T02:51:02.671125+0000","last_active":"2024-09-11T03:07:33.939024+0000","last_peered":"2024-09-11T03:07:33.939024+0000","last_clean":"2024-09-11T03:07:33.939024+0000","last_became_active":"2024-09-11T02:51:02.666973+0000","last_became_peered":"2024-09-11T02:51:02.666973+0000","last_unstale":"2024-09-11T03:07:33.939024+0000","last_undegraded":"2024-09-11T03:07:33.939024+0000","last_fullsized":"2024-09-11T03:07:33.939024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:57.810050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925926+0000","last_change":"2024-09-11T02:51:02.746877+0000","last_active":"2024-09-11T03:07:32.925926+0000","last_peered":"2024-09-11T03:07:32.925926+0000","last_clean":"2024-09-11T03:07:32.925926+0000","last_became_active":"2024-09-11T02:51:02.737993+0000","last_became_peered":"2024-09-11T02:51:02.737993+0000","last_unstale":"2024-09-11T03:07:32.925926+0000","last_undegraded":"2024-09-11T03:07:32.925926+0000","last_fullsized":"2024-09-11T03:07:32.925926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:49.770401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932737+0000","last_change":"2024-09-11T02:51:02.692125+0000","last_active":"2024-09-11T03:07:33.932737+0000","last_peered":"2024-09-11T03:07:33.932737+0000","last_clean":"2024-09-11T03:07:33.932737+0000","last_became_active":"2024-09-11T02:51:02.671301+0000","last_became_peered":"2024-09-11T02:51:02.671301+0000","last_unstale":"2024-09-11T03:07:33.932737+0000","last_undegraded":"2024-09-11T03:07:33.932737+0000","last_fullsized":"2024-09-11T03:07:33.932737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:39:47.378191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930877+0000","last_change":"2024-09-11T02:51:02.713974+0000","last_active":"2024-09-11T03:07:32.930877+0000","last_peered":"2024-09-11T03:07:32.930877+0000","last_clean":"2024-09-11T03:07:32.930877+0000","last_became_active":"2024-09-11T02:51:02.712319+0000","last_became_peered":"2024-09-11T02:51:02.712319+0000","last_unstale":"2024-09-11T03:07:32.930877+0000","last_undegraded":"2024-09-11T03:07:32.930877+0000","last_fullsized":"2024-09-11T03:07:32.930877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:12.353704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1353,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930489+0000","last_change":"2024-09-11T02:51:02.678057+0000","last_active":"2024-09-11T03:07:32.930489+0000","last_peered":"2024-09-11T03:07:32.930489+0000","last_clean":"2024-09-11T03:07:32.930489+0000","last_became_active":"2024-09-11T02:51:02.672559+0000","last_became_peered":"2024-09-11T02:51:02.672559+0000","last_unstale":"2024-09-11T03:07:32.930489+0000","last_undegraded":"2024-09-11T03:07:32.930489+0000","last_fullsized":"2024-09-11T03:07:32.930489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:41.716691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1468,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937864+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T03:07:33.937864+0000","last_peered":"2024-09-11T03:07:33.937864+0000","last_clean":"2024-09-11T03:07:33.937864+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T03:07:33.937864+0000","last_undegraded":"2024-09-11T03:07:33.937864+0000","last_fullsized":"2024-09-11T03:07:33.937864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1371,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930535+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T03:07:32.930535+0000","last_peered":"2024-09-11T03:07:32.930535+0000","last_clean":"2024-09-11T03:07:32.930535+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T03:07:32.930535+0000","last_undegraded":"2024-09-11T03:07:32.930535+0000","last_fullsized":"2024-09-11T03:07:32.930535+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932586+0000","last_change":"2024-09-11T02:51:02.712255+0000","last_active":"2024-09-11T03:07:33.932586+0000","last_peered":"2024-09-11T03:07:33.932586+0000","last_clean":"2024-09-11T03:07:33.932586+0000","last_became_active":"2024-09-11T02:51:02.709268+0000","last_became_peered":"2024-09-11T02:51:02.709268+0000","last_unstale":"2024-09-11T03:07:33.932586+0000","last_undegraded":"2024-09-11T03:07:33.932586+0000","last_fullsized":"2024-09-11T03:07:33.932586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:41.994663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937955+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T03:07:33.937955+0000","last_peered":"2024-09-11T03:07:33.937955+0000","last_clean":"2024-09-11T03:07:33.937955+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T03:07:33.937955+0000","last_undegraded":"2024-09-11T03:07:33.937955+0000","last_fullsized":"2024-09-11T03:07:33.937955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1355,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.931087+0000","last_change":"2024-09-11T02:51:02.592214+0000","last_active":"2024-09-11T03:07:32.931087+0000","last_peered":"2024-09-11T03:07:32.931087+0000","last_clean":"2024-09-11T03:07:32.931087+0000","last_became_active":"2024-09-11T02:51:02.587630+0000","last_became_peered":"2024-09-11T02:51:02.587630+0000","last_unstale":"2024-09-11T03:07:32.931087+0000","last_undegraded":"2024-09-11T03:07:32.931087+0000","last_fullsized":"2024-09-11T03:07:32.931087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:26:26.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924408+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T03:07:32.924408+0000","last_peered":"2024-09-11T03:07:32.924408+0000","last_clean":"2024-09-11T03:07:32.924408+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T03:07:32.924408+0000","last_undegraded":"2024-09-11T03:07:32.924408+0000","last_fullsized":"2024-09-11T03:07:32.924408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932714+0000","last_change":"2024-09-11T02:51:02.686284+0000","last_active":"2024-09-11T03:07:33.932714+0000","last_peered":"2024-09-11T03:07:33.932714+0000","last_clean":"2024-09-11T03:07:33.932714+0000","last_became_active":"2024-09-11T02:51:02.679438+0000","last_became_peered":"2024-09-11T02:51:02.679438+0000","last_unstale":"2024-09-11T03:07:33.932714+0000","last_undegraded":"2024-09-11T03:07:33.932714+0000","last_fullsized":"2024-09-11T03:07:33.932714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:30:09.310176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930275+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T03:07:32.930275+0000","last_peered":"2024-09-11T03:07:32.930275+0000","last_clean":"2024-09-11T03:07:32.930275+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T03:07:32.930275+0000","last_undegraded":"2024-09-11T03:07:32.930275+0000","last_fullsized":"2024-09-11T03:07:32.930275+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1363,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924938+0000","last_change":"2024-09-11T02:51:02.707327+0000","last_active":"2024-09-11T03:07:32.924938+0000","last_peered":"2024-09-11T03:07:32.924938+0000","last_clean":"2024-09-11T03:07:32.924938+0000","last_became_active":"2024-09-11T02:51:02.655843+0000","last_became_peered":"2024-09-11T02:51:02.655843+0000","last_unstale":"2024-09-11T03:07:32.924938+0000","last_undegraded":"2024-09-11T03:07:32.924938+0000","last_fullsized":"2024-09-11T03:07:32.924938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:24.067746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932558+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T03:07:33.932558+0000","last_peered":"2024-09-11T03:07:33.932558+0000","last_clean":"2024-09-11T03:07:33.932558+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T03:07:33.932558+0000","last_undegraded":"2024-09-11T03:07:33.932558+0000","last_fullsized":"2024-09-11T03:07:33.932558+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1365,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932452+0000","last_change":"2024-09-11T02:51:02.618073+0000","last_active":"2024-09-11T03:07:33.932452+0000","last_peered":"2024-09-11T03:07:33.932452+0000","last_clean":"2024-09-11T03:07:33.932452+0000","last_became_active":"2024-09-11T02:51:02.586208+0000","last_became_peered":"2024-09-11T02:51:02.586208+0000","last_unstale":"2024-09-11T03:07:33.932452+0000","last_undegraded":"2024-09-11T03:07:33.932452+0000","last_fullsized":"2024-09-11T03:07:33.932452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:45:11.456188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938063+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T03:07:33.938063+0000","last_peered":"2024-09-11T03:07:33.938063+0000","last_clean":"2024-09-11T03:07:33.938063+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T03:07:33.938063+0000","last_undegraded":"2024-09-11T03:07:33.938063+0000","last_fullsized":"2024-09-11T03:07:33.938063+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938507+0000","last_change":"2024-09-11T02:51:02.645107+0000","last_active":"2024-09-11T03:07:33.938507+0000","last_peered":"2024-09-11T03:07:33.938507+0000","last_clean":"2024-09-11T03:07:33.938507+0000","last_became_active":"2024-09-11T02:51:02.642031+0000","last_became_peered":"2024-09-11T02:51:02.642031+0000","last_unstale":"2024-09-11T03:07:33.938507+0000","last_undegraded":"2024-09-11T03:07:33.938507+0000","last_fullsized":"2024-09-11T03:07:33.938507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:05.310788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932866+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T03:07:33.932866+0000","last_peered":"2024-09-11T03:07:33.932866+0000","last_clean":"2024-09-11T03:07:33.932866+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T03:07:33.932866+0000","last_undegraded":"2024-09-11T03:07:33.932866+0000","last_fullsized":"2024-09-11T03:07:33.932866+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938183+0000","last_change":"2024-09-11T02:51:02.753940+0000","last_active":"2024-09-11T03:07:33.938183+0000","last_peered":"2024-09-11T03:07:33.938183+0000","last_clean":"2024-09-11T03:07:33.938183+0000","last_became_active":"2024-09-11T02:51:02.748048+0000","last_became_peered":"2024-09-11T02:51:02.748048+0000","last_unstale":"2024-09-11T03:07:33.938183+0000","last_undegraded":"2024-09-11T03:07:33.938183+0000","last_fullsized":"2024-09-11T03:07:33.938183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:44.448053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939090+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T03:07:33.939090+0000","last_peered":"2024-09-11T03:07:33.939090+0000","last_clean":"2024-09-11T03:07:33.939090+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T03:07:33.939090+0000","last_undegraded":"2024-09-11T03:07:33.939090+0000","last_fullsized":"2024-09-11T03:07:33.939090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924172+0000","last_change":"2024-09-11T02:51:02.654035+0000","last_active":"2024-09-11T03:07:32.924172+0000","last_peered":"2024-09-11T03:07:32.924172+0000","last_clean":"2024-09-11T03:07:32.924172+0000","last_became_active":"2024-09-11T02:51:02.653755+0000","last_became_peered":"2024-09-11T02:51:02.653755+0000","last_unstale":"2024-09-11T03:07:32.924172+0000","last_undegraded":"2024-09-11T03:07:32.924172+0000","last_fullsized":"2024-09-11T03:07:32.924172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:31.581016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925152+0000","last_change":"2024-09-11T02:51:02.615081+0000","last_active":"2024-09-11T03:07:32.925152+0000","last_peered":"2024-09-11T03:07:32.925152+0000","last_clean":"2024-09-11T03:07:32.925152+0000","last_became_active":"2024-09-11T02:51:02.594230+0000","last_became_peered":"2024-09-11T02:51:02.594230+0000","last_unstale":"2024-09-11T03:07:32.925152+0000","last_undegraded":"2024-09-11T03:07:32.925152+0000","last_fullsized":"2024-09-11T03:07:32.925152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:18.916193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925259+0000","last_change":"2024-09-11T02:51:02.736915+0000","last_active":"2024-09-11T03:07:32.925259+0000","last_peered":"2024-09-11T03:07:32.925259+0000","last_clean":"2024-09-11T03:07:32.925259+0000","last_became_active":"2024-09-11T02:51:02.736590+0000","last_became_peered":"2024-09-11T02:51:02.736590+0000","last_unstale":"2024-09-11T03:07:32.925259+0000","last_undegraded":"2024-09-11T03:07:32.925259+0000","last_fullsized":"2024-09-11T03:07:32.925259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:38.031885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930585+0000","last_change":"2024-09-11T02:51:02.678968+0000","last_active":"2024-09-11T03:07:32.930585+0000","last_peered":"2024-09-11T03:07:32.930585+0000","last_clean":"2024-09-11T03:07:32.930585+0000","last_became_active":"2024-09-11T02:51:02.678439+0000","last_became_peered":"2024-09-11T02:51:02.678439+0000","last_unstale":"2024-09-11T03:07:32.930585+0000","last_undegraded":"2024-09-11T03:07:32.930585+0000","last_fullsized":"2024-09-11T03:07:32.930585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:41:16.872342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1350,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932552+0000","last_change":"2024-09-11T02:51:02.705476+0000","last_active":"2024-09-11T03:07:33.932552+0000","last_peered":"2024-09-11T03:07:33.932552+0000","last_clean":"2024-09-11T03:07:33.932552+0000","last_became_active":"2024-09-11T02:51:02.700809+0000","last_became_peered":"2024-09-11T02:51:02.700809+0000","last_unstale":"2024-09-11T03:07:33.932552+0000","last_undegraded":"2024-09-11T03:07:33.932552+0000","last_fullsized":"2024-09-11T03:07:33.932552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:56.654998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938812+0000","last_change":"2024-09-11T02:51:02.698839+0000","last_active":"2024-09-11T03:07:33.938812+0000","last_peered":"2024-09-11T03:07:33.938812+0000","last_clean":"2024-09-11T03:07:33.938812+0000","last_became_active":"2024-09-11T02:51:02.695758+0000","last_became_peered":"2024-09-11T02:51:02.695758+0000","last_unstale":"2024-09-11T03:07:33.938812+0000","last_undegraded":"2024-09-11T03:07:33.938812+0000","last_fullsized":"2024-09-11T03:07:33.938812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:00.606584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1357,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925801+0000","last_change":"2024-09-11T02:51:02.715299+0000","last_active":"2024-09-11T03:07:32.925801+0000","last_peered":"2024-09-11T03:07:32.925801+0000","last_clean":"2024-09-11T03:07:32.925801+0000","last_became_active":"2024-09-11T02:51:02.715041+0000","last_became_peered":"2024-09-11T02:51:02.715041+0000","last_unstale":"2024-09-11T03:07:32.925801+0000","last_undegraded":"2024-09-11T03:07:32.925801+0000","last_fullsized":"2024-09-11T03:07:32.925801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.561287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939022+0000","last_change":"2024-09-11T02:51:02.582470+0000","last_active":"2024-09-11T03:07:33.939022+0000","last_peered":"2024-09-11T03:07:33.939022+0000","last_clean":"2024-09-11T03:07:33.939022+0000","last_became_active":"2024-09-11T02:51:02.581916+0000","last_became_peered":"2024-09-11T02:51:02.581916+0000","last_unstale":"2024-09-11T03:07:33.939022+0000","last_undegraded":"2024-09-11T03:07:33.939022+0000","last_fullsized":"2024-09-11T03:07:33.939022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:11.711933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930286+0000","last_change":"2024-09-11T02:51:02.708269+0000","last_active":"2024-09-11T03:07:32.930286+0000","last_peered":"2024-09-11T03:07:32.930286+0000","last_clean":"2024-09-11T03:07:32.930286+0000","last_became_active":"2024-09-11T02:51:02.708034+0000","last_became_peered":"2024-09-11T02:51:02.708034+0000","last_unstale":"2024-09-11T03:07:32.930286+0000","last_undegraded":"2024-09-11T03:07:32.930286+0000","last_fullsized":"2024-09-11T03:07:32.930286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:19.177182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924808+0000","last_change":"2024-09-11T02:51:02.711915+0000","last_active":"2024-09-11T03:07:32.924808+0000","last_peered":"2024-09-11T03:07:32.924808+0000","last_clean":"2024-09-11T03:07:32.924808+0000","last_became_active":"2024-09-11T02:51:02.684472+0000","last_became_peered":"2024-09-11T02:51:02.684472+0000","last_unstale":"2024-09-11T03:07:32.924808+0000","last_undegraded":"2024-09-11T03:07:32.924808+0000","last_fullsized":"2024-09-11T03:07:32.924808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:56.385712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924685+0000","last_change":"2024-09-11T02:51:02.637360+0000","last_active":"2024-09-11T03:07:32.924685+0000","last_peered":"2024-09-11T03:07:32.924685+0000","last_clean":"2024-09-11T03:07:32.924685+0000","last_became_active":"2024-09-11T02:51:02.603633+0000","last_became_peered":"2024-09-11T02:51:02.603633+0000","last_unstale":"2024-09-11T03:07:32.924685+0000","last_undegraded":"2024-09-11T03:07:32.924685+0000","last_fullsized":"2024-09-11T03:07:32.924685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:25:57.435918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1358,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937869+0000","last_change":"2024-09-11T02:51:02.659574+0000","last_active":"2024-09-11T03:07:33.937869+0000","last_peered":"2024-09-11T03:07:33.937869+0000","last_clean":"2024-09-11T03:07:33.937869+0000","last_became_active":"2024-09-11T02:51:02.659063+0000","last_became_peered":"2024-09-11T02:51:02.659063+0000","last_unstale":"2024-09-11T03:07:33.937869+0000","last_undegraded":"2024-09-11T03:07:33.937869+0000","last_fullsized":"2024-09-11T03:07:33.937869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:28:59.462620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607774,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77408,"kb_used_data":10128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294432,"statfs":{"total":96636764160,"available":96557498368,"internally_reserved":0,"allocated":10371072,"data_stored":5742132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607772,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77404,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294436,"statfs":{"total":96636764160,"available":96557502464,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607775,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77988,"kb_used_data":10708,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293852,"statfs":{"total":96636764160,"available":96556904448,"internally_reserved":0,"allocated":10964992,"data_stored":6332486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607775,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85724,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286116,"statfs":{"total":96636764160,"available":96548982784,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77412,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294428,"statfs":{"total":96636764160,"available":96557494272,"internally_reserved":0,"allocated":10375168,"data_stored":5742137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55776,"kb_used_data":10704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":10960896,"data_stored":6332481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:08:05.174 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T03:08:05.175 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient: get_monmap_and_config 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient: build_initial_monmap 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient: monmap: 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:05.295725+0000 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:05.295725+0000 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:05.293 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:05.294 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 1 Processor -- start 2024-09-11T03:08:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 1 -- start start 2024-09-11T03:08:05.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient: init 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient: _add_conns ranks=[9,1,0,6,8,7,3,5,10,4,2] 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6c8806fe30 0x7f6c8809f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient(hunting): picked mon.noname-a con 0x7f6c8806fe30 addr v2:172.21.15.98:3300/0 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6c880a2ce0 addr v1:172.21.15.190:6789/0 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c8813c970 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.291+0000 7f6c8c932640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c88137ce0 con 0x7f6c8813c970 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c88073be0 con 0x7f6c880a2ce0 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f6c88070210 con 0x7f6c8806fe30 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 10 monclient(hunting): authenticate will time out at 2039.176039s 2024-09-11T03:08:05.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6c8806fe30 0x7f6c8809f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c86d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f6c8813c970 0x7f6c8813ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39702/0 (socket says 172.21.15.98:39702) 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c86d9d640 1 -- 172.21.15.98:0/2315281409 learned_addr learned my addr 172.21.15.98:0/2315281409 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f6c8806fe30 0x7f6c8809f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54050/0 (socket says 172.21.15.98:54050) 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): get_auth_request con 0x7f6c8806fe30 auth_method 0 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient(hunting): handle_auth_done global_id 104211 payload 274 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 1 -- 172.21.15.98:0/2315281409 >> v1:172.21.15.190:6789/0 conn(0x7f6c880a2ce0 legacy=0x7f6c8809f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:05.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 1 -- 172.21.15.98:0/2315281409 >> v1:172.21.15.98:6789/0 conn(0x7f6c8813c970 legacy=0x7f6c8813ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient: _finish_hunting 0 2024-09-11T03:08:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 1 monclient: found mon.noname-a 2024-09-11T03:08:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 1 -- 172.21.15.98:0/2315281409 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c880a3160 con 0x7f6c8806fe30 2024-09-11T03:08:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 1 -- 172.21.15.98:0/2315281409 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1352683053 0 0) 0x7f6c800014a0 con 0x7f6c8813c970 2024-09-11T03:08:05.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:05.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8659c640 1 --2- 172.21.15.98:0/2315281409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8806fe30 0x7f6c8809f4a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6c780097c0 tx=0x7f6c7801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614d11c1105e90d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:05.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 1 -- 172.21.15.98:0/2315281409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6c780b0070 con 0x7f6c8806fe30 2024-09-11T03:08:05.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: dump: 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.305 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: _finish_auth 0 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: _check_auth_tickets 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 1 -- 172.21.15.98:0/2315281409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c780aa810 con 0x7f6c8806fe30 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 5 monclient: authenticate success, global_id 104211 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:05.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 1 -- 172.21.15.98:0/2315281409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6c780a9300 con 0x7f6c8806fe30 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 4 set_mon_vals no callback set 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8559a640 10 monclient: dump: 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.307 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 10 monclient: shutdown 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 1 -- 172.21.15.98:0/2315281409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8806fe30 msgr2=0x7f6c8809f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.295+0000 7f6c8c932640 1 --2- 172.21.15.98:0/2315281409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8806fe30 0x7f6c8809f4a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6c780097c0 tx=0x7f6c7801c360 comp rx=0 tx=0).stop 2024-09-11T03:08:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- 172.21.15.98:0/2315281409 shutdown_connections 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 --2- 172.21.15.98:0/2315281409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8806fe30 0x7f6c8809f4a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- 172.21.15.98:0/2315281409 >> 172.21.15.98:0/2315281409 conn(0x7f6c8806eed0 msgr2=0x7f6c8806f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- 172.21.15.98:0/2315281409 shutdown_connections 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- 172.21.15.98:0/2315281409 wait complete. 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: build_initial_monmap 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: monmap: 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:05.309 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 librados: starting msgr at 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 librados: starting objecter 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 Processor -- start 2024-09-11T03:08:05.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- start start 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 librados: setting wanted keys 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 librados: calling monclient init 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: init 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _add_conns ranks=[8,0,7,2,6,3,1,4,5] 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6c8806fe30 0x7f6c88136d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): picked mon.noname-e con 0x7f6c8806fe30 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8813c970 0x7f6c880a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): picked mon.noname-a con 0x7f6c8813c970 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6c880a0850 0x7f6c880a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): picked mon.noname-i con 0x7f6c880a0850 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6c88142eb0 con 0x7f6c8813c970 2024-09-11T03:08:05.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f6c880517e0 con 0x7f6c880a0850 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f6c88143030 con 0x7f6c8806fe30 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient(hunting): authenticate will time out at 2039.181660s 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6c8806fe30 0x7f6c88136d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8813c970 0x7f6c880a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8813c970 0x7f6c880a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54056/0 (socket says 172.21.15.98:54056) 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6c8806fe30 0x7f6c88136d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49530/0 (socket says 172.21.15.98:49530) 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 -- 172.21.15.98:0/3234932772 learned_addr learned my addr 172.21.15.98:0/3234932772 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): get_auth_request con 0x7f6c8813c970 auth_method 0 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 10 monclient(hunting): get_auth_request con 0x7f6c8806fe30 auth_method 0 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:05.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c86d9d640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6c880a0850 0x7f6c880a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient(hunting): handle_auth_done global_id 104229 payload 824 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 -- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6c8806fe30 msgr2=0x7f6c88136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6c8806fe30 0x7f6c88136d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 -- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6c880a0850 msgr2=0x7f6c880a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6c880a0850 0x7f6c880a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 monclient: found mon.noname-a 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:05.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 -- 172.21.15.98:0/3234932772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8813bfc0 con 0x7f6c8813c970 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8659c640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6c8806fe30 0x7f6c88136d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c85d9b640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8813c970 0x7f6c880a0310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c4b0 tx=0x7f6c7c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 client.?.objecter ms_handle_connect 0x7f6c8813c970 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 1 -- 172.21.15.98:0/3234932772 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6c7c012aa0 con 0x7f6c8813c970 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: dump: 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: _finish_auth 0 2024-09-11T03:08:05.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 1 -- 172.21.15.98:0/3234932772 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c7c017070 con 0x7f6c8813c970 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 1 -- 172.21.15.98:0/3234932772 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6c7c012c80 con 0x7f6c8813c970 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 5 monclient: authenticate success, global_id 104229 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _renew_subs 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c8813b110 con 0x7f6c8813c970 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c84d99640 4 set_mon_vals no callback set 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 monclient: dump: 2024-09-11T03:08:05.317 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.318 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 client.104229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _renew_subs 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c880a1d70 con 0x7f6c8813c970 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 librados: init done 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 1 -- 172.21.15.98:0/3234932772 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6c7c012630 con 0x7f6c8813c970 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c8809f4a0 con 0x7f6c8813c970 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6c6803dea0 0x7f6c68040360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 1 -- 172.21.15.98:0/3234932772 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f6c7c04f170 con 0x7f6c8813c970 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 10 client.104229.objecter ms_dispatch 0x7f6c88000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 3 client.104229.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 3 client.104229.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.299+0000 7f6c76ffd640 20 client.104229.objecter dump_active .. 0 homeless 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.303+0000 7f6c8659c640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6c6803dea0 0x7f6c68040360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.303+0000 7f6c8659c640 10 monclient: get_auth_request con 0x7f6c6803dea0 auth_method 0 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.303+0000 7f6c76ffd640 10 client.104229.objecter ms_handle_connect 0x7f6c6803dea0 2024-09-11T03:08:05.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.303+0000 7f6c8659c640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6c6803dea0 0x7f6c68040360 secure :-1 s=READY pgs=3712 cs=0 l=1 rev1=1 crypto rx=0x7f6c780ad000 tx=0x7f6c780a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:05.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.307+0000 7f6c76ffd640 1 -- 172.21.15.98:0/3234932772 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f6c7c012e20 con 0x7f6c8813c970 2024-09-11T03:08:05.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.307+0000 7f6c76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:05.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.307+0000 7f6c76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:05.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.459+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6c880a4f10 con 0x7f6c6803dea0 2024-09-11T03:08:05.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c76ffd640 1 -- 172.21.15.98:0/3234932772 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114038 (secure 0 0 0) 0x7f6c780d6d90 con 0x7f6c6803dea0 2024-09-11T03:08:05.468 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T03:08:05.468 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T03:08:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 10 librados: watch_flush enter 2024-09-11T03:08:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 10 librados: watch_flush exit 2024-09-11T03:08:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 20 client.104229.objecter shutdown clearing up homeless session... 2024-09-11T03:08:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 10 client.104229.objecter successfully canceled tick 2024-09-11T03:08:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6c6803dea0 msgr2=0x7f6c68040360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6c6803dea0 0x7f6c68040360 secure :-1 s=READY pgs=3712 cs=0 l=1 rev1=1 crypto rx=0x7f6c780ad000 tx=0x7f6c780a6000 comp rx=0 tx=0).stop 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 10 monclient: shutdown 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8813c970 msgr2=0x7f6c880a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8813c970 0x7f6c880a0310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00c4b0 tx=0x7f6c7c00c970 comp rx=0 tx=0).stop 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 shutdown_connections 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f6c6803dea0 0x7f6c68040360 unknown :-1 s=CLOSED pgs=3712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f6c880a0850 0x7f6c880a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c8813c970 0x7f6c880a0310 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 --2- 172.21.15.98:0/3234932772 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f6c8806fe30 0x7f6c88136d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 >> 172.21.15.98:0/3234932772 conn(0x7f6c8806eed0 msgr2=0x7f6c880a3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 shutdown_connections 2024-09-11T03:08:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 -- 172.21.15.98:0/3234932772 wait complete. 2024-09-11T03:08:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.467+0000 7f6c8c932640 1 librados: shutdown 2024-09-11T03:08:05.485 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1239,"stamp":"2024-09-11T03:08:04.134359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":451712,"kb_used_data":61920,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565779328,"statfs":{"total":579820584960,"available":579358031872,"internally_reserved":0,"allocated":63406080,"data_stored":35633462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":399039138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003885"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925479+0000","last_change":"2024-09-11T02:51:02.707390+0000","last_active":"2024-09-11T03:07:32.925479+0000","last_peered":"2024-09-11T03:07:32.925479+0000","last_clean":"2024-09-11T03:07:32.925479+0000","last_became_active":"2024-09-11T02:51:02.704466+0000","last_became_peered":"2024-09-11T02:51:02.704466+0000","last_unstale":"2024-09-11T03:07:32.925479+0000","last_undegraded":"2024-09-11T03:07:32.925479+0000","last_fullsized":"2024-09-11T03:07:32.925479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:51:38.363080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939381+0000","last_change":"2024-09-11T02:51:02.751947+0000","last_active":"2024-09-11T03:07:33.939381+0000","last_peered":"2024-09-11T03:07:33.939381+0000","last_clean":"2024-09-11T03:07:33.939381+0000","last_became_active":"2024-09-11T02:51:02.747449+0000","last_became_peered":"2024-09-11T02:51:02.747449+0000","last_unstale":"2024-09-11T03:07:33.939381+0000","last_undegraded":"2024-09-11T03:07:33.939381+0000","last_fullsized":"2024-09-11T03:07:33.939381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:34.946819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.923807+0000","last_change":"2024-09-11T02:51:02.757155+0000","last_active":"2024-09-11T03:07:32.923807+0000","last_peered":"2024-09-11T03:07:32.923807+0000","last_clean":"2024-09-11T03:07:32.923807+0000","last_became_active":"2024-09-11T02:51:02.746704+0000","last_became_peered":"2024-09-11T02:51:02.746704+0000","last_unstale":"2024-09-11T03:07:32.923807+0000","last_undegraded":"2024-09-11T03:07:32.923807+0000","last_fullsized":"2024-09-11T03:07:32.923807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:08:27.349903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938159+0000","last_change":"2024-09-11T02:51:02.703570+0000","last_active":"2024-09-11T03:07:33.938159+0000","last_peered":"2024-09-11T03:07:33.938159+0000","last_clean":"2024-09-11T03:07:33.938159+0000","last_became_active":"2024-09-11T02:51:02.695884+0000","last_became_peered":"2024-09-11T02:51:02.695884+0000","last_unstale":"2024-09-11T03:07:33.938159+0000","last_undegraded":"2024-09-11T03:07:33.938159+0000","last_fullsized":"2024-09-11T03:07:33.938159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:24.055472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930411+0000","last_change":"2024-09-11T02:51:02.604922+0000","last_active":"2024-09-11T03:07:32.930411+0000","last_peered":"2024-09-11T03:07:32.930411+0000","last_clean":"2024-09-11T03:07:32.930411+0000","last_became_active":"2024-09-11T02:51:02.604043+0000","last_became_peered":"2024-09-11T02:51:02.604043+0000","last_unstale":"2024-09-11T03:07:32.930411+0000","last_undegraded":"2024-09-11T03:07:32.930411+0000","last_fullsized":"2024-09-11T03:07:32.930411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:55.319066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932296+0000","last_change":"2024-09-11T02:51:02.574684+0000","last_active":"2024-09-11T03:07:33.932296+0000","last_peered":"2024-09-11T03:07:33.932296+0000","last_clean":"2024-09-11T03:07:33.932296+0000","last_became_active":"2024-09-11T02:51:02.574238+0000","last_became_peered":"2024-09-11T02:51:02.574238+0000","last_unstale":"2024-09-11T03:07:33.932296+0000","last_undegraded":"2024-09-11T03:07:33.932296+0000","last_fullsized":"2024-09-11T03:07:33.932296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:10:59.496994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938047+0000","last_change":"2024-09-11T02:51:02.752301+0000","last_active":"2024-09-11T03:07:33.938047+0000","last_peered":"2024-09-11T03:07:33.938047+0000","last_clean":"2024-09-11T03:07:33.938047+0000","last_became_active":"2024-09-11T02:51:02.745188+0000","last_became_peered":"2024-09-11T02:51:02.745188+0000","last_unstale":"2024-09-11T03:07:33.938047+0000","last_undegraded":"2024-09-11T03:07:33.938047+0000","last_fullsized":"2024-09-11T03:07:33.938047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:05:51.425341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924911+0000","last_change":"2024-09-11T02:51:02.583697+0000","last_active":"2024-09-11T03:07:32.924911+0000","last_peered":"2024-09-11T03:07:32.924911+0000","last_clean":"2024-09-11T03:07:32.924911+0000","last_became_active":"2024-09-11T02:51:02.578172+0000","last_became_peered":"2024-09-11T02:51:02.578172+0000","last_unstale":"2024-09-11T03:07:32.924911+0000","last_undegraded":"2024-09-11T03:07:32.924911+0000","last_fullsized":"2024-09-11T03:07:32.924911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:55:49.570822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939024+0000","last_change":"2024-09-11T02:51:02.671125+0000","last_active":"2024-09-11T03:07:33.939024+0000","last_peered":"2024-09-11T03:07:33.939024+0000","last_clean":"2024-09-11T03:07:33.939024+0000","last_became_active":"2024-09-11T02:51:02.666973+0000","last_became_peered":"2024-09-11T02:51:02.666973+0000","last_unstale":"2024-09-11T03:07:33.939024+0000","last_undegraded":"2024-09-11T03:07:33.939024+0000","last_fullsized":"2024-09-11T03:07:33.939024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:57.810050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925926+0000","last_change":"2024-09-11T02:51:02.746877+0000","last_active":"2024-09-11T03:07:32.925926+0000","last_peered":"2024-09-11T03:07:32.925926+0000","last_clean":"2024-09-11T03:07:32.925926+0000","last_became_active":"2024-09-11T02:51:02.737993+0000","last_became_peered":"2024-09-11T02:51:02.737993+0000","last_unstale":"2024-09-11T03:07:32.925926+0000","last_undegraded":"2024-09-11T03:07:32.925926+0000","last_fullsized":"2024-09-11T03:07:32.925926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:49.770401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932737+0000","last_change":"2024-09-11T02:51:02.692125+0000","last_active":"2024-09-11T03:07:33.932737+0000","last_peered":"2024-09-11T03:07:33.932737+0000","last_clean":"2024-09-11T03:07:33.932737+0000","last_became_active":"2024-09-11T02:51:02.671301+0000","last_became_peered":"2024-09-11T02:51:02.671301+0000","last_unstale":"2024-09-11T03:07:33.932737+0000","last_undegraded":"2024-09-11T03:07:33.932737+0000","last_fullsized":"2024-09-11T03:07:33.932737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:39:47.378191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930877+0000","last_change":"2024-09-11T02:51:02.713974+0000","last_active":"2024-09-11T03:07:32.930877+0000","last_peered":"2024-09-11T03:07:32.930877+0000","last_clean":"2024-09-11T03:07:32.930877+0000","last_became_active":"2024-09-11T02:51:02.712319+0000","last_became_peered":"2024-09-11T02:51:02.712319+0000","last_unstale":"2024-09-11T03:07:32.930877+0000","last_undegraded":"2024-09-11T03:07:32.930877+0000","last_fullsized":"2024-09-11T03:07:32.930877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:12.353704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1353,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930489+0000","last_change":"2024-09-11T02:51:02.678057+0000","last_active":"2024-09-11T03:07:32.930489+0000","last_peered":"2024-09-11T03:07:32.930489+0000","last_clean":"2024-09-11T03:07:32.930489+0000","last_became_active":"2024-09-11T02:51:02.672559+0000","last_became_peered":"2024-09-11T02:51:02.672559+0000","last_unstale":"2024-09-11T03:07:32.930489+0000","last_undegraded":"2024-09-11T03:07:32.930489+0000","last_fullsized":"2024-09-11T03:07:32.930489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:41.716691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1468,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937864+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T03:07:33.937864+0000","last_peered":"2024-09-11T03:07:33.937864+0000","last_clean":"2024-09-11T03:07:33.937864+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T03:07:33.937864+0000","last_undegraded":"2024-09-11T03:07:33.937864+0000","last_fullsized":"2024-09-11T03:07:33.937864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1371,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930535+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T03:07:32.930535+0000","last_peered":"2024-09-11T03:07:32.930535+0000","last_clean":"2024-09-11T03:07:32.930535+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T03:07:32.930535+0000","last_undegraded":"2024-09-11T03:07:32.930535+0000","last_fullsized":"2024-09-11T03:07:32.930535+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932586+0000","last_change":"2024-09-11T02:51:02.712255+0000","last_active":"2024-09-11T03:07:33.932586+0000","last_peered":"2024-09-11T03:07:33.932586+0000","last_clean":"2024-09-11T03:07:33.932586+0000","last_became_active":"2024-09-11T02:51:02.709268+0000","last_became_peered":"2024-09-11T02:51:02.709268+0000","last_unstale":"2024-09-11T03:07:33.932586+0000","last_undegraded":"2024-09-11T03:07:33.932586+0000","last_fullsized":"2024-09-11T03:07:33.932586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:41.994663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937955+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T03:07:33.937955+0000","last_peered":"2024-09-11T03:07:33.937955+0000","last_clean":"2024-09-11T03:07:33.937955+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T03:07:33.937955+0000","last_undegraded":"2024-09-11T03:07:33.937955+0000","last_fullsized":"2024-09-11T03:07:33.937955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1355,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.931087+0000","last_change":"2024-09-11T02:51:02.592214+0000","last_active":"2024-09-11T03:07:32.931087+0000","last_peered":"2024-09-11T03:07:32.931087+0000","last_clean":"2024-09-11T03:07:32.931087+0000","last_became_active":"2024-09-11T02:51:02.587630+0000","last_became_peered":"2024-09-11T02:51:02.587630+0000","last_unstale":"2024-09-11T03:07:32.931087+0000","last_undegraded":"2024-09-11T03:07:32.931087+0000","last_fullsized":"2024-09-11T03:07:32.931087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:26:26.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924408+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T03:07:32.924408+0000","last_peered":"2024-09-11T03:07:32.924408+0000","last_clean":"2024-09-11T03:07:32.924408+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T03:07:32.924408+0000","last_undegraded":"2024-09-11T03:07:32.924408+0000","last_fullsized":"2024-09-11T03:07:32.924408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932714+0000","last_change":"2024-09-11T02:51:02.686284+0000","last_active":"2024-09-11T03:07:33.932714+0000","last_peered":"2024-09-11T03:07:33.932714+0000","last_clean":"2024-09-11T03:07:33.932714+0000","last_became_active":"2024-09-11T02:51:02.679438+0000","last_became_peered":"2024-09-11T02:51:02.679438+0000","last_unstale":"2024-09-11T03:07:33.932714+0000","last_undegraded":"2024-09-11T03:07:33.932714+0000","last_fullsized":"2024-09-11T03:07:33.932714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:30:09.310176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930275+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T03:07:32.930275+0000","last_peered":"2024-09-11T03:07:32.930275+0000","last_clean":"2024-09-11T03:07:32.930275+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T03:07:32.930275+0000","last_undegraded":"2024-09-11T03:07:32.930275+0000","last_fullsized":"2024-09-11T03:07:32.930275+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1363,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924938+0000","last_change":"2024-09-11T02:51:02.707327+0000","last_active":"2024-09-11T03:07:32.924938+0000","last_peered":"2024-09-11T03:07:32.924938+0000","last_clean":"2024-09-11T03:07:32.924938+0000","last_became_active":"2024-09-11T02:51:02.655843+0000","last_became_peered":"2024-09-11T02:51:02.655843+0000","last_unstale":"2024-09-11T03:07:32.924938+0000","last_undegraded":"2024-09-11T03:07:32.924938+0000","last_fullsized":"2024-09-11T03:07:32.924938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:24.067746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932558+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T03:07:33.932558+0000","last_peered":"2024-09-11T03:07:33.932558+0000","last_clean":"2024-09-11T03:07:33.932558+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T03:07:33.932558+0000","last_undegraded":"2024-09-11T03:07:33.932558+0000","last_fullsized":"2024-09-11T03:07:33.932558+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1365,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932452+0000","last_change":"2024-09-11T02:51:02.618073+0000","last_active":"2024-09-11T03:07:33.932452+0000","last_peered":"2024-09-11T03:07:33.932452+0000","last_clean":"2024-09-11T03:07:33.932452+0000","last_became_active":"2024-09-11T02:51:02.586208+0000","last_became_peered":"2024-09-11T02:51:02.586208+0000","last_unstale":"2024-09-11T03:07:33.932452+0000","last_undegraded":"2024-09-11T03:07:33.932452+0000","last_fullsized":"2024-09-11T03:07:33.932452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:45:11.456188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938063+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T03:07:33.938063+0000","last_peered":"2024-09-11T03:07:33.938063+0000","last_clean":"2024-09-11T03:07:33.938063+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T03:07:33.938063+0000","last_undegraded":"2024-09-11T03:07:33.938063+0000","last_fullsized":"2024-09-11T03:07:33.938063+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938507+0000","last_change":"2024-09-11T02:51:02.645107+0000","last_active":"2024-09-11T03:07:33.938507+0000","last_peered":"2024-09-11T03:07:33.938507+0000","last_clean":"2024-09-11T03:07:33.938507+0000","last_became_active":"2024-09-11T02:51:02.642031+0000","last_became_peered":"2024-09-11T02:51:02.642031+0000","last_unstale":"2024-09-11T03:07:33.938507+0000","last_undegraded":"2024-09-11T03:07:33.938507+0000","last_fullsized":"2024-09-11T03:07:33.938507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:05.310788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932866+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T03:07:33.932866+0000","last_peered":"2024-09-11T03:07:33.932866+0000","last_clean":"2024-09-11T03:07:33.932866+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T03:07:33.932866+0000","last_undegraded":"2024-09-11T03:07:33.932866+0000","last_fullsized":"2024-09-11T03:07:33.932866+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938183+0000","last_change":"2024-09-11T02:51:02.753940+0000","last_active":"2024-09-11T03:07:33.938183+0000","last_peered":"2024-09-11T03:07:33.938183+0000","last_clean":"2024-09-11T03:07:33.938183+0000","last_became_active":"2024-09-11T02:51:02.748048+0000","last_became_peered":"2024-09-11T02:51:02.748048+0000","last_unstale":"2024-09-11T03:07:33.938183+0000","last_undegraded":"2024-09-11T03:07:33.938183+0000","last_fullsized":"2024-09-11T03:07:33.938183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:44.448053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939090+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T03:07:33.939090+0000","last_peered":"2024-09-11T03:07:33.939090+0000","last_clean":"2024-09-11T03:07:33.939090+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T03:07:33.939090+0000","last_undegraded":"2024-09-11T03:07:33.939090+0000","last_fullsized":"2024-09-11T03:07:33.939090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924172+0000","last_change":"2024-09-11T02:51:02.654035+0000","last_active":"2024-09-11T03:07:32.924172+0000","last_peered":"2024-09-11T03:07:32.924172+0000","last_clean":"2024-09-11T03:07:32.924172+0000","last_became_active":"2024-09-11T02:51:02.653755+0000","last_became_peered":"2024-09-11T02:51:02.653755+0000","last_unstale":"2024-09-11T03:07:32.924172+0000","last_undegraded":"2024-09-11T03:07:32.924172+0000","last_fullsized":"2024-09-11T03:07:32.924172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:31.581016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925152+0000","last_change":"2024-09-11T02:51:02.615081+0000","last_active":"2024-09-11T03:07:32.925152+0000","last_peered":"2024-09-11T03:07:32.925152+0000","last_clean":"2024-09-11T03:07:32.925152+0000","last_became_active":"2024-09-11T02:51:02.594230+0000","last_became_peered":"2024-09-11T02:51:02.594230+0000","last_unstale":"2024-09-11T03:07:32.925152+0000","last_undegraded":"2024-09-11T03:07:32.925152+0000","last_fullsized":"2024-09-11T03:07:32.925152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:18.916193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925259+0000","last_change":"2024-09-11T02:51:02.736915+0000","last_active":"2024-09-11T03:07:32.925259+0000","last_peered":"2024-09-11T03:07:32.925259+0000","last_clean":"2024-09-11T03:07:32.925259+0000","last_became_active":"2024-09-11T02:51:02.736590+0000","last_became_peered":"2024-09-11T02:51:02.736590+0000","last_unstale":"2024-09-11T03:07:32.925259+0000","last_undegraded":"2024-09-11T03:07:32.925259+0000","last_fullsized":"2024-09-11T03:07:32.925259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:38.031885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930585+0000","last_change":"2024-09-11T02:51:02.678968+0000","last_active":"2024-09-11T03:07:32.930585+0000","last_peered":"2024-09-11T03:07:32.930585+0000","last_clean":"2024-09-11T03:07:32.930585+0000","last_became_active":"2024-09-11T02:51:02.678439+0000","last_became_peered":"2024-09-11T02:51:02.678439+0000","last_unstale":"2024-09-11T03:07:32.930585+0000","last_undegraded":"2024-09-11T03:07:32.930585+0000","last_fullsized":"2024-09-11T03:07:32.930585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:41:16.872342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1350,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932552+0000","last_change":"2024-09-11T02:51:02.705476+0000","last_active":"2024-09-11T03:07:33.932552+0000","last_peered":"2024-09-11T03:07:33.932552+0000","last_clean":"2024-09-11T03:07:33.932552+0000","last_became_active":"2024-09-11T02:51:02.700809+0000","last_became_peered":"2024-09-11T02:51:02.700809+0000","last_unstale":"2024-09-11T03:07:33.932552+0000","last_undegraded":"2024-09-11T03:07:33.932552+0000","last_fullsized":"2024-09-11T03:07:33.932552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:56.654998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938812+0000","last_change":"2024-09-11T02:51:02.698839+0000","last_active":"2024-09-11T03:07:33.938812+0000","last_peered":"2024-09-11T03:07:33.938812+0000","last_clean":"2024-09-11T03:07:33.938812+0000","last_became_active":"2024-09-11T02:51:02.695758+0000","last_became_peered":"2024-09-11T02:51:02.695758+0000","last_unstale":"2024-09-11T03:07:33.938812+0000","last_undegraded":"2024-09-11T03:07:33.938812+0000","last_fullsized":"2024-09-11T03:07:33.938812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:00.606584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1357,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925801+0000","last_change":"2024-09-11T02:51:02.715299+0000","last_active":"2024-09-11T03:07:32.925801+0000","last_peered":"2024-09-11T03:07:32.925801+0000","last_clean":"2024-09-11T03:07:32.925801+0000","last_became_active":"2024-09-11T02:51:02.715041+0000","last_became_peered":"2024-09-11T02:51:02.715041+0000","last_unstale":"2024-09-11T03:07:32.925801+0000","last_undegraded":"2024-09-11T03:07:32.925801+0000","last_fullsized":"2024-09-11T03:07:32.925801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.561287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939022+0000","last_change":"2024-09-11T02:51:02.582470+0000","last_active":"2024-09-11T03:07:33.939022+0000","last_peered":"2024-09-11T03:07:33.939022+0000","last_clean":"2024-09-11T03:07:33.939022+0000","last_became_active":"2024-09-11T02:51:02.581916+0000","last_became_peered":"2024-09-11T02:51:02.581916+0000","last_unstale":"2024-09-11T03:07:33.939022+0000","last_undegraded":"2024-09-11T03:07:33.939022+0000","last_fullsized":"2024-09-11T03:07:33.939022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:11.711933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930286+0000","last_change":"2024-09-11T02:51:02.708269+0000","last_active":"2024-09-11T03:07:32.930286+0000","last_peered":"2024-09-11T03:07:32.930286+0000","last_clean":"2024-09-11T03:07:32.930286+0000","last_became_active":"2024-09-11T02:51:02.708034+0000","last_became_peered":"2024-09-11T02:51:02.708034+0000","last_unstale":"2024-09-11T03:07:32.930286+0000","last_undegraded":"2024-09-11T03:07:32.930286+0000","last_fullsized":"2024-09-11T03:07:32.930286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:19.177182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924808+0000","last_change":"2024-09-11T02:51:02.711915+0000","last_active":"2024-09-11T03:07:32.924808+0000","last_peered":"2024-09-11T03:07:32.924808+0000","last_clean":"2024-09-11T03:07:32.924808+0000","last_became_active":"2024-09-11T02:51:02.684472+0000","last_became_peered":"2024-09-11T02:51:02.684472+0000","last_unstale":"2024-09-11T03:07:32.924808+0000","last_undegraded":"2024-09-11T03:07:32.924808+0000","last_fullsized":"2024-09-11T03:07:32.924808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:56.385712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924685+0000","last_change":"2024-09-11T02:51:02.637360+0000","last_active":"2024-09-11T03:07:32.924685+0000","last_peered":"2024-09-11T03:07:32.924685+0000","last_clean":"2024-09-11T03:07:32.924685+0000","last_became_active":"2024-09-11T02:51:02.603633+0000","last_became_peered":"2024-09-11T02:51:02.603633+0000","last_unstale":"2024-09-11T03:07:32.924685+0000","last_undegraded":"2024-09-11T03:07:32.924685+0000","last_fullsized":"2024-09-11T03:07:32.924685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:25:57.435918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1358,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937869+0000","last_change":"2024-09-11T02:51:02.659574+0000","last_active":"2024-09-11T03:07:33.937869+0000","last_peered":"2024-09-11T03:07:33.937869+0000","last_clean":"2024-09-11T03:07:33.937869+0000","last_became_active":"2024-09-11T02:51:02.659063+0000","last_became_peered":"2024-09-11T02:51:02.659063+0000","last_unstale":"2024-09-11T03:07:33.937869+0000","last_undegraded":"2024-09-11T03:07:33.937869+0000","last_fullsized":"2024-09-11T03:07:33.937869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:28:59.462620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607774,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77408,"kb_used_data":10128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294432,"statfs":{"total":96636764160,"available":96557498368,"internally_reserved":0,"allocated":10371072,"data_stored":5742132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607772,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77404,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294436,"statfs":{"total":96636764160,"available":96557502464,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607775,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77988,"kb_used_data":10708,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293852,"statfs":{"total":96636764160,"available":96556904448,"internally_reserved":0,"allocated":10964992,"data_stored":6332486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607775,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85724,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286116,"statfs":{"total":96636764160,"available":96548982784,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77412,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294428,"statfs":{"total":96636764160,"available":96557494272,"internally_reserved":0,"allocated":10375168,"data_stored":5742137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55776,"kb_used_data":10704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":10960896,"data_stored":6332481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:08:05.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T03:08:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.599+0000 7f0ffb8e0640 10 monclient: get_monmap_and_config 2024-09-11T03:08:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.599+0000 7f0ffb8e0640 10 monclient: build_initial_monmap 2024-09-11T03:08:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.599+0000 7f0ffb8e0640 10 monclient: monmap: 2024-09-11T03:08:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.601 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:05.603967+0000 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:05.603967+0000 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:05.602 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:05.603 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:05.603 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.599+0000 7f0ffb8e0640 1 Processor -- start 2024-09-11T03:08:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.599+0000 7f0ffb8e0640 1 -- start start 2024-09-11T03:08:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.599+0000 7f0ffb8e0640 10 monclient: init 2024-09-11T03:08:05.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient: _add_conns ranks=[8,6,7,9,10,2,5,1,0,3,4] 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0ff406f650 0x7f0ff4072640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): picked mon.noname-e con 0x7f0ff406f650 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ff40a8380 0x7f0ff406fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): picked mon.noname-d con 0x7f0ff40a8380 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff41397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): picked mon.noname-i con 0x7f0ff406ff70 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f0ff40a8800 con 0x7f0ff40a8380 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0ff4071a80 con 0x7f0ff406ff70 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0ff4071d60 con 0x7f0ff406f650 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient(hunting): authenticate will time out at 2039.484270s 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0ff406f650 0x7f0ff4072640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ff40a8380 0x7f0ff406fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0ff406f650 0x7f0ff4072640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:49536/0 (socket says 172.21.15.98:49536) 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ff40a8380 0x7f0ff406fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:54526/0 (socket says 172.21.15.98:54526) 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 1 -- 172.21.15.98:0/3765189360 learned_addr learned my addr 172.21.15.98:0/3765189360 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 10 monclient(hunting): get_auth_request con 0x7f0ff406f650 auth_method 0 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): get_auth_request con 0x7f0ff40a8380 auth_method 0 2024-09-11T03:08:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff9e7d640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff41397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff9e7d640 10 monclient(hunting): get_auth_request con 0x7f0ff406ff70 auth_method 0 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff9e7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff9e7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:05.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff9e7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient(hunting): handle_auth_done global_id 84759 payload 274 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 -- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0ff406f650 msgr2=0x7f0ff4072640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0ff406f650 0x7f0ff4072640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 -- 172.21.15.98:0/3765189360 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 msgr2=0x7f0ff41397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff41397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 monclient: found mon.noname-d 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 -- 172.21.15.98:0/3765189360 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff40714c0 con 0x7f0ff40a8380 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff967c640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0ff406f650 0x7f0ff4072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ff8e7b640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ff40a8380 0x7f0ff406fa30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009820 tx=0x7f0ff001c0c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=fbf567f7692788a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 1 -- 172.21.15.98:0/3765189360 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ff00b1070 con 0x7f0ff40a8380 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: dump: 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: _finish_auth 0 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 1 -- 172.21.15.98:0/3765189360 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff00a4cb0 con 0x7f0ff40a8380 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 1 -- 172.21.15.98:0/3765189360 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ff00aac20 con 0x7f0ff40a8380 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 5 monclient: authenticate success, global_id 84759 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0ffb8e0640 4 set_mon_vals no callback set 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.603+0000 7f0febfff640 10 monclient: dump: 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: shutdown 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ff40a8380 msgr2=0x7f0ff406fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ff40a8380 0x7f0ff406fa30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009820 tx=0x7f0ff001c0c0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/3765189360 shutdown_connections 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff41397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f0ff40a8380 0x7f0ff406fa30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/3765189360 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f0ff406f650 0x7f0ff4072640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/3765189360 >> 172.21.15.98:0/3765189360 conn(0x7f0ff40ac8e0 msgr2=0x7f0ff40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/3765189360 shutdown_connections 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/3765189360 wait complete. 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: build_initial_monmap 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: monmap: 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:05.621 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 librados: starting msgr at 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 librados: starting objecter 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 Processor -- start 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- start start 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 librados: setting wanted keys 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 librados: calling monclient init 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: init 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _add_conns ranks=[4,7,3,8,2,0,6,5,1] 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0ff406f650 0x7f0ff41d1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): picked mon.noname-c con 0x7f0ff406f650 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff4136800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): picked mon.noname-i con 0x7f0ff406ff70 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:05.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ff40a8380 0x7f0ff413b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): picked mon.noname-g con 0x7f0ff40a8380 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f0ff4142fc0 con 0x7f0ff40a8380 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0ff4142e40 con 0x7f0ff406f650 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f0ff4143140 con 0x7f0ff406ff70 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient(hunting): authenticate will time out at 2039.489953s 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0ff406f650 0x7f0ff41d1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0ff406f650 0x7f0ff41d1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:57812/0 (socket says 172.21.15.98:57812) 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 -- 172.21.15.98:0/2478262963 learned_addr learned my addr 172.21.15.98:0/2478262963 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:05.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): get_auth_request con 0x7f0ff406f650 auth_method 0 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff8e7b640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff4136800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient(hunting): handle_auth_done global_id 24952 payload 824 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff9e7d640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ff40a8380 0x7f0ff413b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 -- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 msgr2=0x7f0ff4136800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff4136800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 -- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ff40a8380 msgr2=0x7f0ff413b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ff40a8380 0x7f0ff413b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient: _finish_hunting 0 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 monclient: found mon.noname-c 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 -- 172.21.15.98:0/2478262963 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff413f4b0 con 0x7f0ff406f650 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff8e7b640 10 monclient: get_auth_request con 0x7f0ff406ff70 auth_method 0 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff8e7b640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff4136800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ff967c640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0ff406f650 0x7f0ff41d1160 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00efc0 tx=0x7f0fec00c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 client.?.objecter ms_handle_connect 0x7f0ff406f650 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 1 -- 172.21.15.98:0/2478262963 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0fec0090d0 con 0x7f0ff406f650 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: dump: 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.627 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: _finish_auth 0 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 1 -- 172.21.15.98:0/2478262963 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fec019070 con 0x7f0ff406f650 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 5 monclient: authenticate success, global_id 24952 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _renew_subs 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:05.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff413f790 con 0x7f0ff406f650 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 1 -- 172.21.15.98:0/2478262963 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0fec014d60 con 0x7f0ff406f650 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 10 monclient: dump: 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:05.629 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 client.24952.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0feaffd640 4 set_mon_vals no callback set 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _renew_subs 2024-09-11T03:08:05.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff413d430 con 0x7f0ff406f650 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 librados: init done 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff4139770 con 0x7f0ff406f650 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.607+0000 7f0fe9ffb640 1 -- 172.21.15.98:0/2478262963 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f0fec014480 con 0x7f0ff406f650 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0fe9ffb640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0fe003df40 0x7f0fe0040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0fe9ffb640 1 -- 172.21.15.98:0/2478262963 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f0fec04fa40 con 0x7f0ff406f650 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0fe9ffb640 10 client.24952.objecter ms_dispatch 0x7f0ff4000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:05.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0fe9ffb640 3 client.24952.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0fe9ffb640 3 client.24952.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0fe9ffb640 20 client.24952.objecter dump_active .. 0 homeless 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0ff8e7b640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0fe003df40 0x7f0fe0040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0ff8e7b640 10 monclient: get_auth_request con 0x7f0fe003df40 auth_method 0 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0fe9ffb640 10 client.24952.objecter ms_handle_connect 0x7f0fe003df40 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.611+0000 7f0ff8e7b640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0fe003df40 0x7f0fe0040400 secure :-1 s=READY pgs=3713 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00ae000 tx=0x7f0ff0004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.615+0000 7f0fe9ffb640 1 -- 172.21.15.98:0/2478262963 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f0fec022060 con 0x7f0ff406f650 2024-09-11T03:08:05.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.615+0000 7f0fe9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:05.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.615+0000 7f0fe9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.787+0000 7f0ffb8e0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-11T03:08:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.787+0000 7f0ffb8e0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T03:08:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.787+0000 7f0ffb8e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:05.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.787+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ff413efe0 con 0x7f0ff406f650 2024-09-11T03:08:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0fe9ffb640 1 -- 172.21.15.98:0/2478262963 <== mon.4 v2:172.21.15.98:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v699) ==== 74+0+15479 (secure 0 0 0) 0x7f0fec010040 con 0x7f0ff406f650 2024-09-11T03:08:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0fe9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-11T03:08:05.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0fe9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:05.791 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T03:08:05.791 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":699,"fsid":"797ac5bf-8bfb-4032-bff5-2e861cee7479","created":"2024-09-11T02:49:44.669209+0000","modified":"2024-09-11T03:07:32.921333+0000","last_up_change":"2024-09-11T02:49:54.844374+0000","last_in_change":"2024-09-11T02:49:46.462961+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":96,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T02:49:55.787515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T02:50:01.768815+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-11T02:51:01.069208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84a3e39d-63f8-4735-a51d-6248655e4c80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6801","nonce":1456395826}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6803","nonce":1456395826}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6807","nonce":1456395826}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1456395826},{"type":"v1","addr":"172.21.15.98:6805","nonce":1456395826}]},"public_addr":"172.21.15.98:6801/1456395826","cluster_addr":"172.21.15.98:6803/1456395826","heartbeat_back_addr":"172.21.15.98:6807/1456395826","heartbeat_front_addr":"172.21.15.98:6805/1456395826","state":["exists","up"]},{"osd":1,"uuid":"8753f916-e49b-493b-bfbc-790c2d4866a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6809","nonce":2313933591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6811","nonce":2313933591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6815","nonce":2313933591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":2313933591},{"type":"v1","addr":"172.21.15.98:6813","nonce":2313933591}]},"public_addr":"172.21.15.98:6809/2313933591","cluster_addr":"172.21.15.98:6811/2313933591","heartbeat_back_addr":"172.21.15.98:6815/2313933591","heartbeat_front_addr":"172.21.15.98:6813/2313933591","state":["exists","up"]},{"osd":2,"uuid":"0c1bb788-0d6d-49eb-aef4-a94a3b1569cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6817","nonce":3987433753}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6819","nonce":3987433753}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6823","nonce":3987433753}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3987433753},{"type":"v1","addr":"172.21.15.98:6821","nonce":3987433753}]},"public_addr":"172.21.15.98:6817/3987433753","cluster_addr":"172.21.15.98:6819/3987433753","heartbeat_back_addr":"172.21.15.98:6823/3987433753","heartbeat_front_addr":"172.21.15.98:6821/3987433753","state":["exists","up"]},{"osd":3,"uuid":"f33219e6-7712-40ac-8140-765749273dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6801","nonce":373809828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6803","nonce":373809828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6807","nonce":373809828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":373809828},{"type":"v1","addr":"172.21.15.190:6805","nonce":373809828}]},"public_addr":"172.21.15.190:6801/373809828","cluster_addr":"172.21.15.190:6803/373809828","heartbeat_back_addr":"172.21.15.190:6807/373809828","heartbeat_front_addr":"172.21.15.190:6805/373809828","state":["exists","up"]},{"osd":4,"uuid":"655b63d7-a358-4460-85ff-bf7b2330f230","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6817","nonce":1271446291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6819","nonce":1271446291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6823","nonce":1271446291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":1271446291},{"type":"v1","addr":"172.21.15.190:6821","nonce":1271446291}]},"public_addr":"172.21.15.190:6817/1271446291","cluster_addr":"172.21.15.190:6819/1271446291","heartbeat_back_addr":"172.21.15.190:6823/1271446291","heartbeat_front_addr":"172.21.15.190:6821/1271446291","state":["exists","up"]},{"osd":5,"uuid":"3d6351e6-ca58-4eb7-9b62-f52c423c19cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6809","nonce":1497028198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6811","nonce":1497028198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6815","nonce":1497028198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":1497028198},{"type":"v1","addr":"172.21.15.190:6813","nonce":1497028198}]},"public_addr":"172.21.15.190:6809/1497028198","cluster_addr":"172.21.15.190:6811/1497028198","heartbeat_back_addr":"172.21.15.190:6815/1497028198","heartbeat_front_addr":"172.21.15.190:6813/1497028198","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.670036+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.679318+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.766598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.015321+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:53.338711+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T02:49:52.969084+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":1,"length":4}]},{"pool":16,"snaps":[{"begin":4,"length":3}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":63,"snaps":[{"begin":1,"length":1}]},{"pool":92,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":103,"snaps":[{"begin":1,"length":1}]},{"pool":120,"snaps":[{"begin":1,"length":2}]},{"pool":137,"snaps":[{"begin":3,"length":1}]},{"pool":144,"snaps":[{"begin":2,"length":3}]},{"pool":150,"snaps":[{"begin":2,"length":1}]},{"pool":152,"snaps":[{"begin":2,"length":2}]},{"pool":162,"snaps":[{"begin":2,"length":3}]},{"pool":168,"snaps":[{"begin":3,"length":4}]},{"pool":171,"snaps":[{"begin":2,"length":1},{"begin":4,"length":2}]},{"pool":172,"snaps":[{"begin":2,"length":2}]},{"pool":183,"snaps":[{"begin":2,"length":1}]},{"pool":188,"snaps":[{"begin":3,"length":2}]},{"pool":203,"snaps":[{"begin":7,"length":1}]},{"pool":204,"snaps":[{"begin":4,"length":4}]},{"pool":228,"snaps":[{"begin":8,"length":1}]},{"pool":239,"snaps":[{"begin":10,"length":1}]},{"pool":245,"snaps":[{"begin":11,"length":2}]},{"pool":257,"snaps":[{"begin":15,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T03:08:05.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 10 librados: watch_flush enter 2024-09-11T03:08:05.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 10 librados: watch_flush exit 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 20 client.24952.objecter shutdown clearing up homeless session... 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 10 client.24952.objecter successfully canceled tick 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0fe003df40 msgr2=0x7f0fe0040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0fe003df40 0x7f0fe0040400 secure :-1 s=READY pgs=3713 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00ae000 tx=0x7f0ff0004740 comp rx=0 tx=0).stop 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 10 monclient: shutdown 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0ff406f650 msgr2=0x7f0ff41d1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0ff406f650 0x7f0ff41d1160 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f0fec00efc0 tx=0x7f0fec00c490 comp rx=0 tx=0).stop 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 shutdown_connections 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f0fe003df40 0x7f0fe0040400 unknown :-1 s=CLOSED pgs=3713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f0ff40a8380 0x7f0ff413b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f0ff406ff70 0x7f0ff4136800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 --2- 172.21.15.98:0/2478262963 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f0ff406f650 0x7f0ff41d1160 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 >> 172.21.15.98:0/2478262963 conn(0x7f0ff40ac8e0 msgr2=0x7f0ff40a6900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 shutdown_connections 2024-09-11T03:08:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 -- 172.21.15.98:0/2478262963 wait complete. 2024-09-11T03:08:05.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:05.791+0000 7f0ffb8e0640 1 librados: shutdown 2024-09-11T03:08:06.810 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T03:08:06.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T03:08:06.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient: get_monmap_and_config 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient: build_initial_monmap 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient: monmap: 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:06.949619+0000 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:06.949619+0000 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:06.947 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:06.948 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:06.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 1 Processor -- start 2024-09-11T03:08:06.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 1 -- start start 2024-09-11T03:08:06.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient: init 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient: _add_conns ranks=[0,2,8,6,3,9,10,7,5,4,1] 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f447c139b10 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c06fe30 0x7f447c0725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient(hunting): picked mon.noname-b con 0x7f447c06fe30 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f447c13ea00 0x7f447c140df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient(hunting): picked mon.noname-e con 0x7f447c13ea00 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f447c13a600 con 0x7f447c139b10 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f447c0a9650 con 0x7f447c06fe30 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f447c071350 con 0x7f447c13ea00 2024-09-11T03:08:06.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f44831f9640 10 monclient(hunting): authenticate will time out at 2040.829889s 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f447c13ea00 0x7f447c140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4480f95640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f447c139b10 0x7f447c139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:39704/0 (socket says 172.21.15.98:39704) 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4480f95640 1 -- 172.21.15.98:0/3934724941 learned_addr learned my addr 172.21.15.98:0/3934724941 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): get_auth_request con 0x7f447c13ea00 auth_method 0 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 1 --2- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c06fe30 0x7f447c0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:06.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 10 monclient(hunting): get_auth_request con 0x7f447c06fe30 auth_method 0 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient(hunting): handle_auth_done global_id 25541 payload 274 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 1 -- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c06fe30 msgr2=0x7f447c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 1 --2- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c06fe30 0x7f447c0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 1 -- 172.21.15.98:0/3934724941 >> v1:172.21.15.98:6789/0 conn(0x7f447c139b10 legacy=0x7f447c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient: _finish_hunting 0 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 1 monclient: found mon.noname-e 2024-09-11T03:08:06.953 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:06.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:06.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f4481796640 1 -- 172.21.15.98:0/3934724941 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c071630 con 0x7f447c13ea00 2024-09-11T03:08:06.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447b7fe640 1 -- 172.21.15.98:0/3934724941 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026585440 0 0) 0x7f447c13a600 con 0x7f447c139b10 2024-09-11T03:08:06.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.947+0000 7f447b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f4481796640 1 --2- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f447c13ea00 0x7f447c140df0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f4474009820 tx=0x7f447401c060 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a07cfb0858985a1f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 1 -- 172.21.15.98:0/3934724941 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f44740b1070 con 0x7f447c13ea00 2024-09-11T03:08:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:06.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: dump: 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:06.961 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 1 -- 172.21.15.98:0/3934724941 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44740a4ad0 con 0x7f447c13ea00 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 5 monclient: authenticate success, global_id 25541 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:06.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 1 -- 172.21.15.98:0/3934724941 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f44740a4cc0 con 0x7f447c13ea00 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 4 set_mon_vals no callback set 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447b7fe640 10 monclient: dump: 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:06.963 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient: shutdown 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f447c13ea00 msgr2=0x7f447c140df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 --2- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f447c13ea00 0x7f447c140df0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f4474009820 tx=0x7f447401c060 comp rx=0 tx=0).stop 2024-09-11T03:08:06.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- 172.21.15.98:0/3934724941 shutdown_connections 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 --2- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c06fe30 0x7f447c0725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 --2- 172.21.15.98:0/3934724941 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f447c13ea00 0x7f447c140df0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- 172.21.15.98:0/3934724941 >> 172.21.15.98:0/3934724941 conn(0x7f447c0ac770 msgr2=0x7f447c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- 172.21.15.98:0/3934724941 shutdown_connections 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- 172.21.15.98:0/3934724941 wait complete. 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient: build_initial_monmap 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient: monmap: 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:06.965 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 librados: starting msgr at 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 librados: starting objecter 2024-09-11T03:08:06.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 Processor -- start 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- start start 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 librados: setting wanted keys 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 librados: calling monclient init 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient: init 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient: _add_conns ranks=[3,2,7,5,8,1,6,0,4] 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f447c139b10 0x7f447c1d1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): picked mon.noname-g con 0x7f447c139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c13ea00 0x7f447c1389b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): picked mon.noname-b con 0x7f447c13ea00 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f447c138ef0 0x7f447c0a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): picked mon.noname-i con 0x7f447c138ef0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:06.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f447c144700 con 0x7f447c13ea00 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f447c144580 con 0x7f447c139b10 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f447c144880 con 0x7f447c138ef0 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f44831f9640 10 monclient(hunting): authenticate will time out at 2040.835454s 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c13ea00 0x7f447c1389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c13ea00 0x7f447c1389b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60460/0 (socket says 172.21.15.98:60460) 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 -- 172.21.15.98:0/3277751836 learned_addr learned my addr 172.21.15.98:0/3277751836 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): get_auth_request con 0x7f447c13ea00 auth_method 0 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:06.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient(hunting): handle_auth_done global_id 94412 payload 824 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 -- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f447c138ef0 msgr2=0x7f447c0a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f447c138ef0 0x7f447c0a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f4480f95640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f447c139b10 0x7f447c1d1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 -- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f447c139b10 msgr2=0x7f447c1d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f447c139b10 0x7f447c1d1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 monclient: found mon.noname-b 2024-09-11T03:08:06.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.951+0000 7f447bfff640 1 -- 172.21.15.98:0/3277751836 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c1d28a0 con 0x7f447c13ea00 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f447bfff640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c13ea00 0x7f447c1389b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f446c009e10 tx=0x7f446c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.?.objecter ms_handle_connect 0x7f447c13ea00 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f446c018020 con 0x7f447c13ea00 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: dump: 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.970 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:06.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: _finish_auth 0 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: _check_auth_tickets 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f446c00ce70 con 0x7f447c13ea00 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 5 monclient: authenticate success, global_id 94412 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 monclient: _renew_subs 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447c123060 con 0x7f447c13ea00 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f446c014d60 con 0x7f447c13ea00 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:06.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f447affd640 4 set_mon_vals no callback set 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 client.94412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: dump: 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:06.973 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 monclient: _renew_subs 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f447c0a8dd0 con 0x7f447c13ea00 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 1 librados: init done 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 client.94412.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 15 client.94412.objecter _session_command_op_assign -1 1 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 20 client.94412.objecter _get_session osd=-1 returning homeless 2024-09-11T03:08:06.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 client.94412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f447c06e850 con 0x7f447c13ea00 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f446c014480 con 0x7f447c13ea00 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f445c03df40 0x7f445c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f446c0506d0 con 0x7f447c13ea00 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.94412.objecter ms_dispatch 0x7f447c000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 3 client.94412.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 3 client.94412.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.94412.objecter checking command 1 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7f445c045700 0x7f445c047ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 client.94412.objecter _get_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 client.94412.objecter put_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.975 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 15 client.94412.objecter _session_command_op_remove -1 1 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 client.94412.objecter _get_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 client.94412.objecter get_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 15 client.94412.objecter _session_command_op_assign 0 1 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 client.94412.objecter put_session s=0x7f445c045030 osd=0 4 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.94412.objecter _send_command 1 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 --> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f446c00fe50 con 0x7f445c045700 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 client.94412.objecter dump_active .. 0 homeless 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=699) ==== 24+0+0 (secure 0 0 0) 0x7f446c057050 con 0x7f447c13ea00 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f4481796640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7f445c045700 0x7f445c047ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f4481796640 10 monclient: get_auth_request con 0x7f445c045700 auth_method 0 2024-09-11T03:08:06.976 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f4480f95640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f445c03df40 0x7f445c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f4480f95640 10 monclient: get_auth_request con 0x7f445c03df40 auth_method 0 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f4481796640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7f445c045700 0x7f445c047ba0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.94412.objecter ms_handle_connect 0x7f445c045700 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.94412.objecter ms_handle_connect 0x7f445c03df40 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f4480f95640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f445c03df40 0x7f445c040400 secure :-1 s=READY pgs=3714 cs=0 l=1 rev1=1 crypto rx=0x7f4464000f70 tx=0x7f4464009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== osd.0 v2:172.21.15.98:6800/1456395826 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f446c00fe50 con 0x7f445c045700 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.94412.objecter ms_dispatch 0x7f447c000ba0 command_reply(tid 1: 0 ) 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 10 client.94412.objecter _finish_command 1 = system:0 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 20 client.94412.objecter put_session s=0x7f445c045030 osd=0 4 2024-09-11T03:08:06.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.955+0000 7f44797fa640 15 client.94412.objecter _session_command_op_remove 0 1 2024-09-11T03:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 10 client.94412.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T03:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 15 client.94412.objecter _session_command_op_assign -1 2 2024-09-11T03:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 20 client.94412.objecter _get_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 20 client.94412.objecter put_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 20 client.94412.objecter _get_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 15 client.94412.objecter _session_command_op_remove -1 2 2024-09-11T03:08:06.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 20 client.94412.objecter get_session s=0x7f445c045030 osd=0 3 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 15 client.94412.objecter _session_command_op_assign 0 2 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 20 client.94412.objecter put_session s=0x7f445c045030 osd=0 4 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 10 client.94412.objecter _send_command 2 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.967+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 --> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f447c13e210 con 0x7f445c045700 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44797fa640 1 -- 172.21.15.98:0/3277751836 <== osd.0 v2:172.21.15.98:6800/1456395826 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f447c13e210 con 0x7f445c045700 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44797fa640 10 client.94412.objecter ms_dispatch 0x7f447c000ba0 command_reply(tid 2: 0 ) 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44797fa640 10 client.94412.objecter _finish_command 2 = system:0 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44797fa640 20 client.94412.objecter put_session s=0x7f445c045030 osd=0 4 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44797fa640 15 client.94412.objecter _session_command_op_remove 0 2 2024-09-11T03:08:06.979 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:06.980 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T03:08:06.980 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 10 librados: watch_flush enter 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 10 librados: watch_flush exit 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 10 client.94412.objecter close_session for osd.0 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7f445c045700 msgr2=0x7f445c047ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:6800/1456395826,v1:172.21.15.98:6801/1456395826] conn(0x7f445c045700 0x7f445c047ba0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 20 client.94412.objecter put_session s=0x7f445c045030 osd=0 1 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 20 client.94412.objecter shutdown clearing up homeless session... 2024-09-11T03:08:06.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 10 client.94412.objecter successfully canceled tick 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f445c03df40 msgr2=0x7f445c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f445c03df40 0x7f445c040400 secure :-1 s=READY pgs=3714 cs=0 l=1 rev1=1 crypto rx=0x7f4464000f70 tx=0x7f4464009040 comp rx=0 tx=0).stop 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 10 monclient: shutdown 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c13ea00 msgr2=0x7f447c1389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 --2- 172.21.15.98:0/3277751836 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f447c13ea00 0x7f447c1389b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f446c009e10 tx=0x7f446c00c490 comp rx=0 tx=0).stop 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f4481796640 1 -- 172.21.15.98:0/3277751836 reap_dead start 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 shutdown_connections 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 >> 172.21.15.98:0/3277751836 conn(0x7f447c0ac770 msgr2=0x7f447c13f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 shutdown_connections 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 -- 172.21.15.98:0/3277751836 wait complete. 2024-09-11T03:08:06.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:06.971+0000 7f44831f9640 1 librados: shutdown 2024-09-11T03:08:06.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T03:08:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.095+0000 7f523ff1d640 10 monclient: get_monmap_and_config 2024-09-11T03:08:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.095+0000 7f523ff1d640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.095+0000 7f523ff1d640 10 monclient: monmap: 2024-09-11T03:08:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.098 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:07.101012+0000 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:07.101012+0000 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.099 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:07.100 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:07.100 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 Processor -- start 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 -- start start 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient: init 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient: _add_conns ranks=[9,3,8,4,2,5,10,1,6,0,7] 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5238139b80 0x7f5238139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5238139b80 addr v2:172.21.15.98:3300/0 2024-09-11T03:08:07.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f523806fe50 0x7f52380a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): picked mon.noname-g con 0x7f523806fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5238072640 0x7f52380a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): picked mon.noname-e con 0x7f5238072640 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f52380719c0 con 0x7f5238139b80 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f5238071c10 con 0x7f523806fe50 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f52380702d0 con 0x7f5238072640 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523ff1d640 10 monclient(hunting): authenticate will time out at 2040.980998s 2024-09-11T03:08:07.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5238139b80 0x7f5238139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5238072640 0x7f52380a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f5238139b80 0x7f5238139f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54068/0 (socket says 172.21.15.98:54068) 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): get_auth_request con 0x7f5238072640 auth_method 0 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 1 -- 172.21.15.98:0/3403214641 learned_addr learned my addr 172.21.15.98:0/3403214641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 10 monclient(hunting): get_auth_request con 0x7f5238139b80 auth_method 0 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523d4b8640 1 --2- 172.21.15.98:0/3403214641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f523806fe50 0x7f52380a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523d4b8640 10 monclient(hunting): get_auth_request con 0x7f523806fe50 auth_method 0 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523d4b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523d4b8640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523d4b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient(hunting): handle_auth_done global_id 25550 payload 274 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 -- 172.21.15.98:0/3403214641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f523806fe50 msgr2=0x7f52380a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 --2- 172.21.15.98:0/3403214641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f523806fe50 0x7f52380a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 -- 172.21.15.98:0/3403214641 >> v2:172.21.15.98:3300/0 conn(0x7f5238139b80 msgr2=0x7f5238139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 --2- 172.21.15.98:0/3403214641 >> v2:172.21.15.98:3300/0 conn(0x7f5238139b80 0x7f5238139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 monclient: found mon.noname-e 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 -- 172.21.15.98:0/3403214641 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5238071370 con 0x7f5238072640 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523dcb9640 1 --2- 172.21.15.98:0/3403214641 >> v2:172.21.15.98:3300/0 conn(0x7f5238139b80 0x7f5238139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.099+0000 7f523e4ba640 1 --2- 172.21.15.98:0/3403214641 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5238072640 0x7f52380a7f50 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f5230009d90 tx=0x7f523001c830 comp rx=0 tx=0).ready entity=mon.8 client_cookie=899dd75f0cd8372a server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 1 -- 172.21.15.98:0/3403214641 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52300b2070 con 0x7f5238072640 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: dump: 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.107 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.108 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 5 monclient: authenticate success, global_id 25550 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 1 -- 172.21.15.98:0/3403214641 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52300a59f0 con 0x7f5238072640 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 1 -- 172.21.15.98:0/3403214641 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52300a5c20 con 0x7f5238072640 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 4 set_mon_vals no callback set 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ccb7640 10 monclient: dump: 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.110 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 10 monclient: shutdown 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 -- 172.21.15.98:0/3403214641 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5238072640 msgr2=0x7f52380a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 --2- 172.21.15.98:0/3403214641 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5238072640 0x7f52380a7f50 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f5230009d90 tx=0x7f523001c830 comp rx=0 tx=0).stop 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 -- 172.21.15.98:0/3403214641 shutdown_connections 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 --2- 172.21.15.98:0/3403214641 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f5238072640 0x7f52380a7f50 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 --2- 172.21.15.98:0/3403214641 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f523806fe50 0x7f52380a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 --2- 172.21.15.98:0/3403214641 >> v2:172.21.15.98:3300/0 conn(0x7f5238139b80 0x7f5238139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 -- 172.21.15.98:0/3403214641 >> 172.21.15.98:0/3403214641 conn(0x7f52380ac730 msgr2=0x7f52380a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 -- 172.21.15.98:0/3403214641 shutdown_connections 2024-09-11T03:08:07.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.103+0000 7f523ff1d640 1 -- 172.21.15.98:0/3403214641 wait complete. 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: monmap: 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.117 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 librados: starting msgr at 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 librados: starting objecter 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 Processor -- start 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 -- start start 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 librados: setting wanted keys 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 librados: calling monclient init 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: init 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _add_conns ranks=[1,2,5,4,6,3,7,0,8] 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f523806fe50 0x7f5238134770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): picked mon.noname-f con 0x7f523806fe50 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5238072640 0x7f5238134cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5238072640 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5238139b80 0x7f52381351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): picked mon.noname-h con 0x7f5238139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f52381435c0 con 0x7f523806fe50 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f5238143440 con 0x7f5238072640 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f5238143740 con 0x7f5238139b80 2024-09-11T03:08:07.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient(hunting): authenticate will time out at 2040.989040s 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5238072640 0x7f5238134cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5238072640 0x7f5238134cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:60472/0 (socket says 172.21.15.98:60472) 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 -- 172.21.15.98:0/2147485358 learned_addr learned my addr 172.21.15.98:0/2147485358 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): get_auth_request con 0x7f5238072640 auth_method 0 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523dcb9640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f523806fe50 0x7f5238134770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523e4ba640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5238139b80 0x7f52381351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523dcb9640 10 monclient(hunting): get_auth_request con 0x7f523806fe50 auth_method 0 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523dcb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523dcb9640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523dcb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523e4ba640 10 monclient(hunting): get_auth_request con 0x7f5238139b80 auth_method 0 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523e4ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523e4ba640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523e4ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient(hunting): handle_auth_done global_id 94421 payload 824 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 -- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5238139b80 msgr2=0x7f52381351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5238139b80 0x7f52381351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 -- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f523806fe50 msgr2=0x7f5238134770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f523806fe50 0x7f5238134770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 monclient: found mon.noname-b 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 -- 172.21.15.98:0/2147485358 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523813bf40 con 0x7f5238072640 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523d4b8640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5238072640 0x7f5238134cb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f523400d800 tx=0x7f523400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 client.?.objecter ms_handle_connect 0x7f5238072640 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 1 -- 172.21.15.98:0/2147485358 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f523400bb30 con 0x7f5238072640 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: dump: 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.122 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.123 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 5 monclient: authenticate success, global_id 94421 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _renew_subs 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 1 -- 172.21.15.98:0/2147485358 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5234019070 con 0x7f5238072640 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f523813e2a0 con 0x7f5238072640 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 1 -- 172.21.15.98:0/2147485358 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f523400bcf0 con 0x7f5238072640 2024-09-11T03:08:07.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 client.94421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 monclient: dump: 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.125 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _renew_subs 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f523813dfc0 con 0x7f5238072640 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 librados: init done 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522ffff640 4 set_mon_vals no callback set 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 1 -- 172.21.15.98:0/2147485358 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f5234013070 con 0x7f5238072640 2024-09-11T03:08:07.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5220005090 con 0x7f5238072640 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f521c03df40 0x7f521c040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 1 -- 172.21.15.98:0/2147485358 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f523400f040 con 0x7f5238072640 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 10 client.94421.objecter ms_dispatch 0x7f5238000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 3 client.94421.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 3 client.94421.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f522e7fc640 20 client.94421.objecter dump_active .. 0 homeless 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523dcb9640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f521c03df40 0x7f521c040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.107+0000 7f523dcb9640 10 monclient: get_auth_request con 0x7f521c03df40 auth_method 0 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.111+0000 7f522e7fc640 10 client.94421.objecter ms_handle_connect 0x7f521c03df40 2024-09-11T03:08:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.111+0000 7f523dcb9640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f521c03df40 0x7f521c040400 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7f52280099c0 tx=0x7f5228008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.115+0000 7f522e7fc640 1 -- 172.21.15.98:0/2147485358 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f5234022020 con 0x7f5238072640 2024-09-11T03:08:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.115+0000 7f522e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.115+0000 7f522e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:07.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.283+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f5220000bd0 con 0x7f521c03df40 2024-09-11T03:08:07.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.283+0000 7f522e7fc640 1 -- 172.21.15.98:0/2147485358 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5220000bd0 con 0x7f521c03df40 2024-09-11T03:08:07.285 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T03:08:07.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 10 librados: watch_flush enter 2024-09-11T03:08:07.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 10 librados: watch_flush exit 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 20 client.94421.objecter shutdown clearing up homeless session... 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 10 client.94421.objecter successfully canceled tick 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f521c03df40 msgr2=0x7f521c040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f521c03df40 0x7f521c040400 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7f52280099c0 tx=0x7f5228008040 comp rx=0 tx=0).stop 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 10 monclient: shutdown 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5238072640 msgr2=0x7f5238134cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5238072640 0x7f5238134cb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f523400d800 tx=0x7f523400dcc0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 shutdown_connections 2024-09-11T03:08:07.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f5238139b80 0x7f52381351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f521c03df40 0x7f521c040400 unknown :-1 s=CLOSED pgs=3715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f5238072640 0x7f5238134cb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 --2- 172.21.15.98:0/2147485358 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f523806fe50 0x7f5238134770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 >> 172.21.15.98:0/2147485358 conn(0x7f52380ac730 msgr2=0x7f52380ab4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 shutdown_connections 2024-09-11T03:08:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 -- 172.21.15.98:0/2147485358 wait complete. 2024-09-11T03:08:07.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.287+0000 7f523ff1d640 1 librados: shutdown 2024-09-11T03:08:07.305 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T03:08:07.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.423+0000 7f62c716a640 10 monclient: get_monmap_and_config 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.423+0000 7f62c716a640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.423+0000 7f62c716a640 10 monclient: monmap: 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:07.428956+0000 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:07.428956+0000 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.426 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:07.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 1 Processor -- start 2024-09-11T03:08:07.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 1 -- start start 2024-09-11T03:08:07.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient: init 2024-09-11T03:08:07.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient: _add_conns ranks=[7,1,0,6,8,5,4,10,9,2,3] 2024-09-11T03:08:07.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62c00a7aa0 0x7f62c00a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient(hunting): picked mon.noname-i con 0x7f62c00a7aa0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f62c00a1600 addr v1:172.21.15.190:6789/0 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62c00a1dc0 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62c013a7c0 con 0x7f62c00a1dc0 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62c00a1a80 con 0x7f62c00a1600 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f62c00a3170 con 0x7f62c00a7aa0 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 10 monclient(hunting): authenticate will time out at 2041.309234s 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f62c00a1dc0 0x7f62c013a0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48176/0 (socket says 172.21.15.98:48176) 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 1 -- 172.21.15.98:0/3142440890 learned_addr learned my addr 172.21.15.98:0/3142440890 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1698211706 0 0) 0x7f62c013a7c0 con 0x7f62c00a1dc0 2024-09-11T03:08:07.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): _negotiate 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62c00a7aa0 0x7f62c00a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): my global_id is 104256 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62a8003530 con 0x7f62c00a1dc0 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): get_auth_request con 0x7f62c00a7aa0 auth_method 0 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2444411472 0 0) 0x7f62a8003530 con 0x7f62c00a1dc0 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62c00a7aa0 msgr2=0x7f62c00a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62c00a7aa0 0x7f62c00a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 >> v1:172.21.15.190:6789/0 conn(0x7f62c00a1600 legacy=0x7f62c01399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c00a96b0 con 0x7f62c00a1dc0 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f62b8001cf0 con 0x7f62c00a1dc0 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62c00a7aa0 0x7f62c00a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:08:07.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: dump: 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.441 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 >> v1:172.21.15.98:6789/0 conn(0x7f62c00a1dc0 legacy=0x7f62c013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient: _add_conns ranks=[5,6,1,7,4,2,3,0,8] 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f62a8004270 0x7f62a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): picked mon.h con 0x7f62a8004270 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62a8004ec0 0x7f62a80099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): picked mon.d con 0x7f62a8004ec0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62a8009f00 0x7f62a800c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): picked mon.f con 0x7f62a8009f00 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f62c00a3170 con 0x7f62a8009f00 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f62a800cac0 con 0x7f62a8004270 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f62a800cc70 con 0x7f62a8004ec0 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bf7fe640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62a8004ec0 0x7f62a80099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): get_auth_request con 0x7f62a8004ec0 auth_method 0 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f62a8004270 0x7f62a8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62a8009f00 0x7f62a800c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 10 monclient(hunting): get_auth_request con 0x7f62a8009f00 auth_method 0 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): get_auth_request con 0x7f62a8004270 auth_method 0 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c4f06640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient(hunting): handle_auth_done global_id 104256 payload 293 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 1 -- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f62a8004270 msgr2=0x7f62a8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f62a8004270 0x7f62a8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 1 -- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62a8009f00 msgr2=0x7f62a800c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62a8009f00 0x7f62a800c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 1 monclient: found mon.d 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c5707640 1 -- 172.21.15.98:0/3142440890 reap_dead start 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 1 -- 172.21.15.98:0/3142440890 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62a8003530 con 0x7f62a8004ec0 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 5 monclient: authenticate success, global_id 104256 2024-09-11T03:08:07.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.427+0000 7f62c716a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bffff640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62a8004ec0 0x7f62a80099c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62b4007fd0 tx=0x7f62b401c350 comp rx=0 tx=0).ready entity=mon.6 client_cookie=fa5e28368dd1637 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f62b40013e0 con 0x7f62a8004ec0 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 10 monclient: dump: 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.446 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b4003600 con 0x7f62a8004ec0 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 4 set_mon_vals no callback set 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 1 -- 172.21.15.98:0/3142440890 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f62b4003c60 con 0x7f62a8004ec0 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bf7fe640 10 monclient: dump: 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.448 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: shutdown 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- 172.21.15.98:0/3142440890 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62a8004ec0 msgr2=0x7f62a80099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62a8004ec0 0x7f62a80099c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f62b4007fd0 tx=0x7f62b401c350 comp rx=0 tx=0).stop 2024-09-11T03:08:07.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- 172.21.15.98:0/3142440890 shutdown_connections 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 --2- 172.21.15.98:0/3142440890 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62a8004ec0 0x7f62a80099c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- 172.21.15.98:0/3142440890 >> 172.21.15.98:0/3142440890 conn(0x7f62c00ac820 msgr2=0x7f62c00a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- 172.21.15.98:0/3142440890 shutdown_connections 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- 172.21.15.98:0/3142440890 wait complete. 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: monmap: 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:07.450 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 librados: starting msgr at 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 librados: starting objecter 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 Processor -- start 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- start start 2024-09-11T03:08:07.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 librados: setting wanted keys 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 librados: calling monclient init 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: init 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: _add_conns ranks=[7,3,6,2,4,8,0,5,1] 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62a8004ec0 0x7f62c01226e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): picked mon.noname-i con 0x7f62a8004ec0 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62c0122c20 0x7f62c0123000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): picked mon.noname-g con 0x7f62c0122c20 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62c0137230 0x7f62c0137670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): picked mon.noname-d con 0x7f62c0137230 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f62c00517e0 con 0x7f62c0122c20 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f62c00a88a0 con 0x7f62c0137230 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f62c01399a0 con 0x7f62a8004ec0 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient(hunting): authenticate will time out at 2041.314925s 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62c0137230 0x7f62c0137670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62c0137230 0x7f62c0137670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:42934/0 (socket says 172.21.15.98:42934) 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 -- 172.21.15.98:0/3460119600 learned_addr learned my addr 172.21.15.98:0/3460119600 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): get_auth_request con 0x7f62c0137230 auth_method 0 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bffff640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62c0122c20 0x7f62c0123000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c4f06640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62a8004ec0 0x7f62c01226e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient(hunting): handle_auth_done global_id 84768 payload 824 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 -- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62a8004ec0 msgr2=0x7f62c01226e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f62a8004ec0 0x7f62c01226e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 -- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62c0122c20 msgr2=0x7f62c0123000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62c0122c20 0x7f62c0123000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 monclient: found mon.noname-d 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 -- 172.21.15.98:0/3460119600 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c0073130 con 0x7f62c0137230 2024-09-11T03:08:07.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bffff640 10 monclient: get_auth_request con 0x7f62c0122c20 auth_method 0 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bffff640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f62c0122c20 0x7f62c0123000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c5707640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62c0137230 0x7f62c0137670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62b8000f90 tx=0x7f62b8007a60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 client.?.objecter ms_handle_connect 0x7f62c0137230 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f62b800edd0 con 0x7f62c0137230 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: dump: 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.455 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 5 monclient: authenticate success, global_id 84768 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b8018070 con 0x7f62c0137230 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: _renew_subs 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c0138b40 con 0x7f62c0137230 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f62b8013890 con 0x7f62c0137230 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 client.84768.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62beffd640 4 set_mon_vals no callback set 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62bd7fa640 10 monclient: dump: 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.457 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.458 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: _renew_subs 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c00ab200 con 0x7f62c0137230 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.431+0000 7f62c716a640 1 librados: init done 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f62b8013420 con 0x7f62c0137230 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c716a640 10 client.84768.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c716a640 15 client.84768.objecter _session_command_op_assign -1 1 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c716a640 20 client.84768.objecter _get_session osd=-1 returning homeless 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c716a640 10 client.84768.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c716a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f62c00695f0 con 0x7f62c0137230 2024-09-11T03:08:07.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f62ac03df40 0x7f62ac040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f62b8010040 con 0x7f62c0137230 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 client.84768.objecter ms_dispatch 0x7f62c0000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 3 client.84768.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 3 client.84768.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 client.84768.objecter checking command 1 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7f62ac045700 0x7f62ac047ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 20 client.84768.objecter _get_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 20 client.84768.objecter put_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 15 client.84768.objecter _session_command_op_remove -1 1 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 20 client.84768.objecter _get_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 20 client.84768.objecter get_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 15 client.84768.objecter _session_command_op_assign 1 1 2024-09-11T03:08:07.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 20 client.84768.objecter put_session s=0x7f62ac045030 osd=1 4 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 client.84768.objecter _send_command 1 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62b801fe70 con 0x7f62ac045700 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 20 client.84768.objecter dump_active .. 0 homeless 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_get_version_reply(handle=1 version=699) ==== 24+0+0 (secure 0 0 0) 0x7f62b8056050 con 0x7f62c0137230 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bffff640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7f62ac045700 0x7f62ac047ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bffff640 10 monclient: get_auth_request con 0x7f62ac045700 auth_method 0 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c4f06640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f62ac03df40 0x7f62ac040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c4f06640 10 monclient: get_auth_request con 0x7f62ac03df40 auth_method 0 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 client.84768.objecter ms_handle_connect 0x7f62ac045700 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bffff640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7f62ac045700 0x7f62ac047ba0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62c4f06640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f62ac03df40 0x7f62ac040400 secure :-1 s=READY pgs=3716 cs=0 l=1 rev1=1 crypto rx=0x7f62b000c8a0 tx=0x7f62b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 client.84768.objecter ms_handle_connect 0x7f62ac03df40 2024-09-11T03:08:07.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== osd.1 v2:172.21.15.98:6808/2313933591 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62b801fe70 con 0x7f62ac045700 2024-09-11T03:08:07.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 client.84768.objecter ms_dispatch 0x7f62c0000ba0 command_reply(tid 1: 0 ) 2024-09-11T03:08:07.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 10 client.84768.objecter _finish_command 1 = system:0 2024-09-11T03:08:07.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 20 client.84768.objecter put_session s=0x7f62ac045030 osd=1 4 2024-09-11T03:08:07.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.435+0000 7f62bd7fa640 15 client.84768.objecter _session_command_op_remove 1 1 2024-09-11T03:08:07.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 10 client.84768.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 15 client.84768.objecter _session_command_op_assign -1 2 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 client.84768.objecter _get_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 client.84768.objecter put_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 client.84768.objecter _get_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 15 client.84768.objecter _session_command_op_remove -1 2 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 client.84768.objecter get_session s=0x7f62ac045030 osd=1 3 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 15 client.84768.objecter _session_command_op_assign 1 2 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 client.84768.objecter put_session s=0x7f62ac045030 osd=1 4 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 10 client.84768.objecter _send_command 2 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 --> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f62c0139ca0 con 0x7f62ac045700 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62bd7fa640 1 -- 172.21.15.98:0/3460119600 <== osd.1 v2:172.21.15.98:6808/2313933591 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f62c0139ca0 con 0x7f62ac045700 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62bd7fa640 10 client.84768.objecter ms_dispatch 0x7f62c0000ba0 command_reply(tid 2: 0 ) 2024-09-11T03:08:07.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62bd7fa640 10 client.84768.objecter _finish_command 2 = system:0 2024-09-11T03:08:07.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62bd7fa640 20 client.84768.objecter put_session s=0x7f62ac045030 osd=1 4 2024-09-11T03:08:07.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62bd7fa640 15 client.84768.objecter _session_command_op_remove 1 2 2024-09-11T03:08:07.464 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:07.464 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T03:08:07.464 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 10 librados: watch_flush enter 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 10 librados: watch_flush exit 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 10 client.84768.objecter close_session for osd.1 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7f62ac045700 msgr2=0x7f62ac047ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:6808/2313933591,v1:172.21.15.98:6809/2313933591] conn(0x7f62ac045700 0x7f62ac047ba0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 client.84768.objecter put_session s=0x7f62ac045030 osd=1 1 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 client.84768.objecter shutdown clearing up homeless session... 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 10 client.84768.objecter successfully canceled tick 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f62ac03df40 msgr2=0x7f62ac040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f62ac03df40 0x7f62ac040400 secure :-1 s=READY pgs=3716 cs=0 l=1 rev1=1 crypto rx=0x7f62b000c8a0 tx=0x7f62b0008040 comp rx=0 tx=0).stop 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 10 monclient: shutdown 2024-09-11T03:08:07.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62c0137230 msgr2=0x7f62c0137670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 --2- 172.21.15.98:0/3460119600 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f62c0137230 0x7f62c0137670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f62b8000f90 tx=0x7f62b8007a60 comp rx=0 tx=0).stop 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c5707640 1 -- 172.21.15.98:0/3460119600 reap_dead start 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 shutdown_connections 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 >> 172.21.15.98:0/3460119600 conn(0x7f62c00ac820 msgr2=0x7f62c00a4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.447+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 shutdown_connections 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.451+0000 7f62c716a640 1 -- 172.21.15.98:0/3460119600 wait complete. 2024-09-11T03:08:07.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.451+0000 7f62c716a640 1 librados: shutdown 2024-09-11T03:08:07.467 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient: get_monmap_and_config 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient: monmap: 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:07.581867+0000 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:07.581867+0000 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.579 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:07.580 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 1 Processor -- start 2024-09-11T03:08:07.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 1 -- start start 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient: init 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient: _add_conns ranks=[0,10,1,6,5,8,7,4,3,9,2] 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24e4139b80 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f24e406fe50 0x7f24e413ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient(hunting): picked mon.noname-f con 0x7f24e406fe50 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f24e4072640 addr v1:172.21.15.190:6789/0 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24e4071370 con 0x7f24e4139b80 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24e40a9600 con 0x7f24e4072640 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f24e40719c0 con 0x7f24e406fe50 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 10 monclient(hunting): authenticate will time out at 2041.462150s 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e930d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f24e4139b80 0x7f24e4139f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48186/0 (socket says 172.21.15.98:48186) 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e930d640 1 -- 172.21.15.98:0/3943754619 learned_addr learned my addr 172.21.15.98:0/3943754619 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/3943754619 >> v2:172.21.15.190:3300/0 conn(0x7f24e406fe50 0x7f24e413ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 266586654 0 0) 0x7f24e4071370 con 0x7f24e4139b80 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): _negotiate 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): my global_id is 104265 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24d0003530 con 0x7f24e4139b80 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): get_auth_request con 0x7f24e406fe50 auth_method 0 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 289001148 0 0) 0x7f24d0003530 con 0x7f24e4139b80 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 >> v2:172.21.15.190:3300/0 conn(0x7f24e406fe50 msgr2=0x7f24e413ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 --2- 172.21.15.98:0/3943754619 >> v2:172.21.15.190:3300/0 conn(0x7f24e406fe50 0x7f24e413ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 >> v1:172.21.15.190:6789/0 conn(0x7f24e4072640 legacy=0x7f24e4140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e4144690 con 0x7f24e4139b80 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f24e0001970 con 0x7f24e4139b80 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: dump: 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.593 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 >> v1:172.21.15.98:6789/0 conn(0x7f24e4139b80 legacy=0x7f24e4139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient: _add_conns ranks=[3,8,1,6,2,4,0,5,7] 2024-09-11T03:08:07.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24d0004210 0x7f24d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): picked mon.g con 0x7f24d0004210 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f24d0004da0 0x7f24d0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): picked mon.e con 0x7f24d0004da0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0009c40 0x7f24d000c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): picked mon.f con 0x7f24d0009c40 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f24e40719c0 con 0x7f24d0009c40 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f24d000c9b0 con 0x7f24d0004210 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f24d000cb60 con 0x7f24d0004da0 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24dbfff640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:08:07.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f24d0004da0 0x7f24d0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): get_auth_request con 0x7f24d0004da0 auth_method 0 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e930d640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24d0004210 0x7f24d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e930d640 10 monclient(hunting): get_auth_request con 0x7f24d0004210 auth_method 0 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e930d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e930d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e930d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_done global_id 104265 payload 293 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 -- 172.21.15.98:0/3943754619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24d0004210 msgr2=0x7f24d0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f24d0004210 0x7f24d0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 -- 172.21.15.98:0/3943754619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0009c40 msgr2=0x7f24d000c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0009c40 0x7f24d000c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 monclient: found mon.e 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e9b0e640 1 -- 172.21.15.98:0/3943754619 reap_dead start 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 1 -- 172.21.15.98:0/3943754619 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d0003530 con 0x7f24d0004da0 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24e8b0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 5 monclient: authenticate success, global_id 104265 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.579+0000 7f24eb571640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f24d0004da0 0x7f24d0005220 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009ab0 tx=0x7f24d401c9f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=8642a88eeb883b99 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24d40096b0 con 0x7f24d0004da0 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 10 monclient: dump: 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.598 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24d40a88a0 con 0x7f24d0004da0 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 1 -- 172.21.15.98:0/3943754619 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24d40b0400 con 0x7f24d0004da0 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:07.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24dbfff640 10 monclient: dump: 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.600 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.601 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 4 set_mon_vals no callback set 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient: shutdown 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 -- 172.21.15.98:0/3943754619 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f24d0004da0 msgr2=0x7f24d0005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f24d0004da0 0x7f24d0005220 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009ab0 tx=0x7f24d401c9f0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 -- 172.21.15.98:0/3943754619 shutdown_connections 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 --2- 172.21.15.98:0/3943754619 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f24d0004da0 0x7f24d0005220 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 -- 172.21.15.98:0/3943754619 >> 172.21.15.98:0/3943754619 conn(0x7f24e40ac730 msgr2=0x7f24e40a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 -- 172.21.15.98:0/3943754619 shutdown_connections 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 -- 172.21.15.98:0/3943754619 wait complete. 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient: monmap: 2024-09-11T03:08:07.602 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.603 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 librados: starting msgr at 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 librados: starting objecter 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 Processor -- start 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 -- start start 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 librados: setting wanted keys 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 librados: calling monclient init 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient: init 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient: _add_conns ranks=[1,2,5,0,7,6,3,8,4] 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0004da0 0x7f24e4135a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 10 monclient(hunting): picked mon.noname-f con 0x7f24d0004da0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.583+0000 7f24eb571640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e413fee0 0x7f24e4135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient(hunting): picked mon.noname-b con 0x7f24e413fee0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:07.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f24e413dc50 0x7f24e413e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient(hunting): picked mon.noname-h con 0x7f24e413dc50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f24e40a85b0 con 0x7f24d0004da0 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f24e40a8430 con 0x7f24e413fee0 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f24e40a8730 con 0x7f24e413dc50 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient(hunting): authenticate will time out at 2041.468053s 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e413fee0 0x7f24e4135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e413fee0 0x7f24e4135f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:42146/0 (socket says 172.21.15.98:42146) 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 -- 172.21.15.98:0/2177577574 learned_addr learned my addr 172.21.15.98:0/2177577574 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:07.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): get_auth_request con 0x7f24e413fee0 auth_method 0 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e930d640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0004da0 0x7f24e4135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e9b0e640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f24e413dc50 0x7f24e413e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient(hunting): handle_auth_done global_id 94430 payload 824 2024-09-11T03:08:07.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 -- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f24e413dc50 msgr2=0x7f24e413e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f24e413dc50 0x7f24e413e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 -- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0004da0 msgr2=0x7f24e4135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0004da0 0x7f24e4135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 monclient: found mon.noname-b 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 -- 172.21.15.98:0/2177577574 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e4140e40 con 0x7f24e413fee0 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e930d640 10 monclient: get_auth_request con 0x7f24d0004da0 auth_method 0 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e930d640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0004da0 0x7f24e4135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e9b0e640 10 monclient: get_auth_request con 0x7f24e413dc50 auth_method 0 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e9b0e640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f24e413dc50 0x7f24e413e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e8b0c640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e413fee0 0x7f24e4135f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009a00 tx=0x7f24d0005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 client.?.objecter ms_handle_connect 0x7f24e413fee0 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 1 -- 172.21.15.98:0/2177577574 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24d40096b0 con 0x7f24e413fee0 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: dump: 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.608 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 1 -- 172.21.15.98:0/2177577574 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24d4004250 con 0x7f24e413fee0 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 1 -- 172.21.15.98:0/2177577574 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24d40a8e60 con 0x7f24e413fee0 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 5 monclient: authenticate success, global_id 94430 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient: _renew_subs 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e41364c0 con 0x7f24e413fee0 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24db7fe640 4 set_mon_vals no callback set 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 monclient: dump: 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 client.94430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient: _renew_subs 2024-09-11T03:08:07.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e413e5b0 con 0x7f24e413fee0 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 librados: init done 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e413c9f0 con 0x7f24e413fee0 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 1 -- 172.21.15.98:0/2177577574 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f24d4003620 con 0x7f24e413fee0 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f24c803df70 0x7f24c8040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 1 -- 172.21.15.98:0/2177577574 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f24d400f040 con 0x7f24e413fee0 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 client.94430.objecter ms_dispatch 0x7f24e4000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 3 client.94430.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 3 client.94430.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 20 client.94430.objecter dump_active .. 0 homeless 2024-09-11T03:08:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e930d640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f24c803df70 0x7f24c8040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e930d640 10 monclient: get_auth_request con 0x7f24c803df70 auth_method 0 2024-09-11T03:08:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24d9ffb640 10 client.94430.objecter ms_handle_connect 0x7f24c803df70 2024-09-11T03:08:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.587+0000 7f24e930d640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f24c803df70 0x7f24c8040430 secure :-1 s=READY pgs=3717 cs=0 l=1 rev1=1 crypto rx=0x7f24e0001250 tx=0x7f24e0000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.591+0000 7f24d9ffb640 1 -- 172.21.15.98:0/2177577574 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f24d40ac030 con 0x7f24e413fee0 2024-09-11T03:08:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.591+0000 7f24d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.591+0000 7f24d9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:07.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.759+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f24e41442f0 con 0x7f24c803df70 2024-09-11T03:08:07.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.759+0000 7f24d9ffb640 1 -- 172.21.15.98:0/2177577574 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f24e41442f0 con 0x7f24c803df70 2024-09-11T03:08:07.761 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 10 librados: watch_flush enter 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 10 librados: watch_flush exit 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 20 client.94430.objecter shutdown clearing up homeless session... 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 10 client.94430.objecter successfully canceled tick 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f24c803df70 msgr2=0x7f24c8040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f24c803df70 0x7f24c8040430 secure :-1 s=READY pgs=3717 cs=0 l=1 rev1=1 crypto rx=0x7f24e0001250 tx=0x7f24e0000ee0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 10 monclient: shutdown 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e413fee0 msgr2=0x7f24e4135f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e413fee0 0x7f24e4135f80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009a00 tx=0x7f24d0005840 comp rx=0 tx=0).stop 2024-09-11T03:08:07.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 shutdown_connections 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f24e413dc50 0x7f24e413e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f24c803df70 0x7f24c8040430 unknown :-1 s=CLOSED pgs=3717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f24e413fee0 0x7f24e4135f80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 --2- 172.21.15.98:0/2177577574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24d0004da0 0x7f24e4135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 >> 172.21.15.98:0/2177577574 conn(0x7f24e40ac730 msgr2=0x7f24e40a1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 shutdown_connections 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 -- 172.21.15.98:0/2177577574 wait complete. 2024-09-11T03:08:07.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.763+0000 7f24eb571640 1 librados: shutdown 2024-09-11T03:08:07.779 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T03:08:07.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T03:08:07.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.903+0000 7f4f634e0640 10 monclient: get_monmap_and_config 2024-09-11T03:08:07.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.903+0000 7f4f634e0640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.903+0000 7f4f634e0640 10 monclient: monmap: 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:07.908430+0000 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:07.908430+0000 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.906 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.907 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.907 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.907 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.907 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:07.907 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:07.907 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.903+0000 7f4f634e0640 1 Processor -- start 2024-09-11T03:08:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 1 -- start start 2024-09-11T03:08:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient: init 2024-09-11T03:08:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient: _add_conns ranks=[5,3,0,1,4,10,7,8,6,2,9] 2024-09-11T03:08:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f5c0725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient(hunting): picked mon.noname-h con 0x7f4f5c139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c06fe30 0x7f4f5c139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient(hunting): picked mon.noname-g con 0x7f4f5c06fe30 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f5c0a7a10 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f5c0a9650 con 0x7f4f5c0a7a10 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5c071350 con 0x7f4f5c06fe30 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f4f5c071630 con 0x7f4f5c139b10 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 10 monclient(hunting): authenticate will time out at 2041.788705s 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4f5c0a7a10 0x7f4f5c0a7ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48198/0 (socket says 172.21.15.98:48198) 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 1 -- 172.21.15.98:0/4290339252 learned_addr learned my addr 172.21.15.98:0/4290339252 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:07.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f5c0725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c06fe30 0x7f4f5c139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571195535 0 0) 0x7f4f5c0a9650 con 0x7f4f5c0a7a10 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): _negotiate 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): my global_id is 104274 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f48003530 con 0x7f4f5c0a7a10 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): get_auth_request con 0x7f4f5c06fe30 auth_method 0 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): get_auth_request con 0x7f4f5c139b10 auth_method 0 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1755070556 0 0) 0x7f4f48003530 con 0x7f4f5c0a7a10 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 msgr2=0x7f4f5c0725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f5c0725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c06fe30 msgr2=0x7f4f5c139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c06fe30 0x7f4f5c139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:08:07.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5c0702b0 con 0x7f4f5c0a7a10 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f4f54001cf0 con 0x7f4f5c0a7a10 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f5c0725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: dump: 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.924 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.925 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 >> v1:172.21.15.98:6789/0 conn(0x7f4f5c0a7a10 legacy=0x7f4f5c0a7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient: _add_conns ranks=[3,8,5,2,7,4,0,1,6] 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f48004210 0x7f4f48004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): picked mon.g con 0x7f4f48004210 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4f48004d50 0x7f4f480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): picked mon.e con 0x7f4f48004d50 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f4800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): picked mon.h con 0x7f4f5c139b10 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5c071350 con 0x7f4f48004210 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f4f5c071630 con 0x7f4f5c139b10 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f4f4800c1a0 con 0x7f4f48004d50 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f53fff640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4f48004d50 0x7f4f480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): get_auth_request con 0x7f4f48004d50 auth_method 0 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f4800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f48004210 0x7f4f48004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 10 monclient(hunting): get_auth_request con 0x7f4f5c139b10 auth_method 0 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): get_auth_request con 0x7f4f48004210 auth_method 0 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f61a7d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f6127c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_done global_id 104274 payload 293 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 -- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 msgr2=0x7f4f4800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f4800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 -- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f48004210 msgr2=0x7f4f48004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f48004210 0x7f4f48004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 monclient: found mon.e 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 1 -- 172.21.15.98:0/4290339252 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f48003530 con 0x7f4f48004d50 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f60a7b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 5 monclient: authenticate success, global_id 104274 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.907+0000 7f4f634e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4f48004d50 0x7f4f480051d0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0a99e0 tx=0x7f4f4c01c990 comp rx=0 tx=0).ready entity=mon.8 client_cookie=bfc28907fa5d8633 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f4c0096b0 con 0x7f4f48004d50 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 10 monclient: dump: 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.930 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f4c0a68f0 con 0x7f4f48004d50 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:07.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 1 -- 172.21.15.98:0/4290339252 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f4c0a7460 con 0x7f4f48004d50 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 4 set_mon_vals no callback set 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f53fff640 10 monclient: dump: 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.932 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: shutdown 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- 172.21.15.98:0/4290339252 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4f48004d50 msgr2=0x7f4f480051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4f48004d50 0x7f4f480051d0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0a99e0 tx=0x7f4f4c01c990 comp rx=0 tx=0).stop 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- 172.21.15.98:0/4290339252 shutdown_connections 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f4f5c139b10 0x7f4f4800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f48004210 0x7f4f48004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 --2- 172.21.15.98:0/4290339252 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f4f48004d50 0x7f4f480051d0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- 172.21.15.98:0/4290339252 >> 172.21.15.98:0/4290339252 conn(0x7f4f5c0ac770 msgr2=0x7f4f5c0a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- 172.21.15.98:0/4290339252 shutdown_connections 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- 172.21.15.98:0/4290339252 wait complete. 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: build_initial_monmap 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: monmap: 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:07.934 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 librados: starting msgr at 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 librados: starting objecter 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 Processor -- start 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- start start 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 librados: setting wanted keys 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 librados: calling monclient init 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: init 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: _add_conns ranks=[7,4,3,0,2,6,5,1,8] 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4f48004210 0x7f4f5c1d1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): picked mon.noname-i con 0x7f4f48004210 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4f48004d50 0x7f4f5c1d18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): picked mon.noname-c con 0x7f4f48004d50 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c0a7a10 0x7f4f5c1251c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): picked mon.noname-g con 0x7f4f5c0a7a10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f4f5c13cd80 con 0x7f4f5c0a7a10 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f4f5c13cc00 con 0x7f4f48004d50 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f4f5c13cf00 con 0x7f4f48004210 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient(hunting): authenticate will time out at 2041.794972s 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4f48004d50 0x7f4f5c1d18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4f48004d50 0x7f4f5c1d18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53776/0 (socket says 172.21.15.98:53776) 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 -- 172.21.15.98:0/1168704842 learned_addr learned my addr 172.21.15.98:0/1168704842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): get_auth_request con 0x7f4f48004d50 auth_method 0 2024-09-11T03:08:07.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f61a7d640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c0a7a10 0x7f4f5c1251c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f6127c640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4f48004210 0x7f4f5c1d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient(hunting): handle_auth_done global_id 24961 payload 824 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 -- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4f48004210 msgr2=0x7f4f5c1d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4f48004210 0x7f4f5c1d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 -- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c0a7a10 msgr2=0x7f4f5c1251c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c0a7a10 0x7f4f5c1251c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 monclient: found mon.noname-c 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 -- 172.21.15.98:0/1168704842 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f5c138070 con 0x7f4f48004d50 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f61a7d640 10 monclient: get_auth_request con 0x7f4f5c0a7a10 auth_method 0 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f61a7d640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f4f5c0a7a10 0x7f4f5c1251c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f6127c640 10 monclient: get_auth_request con 0x7f4f48004210 auth_method 0 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f6127c640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f4f48004210 0x7f4f5c1d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f60a7b640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4f48004d50 0x7f4f5c1d18c0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c01c9d0 tx=0x7f4f4c019790 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 client.?.objecter ms_handle_connect 0x7f4f48004d50 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f4c0a78d0 con 0x7f4f48004d50 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 monclient: dump: 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.940 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 monclient: _finish_auth 0 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f4c0a7a90 con 0x7f4f48004d50 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 5 monclient: authenticate success, global_id 24961 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: _renew_subs 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f5c126c60 con 0x7f4f48004d50 2024-09-11T03:08:07.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f4c0a7c40 con 0x7f4f48004d50 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 10 client.24961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 monclient: dump: 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:07.942 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.911+0000 7f4f537fe640 4 set_mon_vals no callback set 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 10 monclient: _renew_subs 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f5c128a00 con 0x7f4f48004d50 2024-09-11T03:08:07.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 1 librados: init done 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 10 client.24961.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 15 client.24961.objecter _session_command_op_assign -1 1 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 20 client.24961.objecter _get_session osd=-1 returning homeless 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 10 client.24961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f5c06fe30 con 0x7f4f48004d50 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f4f4c0c2030 con 0x7f4f48004d50 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4f4003df40 0x7f4f40040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f4f4c00f040 con 0x7f4f48004d50 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 client.24961.objecter ms_dispatch 0x7f4f5c000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 3 client.24961.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 3 client.24961.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 client.24961.objecter checking command 1 2024-09-11T03:08:07.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f4f40045700 0x7f4f40047ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 20 client.24961.objecter _get_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 20 client.24961.objecter put_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 15 client.24961.objecter _session_command_op_remove -1 1 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 20 client.24961.objecter _get_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 20 client.24961.objecter get_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 15 client.24961.objecter _session_command_op_assign 2 1 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 20 client.24961.objecter put_session s=0x7f4f40045030 osd=2 4 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 client.24961.objecter _send_command 1 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 --> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f4c0b1ec0 con 0x7f4f40045700 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 20 client.24961.objecter dump_active .. 0 homeless 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_get_version_reply(handle=1 version=699) ==== 24+0+0 (secure 0 0 0) 0x7f4f4c0f0050 con 0x7f4f48004d50 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-11T03:08:07.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f61a7d640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f4f40045700 0x7f4f40047ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f61a7d640 10 monclient: get_auth_request con 0x7f4f40045700 auth_method 0 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f6127c640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4f4003df40 0x7f4f40040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f6127c640 10 monclient: get_auth_request con 0x7f4f4003df40 auth_method 0 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 client.24961.objecter ms_handle_connect 0x7f4f40045700 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f61a7d640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f4f40045700 0x7f4f40047ba0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f6127c640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4f4003df40 0x7f4f40040400 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7f4f580096b0 tx=0x7f4f58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 client.24961.objecter ms_handle_connect 0x7f4f4003df40 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== osd.2 v2:172.21.15.98:6816/3987433753 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f4c0b1ec0 con 0x7f4f40045700 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 client.24961.objecter ms_dispatch 0x7f4f5c000ba0 command_reply(tid 1: 0 ) 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 10 client.24961.objecter _finish_command 1 = system:0 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 20 client.24961.objecter put_session s=0x7f4f40045030 osd=2 4 2024-09-11T03:08:07.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.915+0000 7f4f51ffb640 15 client.24961.objecter _session_command_op_remove 2 1 2024-09-11T03:08:07.947 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:07.947 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T03:08:07.947 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:07.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 10 client.24961.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T03:08:07.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 15 client.24961.objecter _session_command_op_assign -1 2 2024-09-11T03:08:07.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 client.24961.objecter _get_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 client.24961.objecter put_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 client.24961.objecter _get_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 15 client.24961.objecter _session_command_op_remove -1 2 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 client.24961.objecter get_session s=0x7f4f40045030 osd=2 3 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 15 client.24961.objecter _session_command_op_assign 2 2 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 client.24961.objecter put_session s=0x7f4f40045030 osd=2 4 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 10 client.24961.objecter _send_command 2 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 --> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4f5c1d1e00 con 0x7f4f40045700 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f51ffb640 1 -- 172.21.15.98:0/1168704842 <== osd.2 v2:172.21.15.98:6816/3987433753 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4f5c1d1e00 con 0x7f4f40045700 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f51ffb640 10 client.24961.objecter ms_dispatch 0x7f4f5c000ba0 command_reply(tid 2: 0 ) 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f51ffb640 10 client.24961.objecter _finish_command 2 = system:0 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f51ffb640 20 client.24961.objecter put_session s=0x7f4f40045030 osd=2 4 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f51ffb640 15 client.24961.objecter _session_command_op_remove 2 2 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 10 librados: watch_flush enter 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 10 librados: watch_flush exit 2024-09-11T03:08:07.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 10 client.24961.objecter close_session for osd.2 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f4f40045700 msgr2=0x7f4f40047ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:6816/3987433753,v1:172.21.15.98:6817/3987433753] conn(0x7f4f40045700 0x7f4f40047ba0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 client.24961.objecter put_session s=0x7f4f40045030 osd=2 1 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 client.24961.objecter shutdown clearing up homeless session... 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 10 client.24961.objecter successfully canceled tick 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4f4003df40 msgr2=0x7f4f40040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f4f4003df40 0x7f4f40040400 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7f4f580096b0 tx=0x7f4f58008040 comp rx=0 tx=0).stop 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 10 monclient: shutdown 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4f48004d50 msgr2=0x7f4f5c1d18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f634e0640 1 --2- 172.21.15.98:0/1168704842 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f4f48004d50 0x7f4f5c1d18c0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c01c9d0 tx=0x7f4f4c019790 comp rx=0 tx=0).stop 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.931+0000 7f4f61a7d640 1 -- 172.21.15.98:0/1168704842 reap_dead start 2024-09-11T03:08:07.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.935+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 shutdown_connections 2024-09-11T03:08:07.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.935+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 >> 172.21.15.98:0/1168704842 conn(0x7f4f5c0ac770 msgr2=0x7f4f5c072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:07.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.935+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 shutdown_connections 2024-09-11T03:08:07.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.935+0000 7f4f634e0640 1 -- 172.21.15.98:0/1168704842 wait complete. 2024-09-11T03:08:07.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:07.935+0000 7f4f634e0640 1 librados: shutdown 2024-09-11T03:08:07.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient: get_monmap_and_config 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient: monmap: 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:08.085700+0000 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:08.085700+0000 2024-09-11T03:08:08.083 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 1 Processor -- start 2024-09-11T03:08:08.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 1 -- start start 2024-09-11T03:08:08.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient: init 2024-09-11T03:08:08.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient: _add_conns ranks=[9,8,0,3,2,4,6,5,10,7,1] 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fc6b00a5830 0x7fc6b00a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient(hunting): picked mon.noname-a con 0x7fc6b00a5830 addr v2:172.21.15.98:3300/0 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6b00a1770 0x7fc6b00a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient(hunting): picked mon.noname-e con 0x7fc6b00a1770 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6b0071ed0 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b0071c90 con 0x7fc6b0071ed0 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b00a6690 con 0x7fc6b00a5830 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fc6b00739b0 con 0x7fc6b00a1770 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b62f2640 10 monclient(hunting): authenticate will time out at 2041.965758s 2024-09-11T03:08:08.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b488f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc6b0071ed0 0x7fc6b01375e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48210/0 (socket says 172.21.15.98:48210) 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6b488f640 1 -- 172.21.15.98:0/1105440409 learned_addr learned my addr 172.21.15.98:0/1105440409 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 1 --2- 172.21.15.98:0/1105440409 >> v2:172.21.15.98:3300/0 conn(0x7fc6b00a5830 0x7fc6b00a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6af7fe640 1 --2- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6b00a1770 0x7fc6b00a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): get_auth_request con 0x7fc6b00a5830 auth_method 0 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 1 -- 172.21.15.98:0/1105440409 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057753698 0 0) 0x7fc6b0071c90 con 0x7fc6b0071ed0 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 10 monclient(hunting): _negotiate 2024-09-11T03:08:08.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 10 monclient(hunting): my global_id is 104283 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 1 -- 172.21.15.98:0/1105440409 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc698003530 con 0x7fc6b0071ed0 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient(hunting): handle_auth_done global_id 104292 payload 274 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 1 -- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6b00a1770 msgr2=0x7fc6b00a6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 1 --2- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6b00a1770 0x7fc6b00a6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 1 -- 172.21.15.98:0/1105440409 >> v1:172.21.15.98:6789/0 conn(0x7fc6b0071ed0 legacy=0x7fc6b01375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 1 -- 172.21.15.98:0/1105440409 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 708339228 0 0) 0x7fc698003530 con 0x7fc6b0071ed0 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 1 monclient: found mon.noname-a 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6affff640 1 -- 172.21.15.98:0/1105440409 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b0073c90 con 0x7fc6b00a5830 2024-09-11T03:08:08.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.083+0000 7fc6aeffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6affff640 1 --2- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b00a5830 0x7fc6b00a5c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00097f0 tx=0x7fc6a001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd627136be6a55e server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 1 -- 172.21.15.98:0/1105440409 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6a00b0070 con 0x7fc6b00a5830 2024-09-11T03:08:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: dump: 2024-09-11T03:08:08.091 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.092 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 1 -- 172.21.15.98:0/1105440409 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a00aa810 con 0x7fc6b00a5830 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 5 monclient: authenticate success, global_id 104292 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 1 -- 172.21.15.98:0/1105440409 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6a8001cf0 con 0x7fc6b00a5830 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 4 set_mon_vals no callback set 2024-09-11T03:08:08.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6aeffd640 10 monclient: dump: 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 10 monclient: shutdown 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 1 -- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b00a5830 msgr2=0x7fc6b00a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b00a5830 0x7fc6b00a5c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00097f0 tx=0x7fc6a001cd00 comp rx=0 tx=0).stop 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 1 -- 172.21.15.98:0/1105440409 shutdown_connections 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6b00a1770 0x7fc6b00a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6b00a5830 0x7fc6b00a5c10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.087+0000 7fc6b62f2640 1 -- 172.21.15.98:0/1105440409 >> 172.21.15.98:0/1105440409 conn(0x7fc6b00aa420 msgr2=0x7fc6b00a6920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient: get_auth_request con 0x7fc6b00a1770 auth_method 0 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 1 --2- 172.21.15.98:0/1105440409 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fc6b00a1770 0x7fc6b00a6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 -- 172.21.15.98:0/1105440409 shutdown_connections 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 -- 172.21.15.98:0/1105440409 wait complete. 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient: monmap: 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 librados: starting msgr at 2024-09-11T03:08:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 librados: starting objecter 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 Processor -- start 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 -- start start 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 librados: setting wanted keys 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 librados: calling monclient init 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient: init 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient: _add_conns ranks=[1,2,6,7,0,5,8,4,3] 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6b0071ed0 0x7fc6b0135ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): picked mon.noname-f con 0x7fc6b0071ed0 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6b00a5830 0x7fc6b0136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): picked mon.noname-b con 0x7fc6b00a5830 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6b013b6b0 0x7fc6b013bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): picked mon.noname-d con 0x7fc6b013b6b0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6b013b6b0 0x7fc6b013bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6b013b6b0 0x7fc6b013bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:42944/0 (socket says 172.21.15.98:42944) 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 -- 172.21.15.98:0/162724449 learned_addr learned my addr 172.21.15.98:0/162724449 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6b00a5830 0x7fc6b0136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0141df0 con 0x7fc6b0071ed0 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b0141c70 con 0x7fc6b00a5830 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7fc6b0141f70 con 0x7fc6b013b6b0 2024-09-11T03:08:08.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 10 monclient(hunting): authenticate will time out at 2041.974930s 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): get_auth_request con 0x7fc6b013b6b0 auth_method 0 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient(hunting): get_auth_request con 0x7fc6b00a5830 auth_method 0 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6affff640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6b0071ed0 0x7fc6b0135ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6affff640 10 monclient(hunting): get_auth_request con 0x7fc6b0071ed0 auth_method 0 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6affff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient(hunting): handle_auth_done global_id 84777 payload 824 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 -- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6b00a5830 msgr2=0x7fc6b0136410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6b00a5830 0x7fc6b0136410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 -- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6b0071ed0 msgr2=0x7fc6b0135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6b0071ed0 0x7fc6b0135ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 monclient: found mon.noname-d 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.98:3303/0 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b0120c10 con 0x7fc6b013b6b0 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6af7fe640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6b00a5830 0x7fc6b0136410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b488f640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6b013b6b0 0x7fc6b013bab0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80015b0 tx=0x7fc6a800a430 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 10 client.?.objecter ms_handle_connect 0x7fc6b013b6b0 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 1 -- 172.21.15.98:0/162724449 <== mon.6 v2:172.21.15.98:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6a8001cd0 con 0x7fc6b013b6b0 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 10 monclient: dump: 2024-09-11T03:08:08.105 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.106 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.091+0000 7fc6b62f2640 5 monclient: authenticate success, global_id 84777 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 monclient: _renew_subs 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b0120ef0 con 0x7fc6b013b6b0 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 1 -- 172.21.15.98:0/162724449 <== mon.6 v2:172.21.15.98:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a8015070 con 0x7fc6b013b6b0 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 1 -- 172.21.15.98:0/162724449 <== mon.6 v2:172.21.15.98:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6a800ad20 con 0x7fc6b013b6b0 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6ae7fc640 4 set_mon_vals no callback set 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 10 monclient: dump: 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.108 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 client.84777.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 monclient: _renew_subs 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b00a99f0 con 0x7fc6b013b6b0 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 1 librados: init done 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 1 -- 172.21.15.98:0/162724449 <== mon.6 v2:172.21.15.98:3303/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fc6a8011560 con 0x7fc6b013b6b0 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc69c03de70 0x7fc69c040330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.98:3303/0 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 1 -- 172.21.15.98:0/162724449 <== mon.6 v2:172.21.15.98:3303/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fc6a8051820 con 0x7fc6b013b6b0 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 10 client.84777.objecter ms_dispatch 0x7fc6b0000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 3 client.84777.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 3 client.84777.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b01375e0 con 0x7fc6b013b6b0 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 20 client.84777.objecter dump_active .. 0 homeless 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6affff640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc69c03de70 0x7fc69c040330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6affff640 10 monclient: get_auth_request con 0x7fc69c03de70 auth_method 0 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6acff9640 10 client.84777.objecter ms_handle_connect 0x7fc69c03de70 2024-09-11T03:08:08.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.095+0000 7fc6affff640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc69c03de70 0x7fc69c040330 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00ad000 tx=0x7fc6a0004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.103+0000 7fc6acff9640 1 -- 172.21.15.98:0/162724449 <== mon.6 v2:172.21.15.98:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fc6a801a470 con 0x7fc6b013b6b0 2024-09-11T03:08:08.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.103+0000 7fc6acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.103+0000 7fc6acff9640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:08.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fc6b013eb00 con 0x7fc69c03de70 2024-09-11T03:08:08.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6acff9640 1 -- 172.21.15.98:0/162724449 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc6b013eb00 con 0x7fc69c03de70 2024-09-11T03:08:08.275 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T03:08:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 10 librados: watch_flush enter 2024-09-11T03:08:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 10 librados: watch_flush exit 2024-09-11T03:08:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 20 client.84777.objecter shutdown clearing up homeless session... 2024-09-11T03:08:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 10 client.84777.objecter successfully canceled tick 2024-09-11T03:08:08.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc69c03de70 msgr2=0x7fc69c040330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc69c03de70 0x7fc69c040330 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7fc6a00ad000 tx=0x7fc6a0004740 comp rx=0 tx=0).stop 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 10 monclient: shutdown 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6b013b6b0 msgr2=0x7fc6b013bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6b013b6b0 0x7fc6b013bab0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80015b0 tx=0x7fc6a800a430 comp rx=0 tx=0).stop 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 shutdown_connections 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7fc6b013b6b0 0x7fc6b013bab0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fc69c03de70 0x7fc69c040330 unknown :-1 s=CLOSED pgs=3719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7fc6b00a5830 0x7fc6b0136410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 --2- 172.21.15.98:0/162724449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6b0071ed0 0x7fc6b0135ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 >> 172.21.15.98:0/162724449 conn(0x7fc6b00aa420 msgr2=0x7fc6b00a30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 shutdown_connections 2024-09-11T03:08:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.275+0000 7fc6b62f2640 1 -- 172.21.15.98:0/162724449 wait complete. 2024-09-11T03:08:08.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.279+0000 7fc6b62f2640 1 librados: shutdown 2024-09-11T03:08:08.293 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T03:08:08.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T03:08:08.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient: get_monmap_and_config 2024-09-11T03:08:08.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient: monmap: 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:08.409452+0000 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:08.409452+0000 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.407 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:08.408 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:08.408 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 1 Processor -- start 2024-09-11T03:08:08.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 1 -- start start 2024-09-11T03:08:08.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient: init 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient: _add_conns ranks=[3,7,1,9,2,0,10,5,6,8,4] 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc90072690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient(hunting): picked mon.noname-g con 0x7fbc90139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbc90139f60 0x7fbc900a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient(hunting): picked mon.noname-i con 0x7fbc90139f60 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbc9013a380 addr v1:172.21.15.190:6789/0 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 1 -- --> v1:172.21.15.190:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc90070320 con 0x7fbc9013a380 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbc900713c0 con 0x7fbc90139b80 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbc90072bd0 con 0x7fbc90139f60 2024-09-11T03:08:08.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc94b8f640 10 monclient(hunting): authenticate will time out at 2042.289689s 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbc90139f60 0x7fbc900a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc90072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbc90139f60 0x7fbc900a7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3303/0 says I am v2:172.21.15.98:34714/0 (socket says 172.21.15.98:34714) 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8ed9d640 1 --1- >> v1:172.21.15.190:6789/0 conn(0x7fbc9013a380 0x7fbc900a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.190:6789/0 says I am v1:172.21.15.98:38156/0 (socket says 172.21.15.98:38156) 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/1008278930 learned_addr learned my addr 172.21.15.98:0/1008278930 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): get_auth_request con 0x7fbc90139f60 auth_method 0 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request con 0x7fbc90139b80 auth_method 0 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8d59a640 1 -- 172.21.15.98:0/1008278930 <== mon.1 v1:172.21.15.190:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4083933706 0 0) 0x7fbc90070320 con 0x7fbc9013a380 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8d59a640 10 monclient(hunting): _negotiate 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8d59a640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8d59a640 10 monclient(hunting): my global_id is 26407 2024-09-11T03:08:08.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8d59a640 1 -- 172.21.15.98:0/1008278930 --> v1:172.21.15.190:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc78003530 con 0x7fbc9013a380 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_done global_id 104200 payload 274 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 msgr2=0x7fbc90072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 --2- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc90072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/1008278930 >> v1:172.21.15.190:6789/0 conn(0x7fbc9013a380 legacy=0x7fbc900a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 monclient: found mon.noname-i 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.190:3303/0 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/1008278930 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9006fff0 con 0x7fbc90139f60 2024-09-11T03:08:08.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.407+0000 7fbc8e59c640 1 --2- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc90072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 --2- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbc90139f60 0x7fbc900a7a10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009e80 tx=0x7fbc8001c910 comp rx=0 tx=0).ready entity=mon.7 client_cookie=ba68c0adfce388b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 1 -- 172.21.15.98:0/1008278930 <== mon.7 v2:172.21.15.190:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc800090d0 con 0x7fbc90139f60 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: dump: 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.421 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 1 -- 172.21.15.98:0/1008278930 <== mon.7 v2:172.21.15.190:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc800a5a00 con 0x7fbc90139f60 2024-09-11T03:08:08.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 5 monclient: authenticate success, global_id 104200 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 4 set_mon_vals no callback set 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 1 -- 172.21.15.98:0/1008278930 <== mon.7 v2:172.21.15.190:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc800a5bb0 con 0x7fbc90139f60 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8d59a640 10 monclient: dump: 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.423 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient: shutdown 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbc90139f60 msgr2=0x7fbc900a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 --2- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbc90139f60 0x7fbc900a7a10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009e80 tx=0x7fbc8001c910 comp rx=0 tx=0).stop 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- 172.21.15.98:0/1008278930 shutdown_connections 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 --2- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbc90139f60 0x7fbc900a7a10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 --2- 172.21.15.98:0/1008278930 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc90072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- 172.21.15.98:0/1008278930 >> 172.21.15.98:0/1008278930 conn(0x7fbc900ac7f0 msgr2=0x7fbc900a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- 172.21.15.98:0/1008278930 shutdown_connections 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- 172.21.15.98:0/1008278930 wait complete. 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient: monmap: 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:08.425 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 librados: starting msgr at 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 librados: starting objecter 2024-09-11T03:08:08.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 Processor -- start 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- start start 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 librados: setting wanted keys 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 librados: calling monclient init 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient: init 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient: _add_conns ranks=[3,8,5,4,0,6,1,2,7] 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc900a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): picked mon.noname-g con 0x7fbc90139b80 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc90139f60 0x7fbc900a2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): picked mon.noname-e con 0x7fbc90139f60 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbc9013a380 0x7fbc900a2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): picked mon.noname-h con 0x7fbc9013a380 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:08.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fbc901444e0 con 0x7fbc90139b80 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbc900a8870 con 0x7fbc9013a380 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fbc90144660 con 0x7fbc90139f60 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc94b8f640 10 monclient(hunting): authenticate will time out at 2042.295385s 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc90139f60 0x7fbc900a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc90139f60 0x7fbc900a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35258/0 (socket says 172.21.15.98:35258) 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/2548287525 learned_addr learned my addr 172.21.15.98:0/2548287525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): get_auth_request con 0x7fbc90139f60 auth_method 0 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8e59c640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc900a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8ed9d640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbc9013a380 0x7fbc900a2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request con 0x7fbc90139b80 auth_method 0 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8ed9d640 10 monclient(hunting): get_auth_request con 0x7fbc9013a380 auth_method 0 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient(hunting): handle_auth_done global_id 25559 payload 824 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbc9013a380 msgr2=0x7fbc900a2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbc9013a380 0x7fbc900a2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 msgr2=0x7fbc900a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fbc90139b80 0x7fbc900a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 monclient: found mon.noname-e 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.411+0000 7fbc8dd9b640 1 -- 172.21.15.98:0/2548287525 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9013b1b0 con 0x7fbc90139f60 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc8dd9b640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc90139f60 0x7fbc900a2860 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009e20 tx=0x7fbc800196b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 client.?.objecter ms_handle_connect 0x7fbc90139f60 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc800bc020 con 0x7fbc90139f60 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: dump: 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.431 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc80003e10 con 0x7fbc90139f60 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc80009210 con 0x7fbc90139f60 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: dump: 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.433 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7ffff640 4 set_mon_vals no callback set 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 5 monclient: authenticate success, global_id 25559 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 monclient: _renew_subs 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9013ad90 con 0x7fbc90139f60 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 client.25559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 monclient: _renew_subs 2024-09-11T03:08:08.434 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc900ab830 con 0x7fbc90139f60 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 1 librados: init done 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 client.25559.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 15 client.25559.objecter _session_command_op_assign -1 1 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 20 client.25559.objecter _get_session osd=-1 returning homeless 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 client.25559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbc900a5240 con 0x7fbc90139f60 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbc800039a0 con 0x7fbc90139f60 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc7003df70 0x7fbc70040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fbc800ea590 con 0x7fbc90139f60 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 client.25559.objecter ms_dispatch 0x7fbc90000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 3 client.25559.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 3 client.25559.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:08.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 client.25559.objecter checking command 1 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7fbc70045730 0x7fbc70047bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 20 client.25559.objecter _get_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 20 client.25559.objecter put_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 15 client.25559.objecter _session_command_op_remove -1 1 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 20 client.25559.objecter _get_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 20 client.25559.objecter get_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 15 client.25559.objecter _session_command_op_assign 3 1 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 20 client.25559.objecter put_session s=0x7fbc70045060 osd=3 4 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 client.25559.objecter _send_command 1 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 --> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc8000ae60 con 0x7fbc70045730 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 20 client.25559.objecter dump_active .. 0 homeless 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_get_version_reply(handle=1 version=699) ==== 24+0+0 (secure 0 0 0) 0x7fbc800f1050 con 0x7fbc90139f60 2024-09-11T03:08:08.436 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc8e59c640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc7003df70 0x7fbc70040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc8ed9d640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7fbc70045730 0x7fbc70047bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc8e59c640 10 monclient: get_auth_request con 0x7fbc7003df70 auth_method 0 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc8ed9d640 10 monclient: get_auth_request con 0x7fbc70045730 auth_method 0 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 client.25559.objecter ms_handle_connect 0x7fbc7003df70 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc8e59c640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc7003df70 0x7fbc70040430 secure :-1 s=READY pgs=3720 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009580 tx=0x7fbc84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc7effd640 10 client.25559.objecter ms_handle_connect 0x7fbc70045730 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.415+0000 7fbc8ed9d640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7fbc70045730 0x7fbc70047bd0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.419+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== osd.3 v2:172.21.15.190:6800/373809828 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbc8000ae60 con 0x7fbc70045730 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.419+0000 7fbc7effd640 10 client.25559.objecter ms_dispatch 0x7fbc90000ba0 command_reply(tid 1: 0 ) 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.419+0000 7fbc7effd640 10 client.25559.objecter _finish_command 1 = system:0 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.419+0000 7fbc7effd640 20 client.25559.objecter put_session s=0x7fbc70045060 osd=3 4 2024-09-11T03:08:08.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.419+0000 7fbc7effd640 15 client.25559.objecter _session_command_op_remove 3 1 2024-09-11T03:08:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 10 client.25559.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T03:08:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 15 client.25559.objecter _session_command_op_assign -1 2 2024-09-11T03:08:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 client.25559.objecter _get_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 client.25559.objecter put_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 client.25559.objecter _get_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 15 client.25559.objecter _session_command_op_remove -1 2 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 client.25559.objecter get_session s=0x7fbc70045060 osd=3 3 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 15 client.25559.objecter _session_command_op_assign 3 2 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 client.25559.objecter put_session s=0x7fbc70045060 osd=3 4 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 10 client.25559.objecter _send_command 2 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 --> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbc90069670 con 0x7fbc70045730 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc7effd640 1 -- 172.21.15.98:0/2548287525 <== osd.3 v2:172.21.15.190:6800/373809828 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbc90069670 con 0x7fbc70045730 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc7effd640 10 client.25559.objecter ms_dispatch 0x7fbc90000ba0 command_reply(tid 2: 0 ) 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc7effd640 10 client.25559.objecter _finish_command 2 = system:0 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc7effd640 20 client.25559.objecter put_session s=0x7fbc70045060 osd=3 4 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc7effd640 15 client.25559.objecter _session_command_op_remove 3 2 2024-09-11T03:08:08.439 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:08.440 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T03:08:08.440 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 10 librados: watch_flush enter 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 10 librados: watch_flush exit 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 10 client.25559.objecter close_session for osd.3 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7fbc70045730 msgr2=0x7fbc70047bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6800/373809828,v1:172.21.15.190:6801/373809828] conn(0x7fbc70045730 0x7fbc70047bd0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 client.25559.objecter put_session s=0x7fbc70045060 osd=3 1 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 client.25559.objecter shutdown clearing up homeless session... 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 10 client.25559.objecter successfully canceled tick 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc7003df70 msgr2=0x7fbc70040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbc7003df70 0x7fbc70040430 secure :-1 s=READY pgs=3720 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009580 tx=0x7fbc84008040 comp rx=0 tx=0).stop 2024-09-11T03:08:08.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 10 monclient: shutdown 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc90139f60 msgr2=0x7fbc900a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 --2- 172.21.15.98:0/2548287525 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fbc90139f60 0x7fbc900a2860 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009e20 tx=0x7fbc800196b0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc8ed9d640 1 -- 172.21.15.98:0/2548287525 reap_dead start 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 shutdown_connections 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 >> 172.21.15.98:0/2548287525 conn(0x7fbc900ac7f0 msgr2=0x7fbc900aa020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 shutdown_connections 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 -- 172.21.15.98:0/2548287525 wait complete. 2024-09-11T03:08:08.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.431+0000 7fbc94b8f640 1 librados: shutdown 2024-09-11T03:08:08.446 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient: get_monmap_and_config 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient: monmap: 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:08.557376+0000 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:08.557376+0000 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:08.555 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 1 Processor -- start 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 1 -- start start 2024-09-11T03:08:08.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient: init 2024-09-11T03:08:08.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient: _add_conns ranks=[0,7,10,8,5,3,9,4,1,2,6] 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84e409f4a0 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84e4073860 0x7f84e413c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient(hunting): picked mon.noname-i con 0x7f84e4073860 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f84e4137a00 0x7f84e413edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient(hunting): picked mon.noname-f con 0x7f84e4137a00 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e40a2d80 con 0x7f84e409f4a0 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f84e40a3000 con 0x7f84e4137a00 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f84e4076b00 con 0x7f84e4073860 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 10 monclient(hunting): authenticate will time out at 2042.437584s 2024-09-11T03:08:08.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f84e409f4a0 0x7f84e409f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48224/0 (socket says 172.21.15.98:48224) 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 -- 172.21.15.98:0/3049519229 learned_addr learned my addr 172.21.15.98:0/3049519229 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 1 --2- 172.21.15.98:0/3049519229 >> v2:172.21.15.190:3300/0 conn(0x7f84e4137a00 0x7f84e413edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): get_auth_request con 0x7f84e4137a00 auth_method 0 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84e4073860 0x7f84e413c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1161099797 0 0) 0x7f84e40a2d80 con 0x7f84e409f4a0 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): _negotiate 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): my global_id is 104301 2024-09-11T03:08:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84d0003530 con 0x7f84e409f4a0 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): get_auth_request con 0x7f84e4073860 auth_method 0 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 <== mon.0 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 561496634 0 0) 0x7f84d0003530 con 0x7f84e409f4a0 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84e4073860 msgr2=0x7f84e413c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84e4073860 0x7f84e413c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 >> v2:172.21.15.190:3300/0 conn(0x7f84e4137a00 msgr2=0x7f84e413edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 --2- 172.21.15.98:0/3049519229 >> v2:172.21.15.190:3300/0 conn(0x7f84e4137a00 0x7f84e413edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 monclient: found mon.noname-a-legacy 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.98:6789/0 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e4077880 con 0x7f84e409f4a0 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 <== mon.0 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 263039782 0 0) 0x7f84dc001cf0 con 0x7f84e409f4a0 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: dump: 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.570 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] but i'm connected to v1:172.21.15.98:6789/0, reconnecting 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 >> v1:172.21.15.98:6789/0 conn(0x7f84e409f4a0 legacy=0x7f84e409f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient: _add_conns ranks=[8,7,4,2,1,5,0,6,3] 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84d0004210 0x7f84d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): picked mon.e con 0x7f84d0004210 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84d0004d50 0x7f84d0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): picked mon.i con 0x7f84d0004d50 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:08.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84d00099c0 0x7f84d000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): picked mon.c con 0x7f84d00099c0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f84e4076b00 con 0x7f84d00099c0 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f84d000c590 con 0x7f84d0004d50 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f84d000c740 con 0x7f84d0004210 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84d0004210 0x7f84d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): get_auth_request con 0x7f84d0004210 auth_method 0 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84d00099c0 0x7f84d000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): get_auth_request con 0x7f84d00099c0 auth_method 0 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84d0004d50 0x7f84d0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): get_auth_request con 0x7f84d0004d50 auth_method 0 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient(hunting): handle_auth_done global_id 104301 payload 293 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 -- 172.21.15.98:0/3049519229 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84d0004d50 msgr2=0x7f84d0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f84d0004d50 0x7f84d0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 -- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84d00099c0 msgr2=0x7f84d000bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84d00099c0 0x7f84d000bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 monclient: found mon.e 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:08.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 1 -- 172.21.15.98:0/3049519229 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d0003530 con 0x7f84d0004210 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e37fe640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84d00099c0 0x7f84d000bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 5 monclient: authenticate success, global_id 104301 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.555+0000 7f84e95b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84d0004210 0x7f84d0004810 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00a6a0 tx=0x7f84dc01f860 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b0609b036b425b9f server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84dc00a2f0 con 0x7f84d0004210 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 10 monclient: dump: 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.575 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.576 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84dc0b2070 con 0x7f84d0004210 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 1 -- 172.21.15.98:0/3049519229 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84dc0add00 con 0x7f84d0004210 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 4 set_mon_vals no callback set 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e1ffb640 10 monclient: dump: 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.577 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.578 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: shutdown 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84d0004210 msgr2=0x7f84d0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 --2- 172.21.15.98:0/3049519229 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f84d0004210 0x7f84d0004810 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00a6a0 tx=0x7f84dc01f860 comp rx=0 tx=0).stop 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 1 -- 172.21.15.98:0/3049519229 reap_dead start 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- 172.21.15.98:0/3049519229 shutdown_connections 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- 172.21.15.98:0/3049519229 >> 172.21.15.98:0/3049519229 conn(0x7f84e406eed0 msgr2=0x7f84e406f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- 172.21.15.98:0/3049519229 shutdown_connections 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- 172.21.15.98:0/3049519229 wait complete. 2024-09-11T03:08:08.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: monmap: 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.580 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 librados: starting msgr at 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 librados: starting objecter 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 Processor -- start 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- start start 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 librados: setting wanted keys 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 librados: calling monclient init 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: init 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: _add_conns ranks=[1,2,4,8,7,0,5,3,6] 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84d0004210 0x7f84e41cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): picked mon.noname-f con 0x7f84d0004210 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e41cc4b0 0x7f84e411fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): picked mon.noname-b con 0x7f84e41cc4b0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84e41cc890 0x7f84e41202e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): picked mon.noname-c con 0x7f84e41cc890 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f84e413a9d0 con 0x7f84d0004210 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f84e413a850 con 0x7f84e41cc4b0 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f84e413ab50 con 0x7f84e41cc890 2024-09-11T03:08:08.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient(hunting): authenticate will time out at 2042.443151s 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e41cc4b0 0x7f84e411fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84e41cc890 0x7f84e41202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e41cc4b0 0x7f84e411fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:42164/0 (socket says 172.21.15.98:42164) 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 -- 172.21.15.98:0/647849325 learned_addr learned my addr 172.21.15.98:0/647849325 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): get_auth_request con 0x7f84e41cc4b0 auth_method 0 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 10 monclient(hunting): get_auth_request con 0x7f84e41cc890 auth_method 0 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84d0004210 0x7f84e41cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient(hunting): handle_auth_done global_id 94448 payload 824 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 -- 172.21.15.98:0/647849325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84e41cc890 msgr2=0x7f84e41202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84e41cc890 0x7f84e41202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 -- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84d0004210 msgr2=0x7f84e41cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84d0004210 0x7f84e41cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 monclient: found mon.noname-b 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 -- 172.21.15.98:0/647849325 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e4123820 con 0x7f84e41cc4b0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e2ffd640 10 monclient: get_auth_request con 0x7f84d0004210 auth_method 0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84d0004210 0x7f84e41cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e27fc640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e41cc4b0 0x7f84e411fda0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84cc011590 tx=0x7f84cc011990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 client.?.objecter ms_handle_connect 0x7f84e41cc4b0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 1 -- 172.21.15.98:0/647849325 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84cc019020 con 0x7f84e41cc4b0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: dump: 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.586 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 1 -- 172.21.15.98:0/647849325 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84cc004300 con 0x7f84e41cc4b0 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 5 monclient: authenticate success, global_id 94448 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: _renew_subs 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e4123b00 con 0x7f84e41cc4b0 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 1 -- 172.21.15.98:0/647849325 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84cc0044f0 con 0x7f84e41cc4b0 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84db7fe640 10 monclient: dump: 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e17fa640 4 set_mon_vals no callback set 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 client.94448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: _renew_subs 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.559+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e40a7100 con 0x7f84e41cc4b0 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e95b0640 1 librados: init done 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e95b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e95b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e95b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84e4073860 con 0x7f84e41cc4b0 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 1 -- 172.21.15.98:0/647849325 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f84cc003620 con 0x7f84e41cc4b0 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f84c403df70 0x7f84c4040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 1 -- 172.21.15.98:0/647849325 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f84cc04f7d0 con 0x7f84e41cc4b0 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 10 client.94448.objecter ms_dispatch 0x7f84e4000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 3 client.94448.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 3 client.94448.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 20 client.94448.objecter dump_active .. 0 homeless 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f84c403df70 0x7f84c4040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e2ffd640 10 monclient: get_auth_request con 0x7f84c403df70 auth_method 0 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84db7fe640 10 client.94448.objecter ms_handle_connect 0x7f84c403df70 2024-09-11T03:08:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.563+0000 7f84e2ffd640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f84c403df70 0x7f84c4040430 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f84dc002240 tx=0x7f84dc002060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.567+0000 7f84db7fe640 1 -- 172.21.15.98:0/647849325 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f84cc023070 con 0x7f84e41cc4b0 2024-09-11T03:08:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.567+0000 7f84db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.567+0000 7f84db7fe640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:08.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.727+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f84e4135b30 con 0x7f84c403df70 2024-09-11T03:08:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.727+0000 7f84db7fe640 1 -- 172.21.15.98:0/647849325 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f84e4135b30 con 0x7f84c403df70 2024-09-11T03:08:08.729 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T03:08:08.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 10 librados: watch_flush enter 2024-09-11T03:08:08.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 10 librados: watch_flush exit 2024-09-11T03:08:08.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 20 client.94448.objecter shutdown clearing up homeless session... 2024-09-11T03:08:08.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 10 client.94448.objecter successfully canceled tick 2024-09-11T03:08:08.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f84c403df70 msgr2=0x7f84c4040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f84c403df70 0x7f84c4040430 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f84dc002240 tx=0x7f84dc002060 comp rx=0 tx=0).stop 2024-09-11T03:08:08.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 10 monclient: shutdown 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e41cc4b0 msgr2=0x7f84e411fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e41cc4b0 0x7f84e411fda0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f84cc011590 tx=0x7f84cc011990 comp rx=0 tx=0).stop 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 shutdown_connections 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f84e41cc890 0x7f84e41202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f84c403df70 0x7f84c4040430 unknown :-1 s=CLOSED pgs=3721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f84e41cc4b0 0x7f84e411fda0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 --2- 172.21.15.98:0/647849325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84d0004210 0x7f84e41cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 >> 172.21.15.98:0/647849325 conn(0x7f84e406eed0 msgr2=0x7f84e41366f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 shutdown_connections 2024-09-11T03:08:08.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 -- 172.21.15.98:0/647849325 wait complete. 2024-09-11T03:08:08.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.731+0000 7f84e95b0640 1 librados: shutdown 2024-09-11T03:08:08.746 INFO:tasks.ceph:Scrubbing osd.4 2024-09-11T03:08:08.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-11T03:08:08.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient: get_monmap_and_config 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient: monmap: 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:08.861684+0000 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:08.861684+0000 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:08.859 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:08.860 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 1 Processor -- start 2024-09-11T03:08:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 1 -- start start 2024-09-11T03:08:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient: init 2024-09-11T03:08:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient: _add_conns ranks=[0,5,2,9,1,3,6,4,7,10,8] 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f635c0a7aa0 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1600 0x7f635c13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient(hunting): picked mon.noname-h con 0x7f635c0a1600 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c0a1dc0 0x7f635c1411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient(hunting): picked mon.noname-b con 0x7f635c0a1dc0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f635c0a1a80 con 0x7f635c0a7aa0 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f635c0a3170 con 0x7f635c0a1dc0 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f635c0a96b0 con 0x7f635c0a1600 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6364058640 10 monclient(hunting): authenticate will time out at 2042.742031s 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c0a1dc0 0x7f635c1411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c0a1dc0 0x7f635c1411a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:42174/0 (socket says 172.21.15.98:42174) 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 -- 172.21.15.98:0/3656546455 learned_addr learned my addr 172.21.15.98:0/3656546455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): get_auth_request con 0x7f635c0a1dc0 auth_method 0 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f6361df4640 1 --1- 172.21.15.98:0/3656546455 >> v1:172.21.15.98:6789/0 conn(0x7f635c0a7aa0 0x7f635c0a7e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48228/0 (socket says 172.21.15.98:48228) 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient(hunting): handle_auth_done global_id 94457 payload 274 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63615f3640 1 --2- 172.21.15.98:0/3656546455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1600 0x7f635c13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 -- 172.21.15.98:0/3656546455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1600 msgr2=0x7f635c13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 --2- 172.21.15.98:0/3656546455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1600 0x7f635c13ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 -- 172.21.15.98:0/3656546455 >> v1:172.21.15.98:6789/0 conn(0x7f635c0a7aa0 legacy=0x7f635c0a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 monclient: found mon.noname-b 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63625f5640 1 -- 172.21.15.98:0/3656546455 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635c0a3ea0 con 0x7f635c0a1dc0 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63615f3640 10 monclient: get_auth_request con 0x7f635c0a1600 auth_method 0 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.859+0000 7f63615f3640 1 --2- 172.21.15.98:0/3656546455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1600 0x7f635c13ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 --2- 172.21.15.98:0/3656546455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c0a1dc0 0x7f635c1411a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e80 tx=0x7f635401c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc6082bf76d12bfc server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 1 -- 172.21.15.98:0/3656546455 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63540090d0 con 0x7f635c0a1dc0 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: dump: 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.868 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 5 monclient: authenticate success, global_id 94457 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 1 -- 172.21.15.98:0/3656546455 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63540a5a00 con 0x7f635c0a1dc0 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 4 set_mon_vals no callback set 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 1 -- 172.21.15.98:0/3656546455 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63540a5bf0 con 0x7f635c0a1dc0 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6360df2640 10 monclient: dump: 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.870 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: shutdown 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- 172.21.15.98:0/3656546455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c0a1dc0 msgr2=0x7f635c1411a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 --2- 172.21.15.98:0/3656546455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c0a1dc0 0x7f635c1411a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6354009e80 tx=0x7f635401c910 comp rx=0 tx=0).stop 2024-09-11T03:08:08.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- 172.21.15.98:0/3656546455 shutdown_connections 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 --2- 172.21.15.98:0/3656546455 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c0a1dc0 0x7f635c1411a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 --2- 172.21.15.98:0/3656546455 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1600 0x7f635c13ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- 172.21.15.98:0/3656546455 >> 172.21.15.98:0/3656546455 conn(0x7f635c0ac820 msgr2=0x7f635c0a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- 172.21.15.98:0/3656546455 shutdown_connections 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- 172.21.15.98:0/3656546455 wait complete. 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: build_initial_monmap 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: monmap: 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:08.872 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 librados: starting msgr at 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 librados: starting objecter 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 Processor -- start 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- start start 2024-09-11T03:08:08.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 librados: setting wanted keys 2024-09-11T03:08:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 librados: calling monclient init 2024-09-11T03:08:08.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: init 2024-09-11T03:08:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:08.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: _add_conns ranks=[5,6,2,3,7,0,4,1,8] 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1dc0 0x7f635c1d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): picked mon.noname-h con 0x7f635c0a1dc0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f635c0a7aa0 0x7f635c1d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): picked mon.noname-d con 0x7f635c0a7aa0 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c13a8d0 0x7f635c13acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): picked mon.noname-b con 0x7f635c13a8d0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f635c144290 con 0x7f635c13a8d0 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f635c144110 con 0x7f635c0a1dc0 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:08.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7f635c144410 con 0x7f635c0a7aa0 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient(hunting): authenticate will time out at 2042.746884s 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f635c0a7aa0 0x7f635c1d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c13a8d0 0x7f635c13acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f635c0a7aa0 0x7f635c1d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:42956/0 (socket says 172.21.15.98:42956) 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 1 -- 172.21.15.98:0/80472316 learned_addr learned my addr 172.21.15.98:0/80472316 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c13a8d0 0x7f635c13acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:42184/0 (socket says 172.21.15.98:42184) 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 10 monclient(hunting): get_auth_request con 0x7f635c0a7aa0 auth_method 0 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): get_auth_request con 0x7f635c13a8d0 auth_method 0 2024-09-11T03:08:08.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient(hunting): handle_auth_done global_id 94466 payload 824 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6361df4640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1dc0 0x7f635c1d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 -- 172.21.15.98:0/80472316 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f635c0a7aa0 msgr2=0x7f635c1d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f635c0a7aa0 0x7f635c1d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 -- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1dc0 msgr2=0x7f635c1d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f635c0a1dc0 0x7f635c1d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient: _finish_hunting 0 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 monclient: found mon.noname-b 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 -- 172.21.15.98:0/80472316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635c13a4b0 con 0x7f635c13a8d0 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63615f3640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7f635c0a7aa0 0x7f635c1d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63625f5640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c13a8d0 0x7f635c13acf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63540ab000 tx=0x7f63540196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 client.?.objecter ms_handle_connect 0x7f635c13a8d0 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63540bc020 con 0x7f635c13a8d0 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: dump: 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.883 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 5 monclient: authenticate success, global_id 94466 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6354003c30 con 0x7f635c13a8d0 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: _renew_subs 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f635c13e5b0 con 0x7f635c13a8d0 2024-09-11T03:08:08.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6354003de0 con 0x7f635c13a8d0 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6353fff640 4 set_mon_vals no callback set 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 client.94466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f63527fc640 10 monclient: dump: 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:08.885 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: _renew_subs 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f635c13c250 con 0x7f635c13a8d0 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.863+0000 7f6364058640 1 librados: init done 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6364058640 10 client.94466.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6364058640 15 client.94466.objecter _session_command_op_assign -1 1 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6364058640 20 client.94466.objecter _get_session osd=-1 returning homeless 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6364058640 10 client.94466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6364058640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f635c139f60 con 0x7f635c13a8d0 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f6354004040 con 0x7f635c13a8d0 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f634003df70 0x7f6340040430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f63540ea8a0 con 0x7f635c13a8d0 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 10 client.94466.objecter ms_dispatch 0x7f635c000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:08.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 3 client.94466.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 3 client.94466.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 10 client.94466.objecter checking command 1 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f6340045730 0x7f6340047bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 20 client.94466.objecter _get_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 20 client.94466.objecter put_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 15 client.94466.objecter _session_command_op_remove -1 1 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 20 client.94466.objecter _get_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 20 client.94466.objecter get_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 15 client.94466.objecter _session_command_op_assign 4 1 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 20 client.94466.objecter put_session s=0x7f6340045060 osd=4 4 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 10 client.94466.objecter _send_command 1 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 --> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63540020c0 con 0x7f6340045730 2024-09-11T03:08:08.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 20 client.94466.objecter dump_active .. 0 homeless 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_get_version_reply(handle=1 version=699) ==== 24+0+0 (secure 0 0 0) 0x7f63540ef2c0 con 0x7f635c13a8d0 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6361df4640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f634003df70 0x7f6340040430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6361df4640 10 monclient: get_auth_request con 0x7f634003df70 auth_method 0 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63615f3640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f6340045730 0x7f6340047bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63615f3640 10 monclient: get_auth_request con 0x7f6340045730 auth_method 0 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 10 client.94466.objecter ms_handle_connect 0x7f634003df70 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f6361df4640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f634003df70 0x7f6340040430 secure :-1 s=READY pgs=3722 cs=0 l=1 rev1=1 crypto rx=0x7f634c000f70 tx=0x7f634c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63527fc640 10 client.94466.objecter ms_handle_connect 0x7f6340045730 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.867+0000 7f63615f3640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f6340045730 0x7f6340047bd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.871+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== osd.4 v2:172.21.15.190:6816/1271446291 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f63540020c0 con 0x7f6340045730 2024-09-11T03:08:08.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.871+0000 7f63527fc640 10 client.94466.objecter ms_dispatch 0x7f635c000ba0 command_reply(tid 1: 0 ) 2024-09-11T03:08:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.871+0000 7f63527fc640 10 client.94466.objecter _finish_command 1 = system:0 2024-09-11T03:08:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.871+0000 7f63527fc640 20 client.94466.objecter put_session s=0x7f6340045060 osd=4 4 2024-09-11T03:08:08.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.871+0000 7f63527fc640 15 client.94466.objecter _session_command_op_remove 4 1 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 10 client.94466.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 15 client.94466.objecter _session_command_op_assign -1 2 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 client.94466.objecter _get_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 client.94466.objecter put_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 client.94466.objecter _get_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 15 client.94466.objecter _session_command_op_remove -1 2 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 client.94466.objecter get_session s=0x7f6340045060 osd=4 3 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 15 client.94466.objecter _session_command_op_assign 4 2 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 client.94466.objecter put_session s=0x7f6340045060 osd=4 4 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 10 client.94466.objecter _send_command 2 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 --> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f635c140520 con 0x7f6340045730 2024-09-11T03:08:08.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f63527fc640 1 -- 172.21.15.98:0/80472316 <== osd.4 v2:172.21.15.190:6816/1271446291 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f63580059a0 con 0x7f6340045730 2024-09-11T03:08:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f63527fc640 10 client.94466.objecter ms_dispatch 0x7f635c000ba0 command_reply(tid 2: 0 ) 2024-09-11T03:08:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f63527fc640 10 client.94466.objecter _finish_command 2 = system:0 2024-09-11T03:08:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f63527fc640 20 client.94466.objecter put_session s=0x7f6340045060 osd=4 4 2024-09-11T03:08:08.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f63527fc640 15 client.94466.objecter _session_command_op_remove 4 2 2024-09-11T03:08:08.892 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:08.892 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T03:08:08.892 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 10 librados: watch_flush enter 2024-09-11T03:08:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 10 librados: watch_flush exit 2024-09-11T03:08:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 10 client.94466.objecter close_session for osd.4 2024-09-11T03:08:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f6340045730 msgr2=0x7f6340047bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6816/1271446291,v1:172.21.15.190:6817/1271446291] conn(0x7f6340045730 0x7f6340047bd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 client.94466.objecter put_session s=0x7f6340045060 osd=4 1 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 client.94466.objecter shutdown clearing up homeless session... 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 10 client.94466.objecter successfully canceled tick 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f634003df70 msgr2=0x7f6340040430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f634003df70 0x7f6340040430 secure :-1 s=READY pgs=3722 cs=0 l=1 rev1=1 crypto rx=0x7f634c000f70 tx=0x7f634c009040 comp rx=0 tx=0).stop 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 10 monclient: shutdown 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c13a8d0 msgr2=0x7f635c13acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 --2- 172.21.15.98:0/80472316 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f635c13a8d0 0x7f635c13acf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63540ab000 tx=0x7f63540196b0 comp rx=0 tx=0).stop 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f63625f5640 1 -- 172.21.15.98:0/80472316 reap_dead start 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 shutdown_connections 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 >> 172.21.15.98:0/80472316 conn(0x7f635c0ac820 msgr2=0x7f635c0a54c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 shutdown_connections 2024-09-11T03:08:08.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 -- 172.21.15.98:0/80472316 wait complete. 2024-09-11T03:08:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:08.883+0000 7f6364058640 1 librados: shutdown 2024-09-11T03:08:08.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-11T03:08:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.003+0000 7fbdecf75640 10 monclient: get_monmap_and_config 2024-09-11T03:08:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.003+0000 7fbdecf75640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.003+0000 7fbdecf75640 10 monclient: monmap: 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:09.008499+0000 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:09.008499+0000 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.006 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:09.007 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:09.007 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 1 Processor -- start 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 1 -- start start 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient: init 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient: _add_conns ranks=[10,4,0,7,3,5,2,1,6,8,9] 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7fbde8139b80 0x7fbde8139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient(hunting): picked mon.noname-f con 0x7fbde8139b80 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbde806fce0 0x7fbde80a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient(hunting): picked mon.noname-c con 0x7fbde806fce0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:09.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbde80724b0 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde80a94a0 con 0x7fbde80724b0 2024-09-11T03:08:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7fbde8071850 con 0x7fbde8139b80 2024-09-11T03:08:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7fbde8071aa0 con 0x7fbde806fce0 2024-09-11T03:08:09.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbdecf75640 10 monclient(hunting): authenticate will time out at 2042.889214s 2024-09-11T03:08:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde6d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbde80724b0 0x7fbde80a7f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48232/0 (socket says 172.21.15.98:48232) 2024-09-11T03:08:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde6d9d640 1 -- 172.21.15.98:0/3914015352 learned_addr learned my addr 172.21.15.98:0/3914015352 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 1 --2- 172.21.15.98:0/3914015352 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbde806fce0 0x7fbde80a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): get_auth_request con 0x7fbde806fce0 auth_method 0 2024-09-11T03:08:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde659c640 1 --2- 172.21.15.98:0/3914015352 >> v2:172.21.15.190:3300/0 conn(0x7fbde8139b80 0x7fbde8139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde659c640 10 monclient(hunting): get_auth_request con 0x7fbde8139b80 auth_method 0 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient(hunting): handle_auth_done global_id 24979 payload 274 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 1 -- 172.21.15.98:0/3914015352 >> v2:172.21.15.190:3300/0 conn(0x7fbde8139b80 msgr2=0x7fbde8139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 1 --2- 172.21.15.98:0/3914015352 >> v2:172.21.15.190:3300/0 conn(0x7fbde8139b80 0x7fbde8139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 1 -- 172.21.15.98:0/3914015352 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251774096 0 0) 0x7fbde80a94a0 con 0x7fbde80724b0 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 1 -- 172.21.15.98:0/3914015352 >> v1:172.21.15.98:6789/0 conn(0x7fbde80724b0 legacy=0x7fbde80a7f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 1 monclient: found mon.noname-c 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 1 -- 172.21.15.98:0/3914015352 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde813a4a0 con 0x7fbde806fce0 2024-09-11T03:08:09.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde5d9b640 1 --2- 172.21.15.98:0/3914015352 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbde806fce0 0x7fbde80a7a10 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0097c0 tx=0x7fbddc01c3f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=21570a632d7a4f1d server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 1 -- 172.21.15.98:0/3914015352 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbddc0b0070 con 0x7fbde806fce0 2024-09-11T03:08:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 10 monclient: dump: 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.019 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.007+0000 7fbde559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde559a640 1 -- 172.21.15.98:0/3914015352 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbddc0aa810 con 0x7fbde806fce0 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde559a640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde559a640 1 -- 172.21.15.98:0/3914015352 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbddc0a9300 con 0x7fbde806fce0 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 5 monclient: authenticate success, global_id 24979 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:09.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 4 set_mon_vals no callback set 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde559a640 10 monclient: dump: 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.021 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: shutdown 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/3914015352 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbde806fce0 msgr2=0x7fbde80a7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 --2- 172.21.15.98:0/3914015352 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbde806fce0 0x7fbde80a7a10 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0097c0 tx=0x7fbddc01c3f0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/3914015352 shutdown_connections 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 --2- 172.21.15.98:0/3914015352 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7fbde806fce0 0x7fbde80a7a10 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 --2- 172.21.15.98:0/3914015352 >> v2:172.21.15.190:3300/0 conn(0x7fbde8139b80 0x7fbde8139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/3914015352 >> 172.21.15.98:0/3914015352 conn(0x7fbde80ac6b0 msgr2=0x7fbde80a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/3914015352 shutdown_connections 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/3914015352 wait complete. 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: monmap: 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.023 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 librados: starting msgr at 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 librados: starting objecter 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 Processor -- start 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- start start 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 librados: setting wanted keys 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 librados: calling monclient init 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: init 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _add_conns ranks=[0,5,7,4,8,2,3,6,1] 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbde806fce0 0x7fbde81408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): picked mon.noname-a con 0x7fbde806fce0 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbde80724b0 0x7fbde81d1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): picked mon.noname-h con 0x7fbde80724b0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbde8139b80 0x7fbde813e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): picked mon.noname-i con 0x7fbde8139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbde80a8870 con 0x7fbde806fce0 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7fbde8070160 con 0x7fbde80724b0 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fbde8144450 con 0x7fbde8139b80 2024-09-11T03:08:09.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient(hunting): authenticate will time out at 2042.894004s 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbde806fce0 0x7fbde81408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbde806fce0 0x7fbde81408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44722/0 (socket says 172.21.15.98:44722) 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 -- 172.21.15.98:0/1841360772 learned_addr learned my addr 172.21.15.98:0/1841360772 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): get_auth_request con 0x7fbde806fce0 auth_method 0 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient(hunting): handle_auth_done global_id 104328 payload 824 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 -- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbde8139b80 msgr2=0x7fbde813e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbde8139b80 0x7fbde813e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 -- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbde80724b0 msgr2=0x7fbde81d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbde80724b0 0x7fbde81d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 monclient: found mon.noname-a 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 -- 172.21.15.98:0/1841360772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde81226f0 con 0x7fbde806fce0 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde659c640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbde806fce0 0x7fbde81408d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800d5f0 tx=0x7fbdd800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 client.?.objecter ms_handle_connect 0x7fbde806fce0 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 1 -- 172.21.15.98:0/1841360772 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbdd8012a60 con 0x7fbde806fce0 2024-09-11T03:08:09.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: dump: 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.028 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 1 -- 172.21.15.98:0/1841360772 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdd8019070 con 0x7fbde806fce0 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 1 -- 172.21.15.98:0/1841360772 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbdd8012c20 con 0x7fbde806fce0 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 5 monclient: authenticate success, global_id 104328 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _renew_subs 2024-09-11T03:08:09.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde8122c40 con 0x7fbde806fce0 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbde4d99640 4 set_mon_vals no callback set 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 monclient: dump: 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.030 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 client.104328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _renew_subs 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde80735f0 con 0x7fbde806fce0 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 librados: init done 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde813a260 con 0x7fbde806fce0 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 1 -- 172.21.15.98:0/1841360772 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fbdd8013070 con 0x7fbde806fce0 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbdc803df40 0x7fbdc8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 1 -- 172.21.15.98:0/1841360772 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fbdd800f040 con 0x7fbde806fce0 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 10 client.104328.objecter ms_dispatch 0x7fbde8000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 3 client.104328.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 3 client.104328.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.011+0000 7fbdd6ffd640 20 client.104328.objecter dump_active .. 0 homeless 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.015+0000 7fbde5d9b640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbdc803df40 0x7fbdc8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.015+0000 7fbde5d9b640 10 monclient: get_auth_request con 0x7fbdc803df40 auth_method 0 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.015+0000 7fbdd6ffd640 10 client.104328.objecter ms_handle_connect 0x7fbdc803df40 2024-09-11T03:08:09.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.015+0000 7fbde5d9b640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbdc803df40 0x7fbdc8040400 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0ad000 tx=0x7fbddc004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.019+0000 7fbdd6ffd640 1 -- 172.21.15.98:0/1841360772 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fbdd8023020 con 0x7fbde806fce0 2024-09-11T03:08:09.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.019+0000 7fbdd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.019+0000 7fbdd6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:09.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.187+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fbde80a94a0 con 0x7fbdc803df40 2024-09-11T03:08:09.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.187+0000 7fbdd6ffd640 1 -- 172.21.15.98:0/1841360772 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbde80a94a0 con 0x7fbdc803df40 2024-09-11T03:08:09.190 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 4 to deep-scrub 2024-09-11T03:08:09.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 10 librados: watch_flush enter 2024-09-11T03:08:09.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 10 librados: watch_flush exit 2024-09-11T03:08:09.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 20 client.104328.objecter shutdown clearing up homeless session... 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 10 client.104328.objecter successfully canceled tick 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbdc803df40 msgr2=0x7fbdc8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbdc803df40 0x7fbdc8040400 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0ad000 tx=0x7fbddc004740 comp rx=0 tx=0).stop 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 10 monclient: shutdown 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbde806fce0 msgr2=0x7fbde81408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbde806fce0 0x7fbde81408d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800d5f0 tx=0x7fbdd800dab0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 shutdown_connections 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fbdc803df40 0x7fbdc8040400 unknown :-1 s=CLOSED pgs=3723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fbde8139b80 0x7fbde813e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7fbde80724b0 0x7fbde81d1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 --2- 172.21.15.98:0/1841360772 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbde806fce0 0x7fbde81408d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 >> 172.21.15.98:0/1841360772 conn(0x7fbde80ac6b0 msgr2=0x7fbde80a9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 shutdown_connections 2024-09-11T03:08:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 -- 172.21.15.98:0/1841360772 wait complete. 2024-09-11T03:08:09.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.191+0000 7fbdecf75640 1 librados: shutdown 2024-09-11T03:08:09.212 INFO:tasks.ceph:Scrubbing osd.5 2024-09-11T03:08:09.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient: get_monmap_and_config 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient: monmap: 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:09.330150+0000 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:09.330150+0000 2024-09-11T03:08:09.327 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:09.328 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 1 Processor -- start 2024-09-11T03:08:09.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 1 -- start start 2024-09-11T03:08:09.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient: init 2024-09-11T03:08:09.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient: _add_conns ranks=[6,0,4,1,10,2,5,3,7,8,9] 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff424139b10 0x7ff4240725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient(hunting): picked mon.noname-d con 0x7ff424139b10 addr [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff42406fe30 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff42413ea00 0x7ff424140df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient(hunting): picked mon.noname-c con 0x7ff42413ea00 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff42413a600 con 0x7ff42406fe30 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff4240a9650 con 0x7ff42413ea00 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 1 -- --> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] -- mon_getmap magic: 0 -- 0x7ff424071350 con 0x7ff424139b10 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42aef1640 10 monclient(hunting): authenticate will time out at 2043.210497s 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff424139b10 0x7ff4240725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff42413ea00 0x7ff424140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff42413ea00 0x7ff424140df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53790/0 (socket says 172.21.15.98:53790) 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff423fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff42406fe30 0x7ff424139ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48244/0 (socket says 172.21.15.98:48244) 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 -- 172.21.15.98:0/2066572921 learned_addr learned my addr 172.21.15.98:0/2066572921 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): get_auth_request con 0x7ff42413ea00 auth_method 0 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 1 --2- >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff424139b10 0x7ff4240725d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3303/0 says I am v2:172.21.15.98:42966/0 (socket says 172.21.15.98:42966) 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 10 monclient(hunting): get_auth_request con 0x7ff424139b10 auth_method 0 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient(hunting): handle_auth_done global_id 24988 payload 274 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 -- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff424139b10 msgr2=0x7ff4240725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 --2- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff424139b10 0x7ff4240725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 -- 172.21.15.98:0/2066572921 >> v1:172.21.15.98:6789/0 conn(0x7ff42406fe30 legacy=0x7ff424139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 monclient: found mon.noname-c 2024-09-11T03:08:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff42948e640 1 -- 172.21.15.98:0/2066572921 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff424071630 con 0x7ff42413ea00 2024-09-11T03:08:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff4237fe640 1 -- 172.21.15.98:0/2066572921 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4083781209 0 0) 0x7ff42413a600 con 0x7ff42406fe30 2024-09-11T03:08:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff428c8d640 1 --2- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff424139b10 0x7ff4240725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.327+0000 7ff4237fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42948e640 1 --2- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff42413ea00 0x7ff424140df0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098a0 tx=0x7ff41401c120 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7d5f70549413c06b server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 1 -- 172.21.15.98:0/2066572921 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4140b1070 con 0x7ff42413ea00 2024-09-11T03:08:09.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: dump: 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 1 -- 172.21.15.98:0/2066572921 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4140a4b90 con 0x7ff42413ea00 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 5 monclient: authenticate success, global_id 24988 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 4 set_mon_vals no callback set 2024-09-11T03:08:09.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 1 -- 172.21.15.98:0/2066572921 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4140a4d40 con 0x7ff42413ea00 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4237fe640 10 monclient: dump: 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.343 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: shutdown 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff42413ea00 msgr2=0x7ff424140df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 --2- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff42413ea00 0x7ff424140df0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ff4140098a0 tx=0x7ff41401c120 comp rx=0 tx=0).stop 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- 172.21.15.98:0/2066572921 shutdown_connections 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 --2- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff42413ea00 0x7ff424140df0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 --2- 172.21.15.98:0/2066572921 >> [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] conn(0x7ff424139b10 0x7ff4240725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- 172.21.15.98:0/2066572921 >> 172.21.15.98:0/2066572921 conn(0x7ff4240ac770 msgr2=0x7ff4240a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- 172.21.15.98:0/2066572921 shutdown_connections 2024-09-11T03:08:09.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- 172.21.15.98:0/2066572921 wait complete. 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: monmap: 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.345 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.346 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 librados: starting msgr at 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 librados: starting objecter 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 Processor -- start 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- start start 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 librados: setting wanted keys 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 librados: calling monclient init 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: init 2024-09-11T03:08:09.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: _add_conns ranks=[4,7,5,8,3,6,1,2,0] 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff424139b10 0x7ff4240a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): picked mon.noname-c con 0x7ff424139b10 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff42413ea00 0x7ff4241d1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): picked mon.noname-i con 0x7ff42413ea00 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff4241d23d0 0x7ff4240a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): picked mon.noname-h con 0x7ff4241d23d0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7ff424144700 con 0x7ff424139b10 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7ff424144580 con 0x7ff4241d23d0 2024-09-11T03:08:09.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7ff424144880 con 0x7ff42413ea00 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient(hunting): authenticate will time out at 2043.215236s 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff424139b10 0x7ff4240a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff424139b10 0x7ff4240a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3302/0 says I am v2:172.21.15.98:53800/0 (socket says 172.21.15.98:53800) 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 -- 172.21.15.98:0/2792856834 learned_addr learned my addr 172.21.15.98:0/2792856834 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): get_auth_request con 0x7ff424139b10 auth_method 0 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient(hunting): handle_auth_done global_id 24997 payload 824 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 -- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff42413ea00 msgr2=0x7ff4241d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7ff42413ea00 0x7ff4241d1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 -- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff4241d23d0 msgr2=0x7ff4240a2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7ff4241d23d0 0x7ff4240a2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 monclient: found mon.noname-c 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.98:3302/0 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 -- 172.21.15.98:0/2792856834 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff424122c90 con 0x7ff424139b10 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff428c8d640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff424139b10 0x7ff4240a51d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00ec10 tx=0x7ff41c00c520 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 client.?.objecter ms_handle_connect 0x7ff424139b10 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== mon.4 v2:172.21.15.98:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff41c017070 con 0x7ff424139b10 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: dump: 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 5 monclient: authenticate success, global_id 24997 2024-09-11T03:08:09.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== mon.4 v2:172.21.15.98:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff41c013470 con 0x7ff424139b10 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: _renew_subs 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff424122f70 con 0x7ff424139b10 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== mon.4 v2:172.21.15.98:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff41c013a90 con 0x7ff424139b10 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff422ffd640 4 set_mon_vals no callback set 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff42aef1640 10 client.24997.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.331+0000 7ff4217fa640 10 monclient: dump: 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.354 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 10 monclient: _renew_subs 2024-09-11T03:08:09.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4240a8db0 con 0x7ff424139b10 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 1 librados: init done 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 10 client.24997.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 15 client.24997.objecter _session_command_op_assign -1 1 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 20 client.24997.objecter _get_session osd=-1 returning homeless 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 10 client.24997.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.98:3302/0 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff42413e510 con 0x7ff424139b10 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== mon.4 v2:172.21.15.98:3302/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7ff41c013c30 con 0x7ff424139b10 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff40403df40 0x7ff404040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== mon.4 v2:172.21.15.98:3302/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7ff41c00f040 con 0x7ff424139b10 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 client.24997.objecter ms_dispatch 0x7ff424000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:09.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 3 client.24997.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 3 client.24997.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 client.24997.objecter checking command 1 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7ff404045700 0x7ff404047ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 20 client.24997.objecter _get_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 20 client.24997.objecter put_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 15 client.24997.objecter _session_command_op_remove -1 1 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 20 client.24997.objecter _get_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 20 client.24997.objecter get_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 15 client.24997.objecter _session_command_op_assign 5 1 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 20 client.24997.objecter put_session s=0x7ff404045030 osd=5 4 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 client.24997.objecter _send_command 1 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff41c01bec0 con 0x7ff404045700 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 20 client.24997.objecter dump_active .. 0 homeless 2024-09-11T03:08:09.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== mon.4 v2:172.21.15.98:3302/0 6 ==== mon_get_version_reply(handle=1 version=699) ==== 24+0+0 (secure 0 0 0) 0x7ff41c056050 con 0x7ff424139b10 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 monclient: handle_get_version_reply finishing 1 version 699 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff423fff640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff40403df40 0x7ff404040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42948e640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7ff404045700 0x7ff404047ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff423fff640 10 monclient: get_auth_request con 0x7ff40403df40 auth_method 0 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42948e640 10 monclient: get_auth_request con 0x7ff404045700 auth_method 0 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 client.24997.objecter ms_handle_connect 0x7ff404045700 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff42948e640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7ff404045700 0x7ff404047ba0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 client.24997.objecter ms_handle_connect 0x7ff40403df40 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff423fff640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff40403df40 0x7ff404040400 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7ff40c000f90 tx=0x7ff40c007c60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== osd.5 v2:172.21.15.190:6808/1497028198 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff41c01bec0 con 0x7ff404045700 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 client.24997.objecter ms_dispatch 0x7ff424000ba0 command_reply(tid 1: 0 ) 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 10 client.24997.objecter _finish_command 1 = system:0 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 20 client.24997.objecter put_session s=0x7ff404045030 osd=5 4 2024-09-11T03:08:09.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.335+0000 7ff4217fa640 15 client.24997.objecter _session_command_op_remove 5 1 2024-09-11T03:08:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.347+0000 7ff42aef1640 10 client.24997.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-11T03:08:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 15 client.24997.objecter _session_command_op_assign -1 2 2024-09-11T03:08:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 client.24997.objecter _get_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 client.24997.objecter put_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 client.24997.objecter _get_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 15 client.24997.objecter _session_command_op_remove -1 2 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 client.24997.objecter get_session s=0x7ff404045030 osd=5 3 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 15 client.24997.objecter _session_command_op_assign 5 2 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 client.24997.objecter put_session s=0x7ff404045030 osd=5 4 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 10 client.24997.objecter _send_command 2 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 --> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff4240725d0 con 0x7ff404045700 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff4217fa640 1 -- 172.21.15.98:0/2792856834 <== osd.5 v2:172.21.15.190:6808/1497028198 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff4240725d0 con 0x7ff404045700 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff4217fa640 10 client.24997.objecter ms_dispatch 0x7ff424000ba0 command_reply(tid 2: 0 ) 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff4217fa640 10 client.24997.objecter _finish_command 2 = system:0 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff4217fa640 20 client.24997.objecter put_session s=0x7ff404045030 osd=5 4 2024-09-11T03:08:09.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff4217fa640 15 client.24997.objecter _session_command_op_remove 5 2 2024-09-11T03:08:09.361 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-11T03:08:09.361 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T03:08:09.361 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 10 librados: watch_flush enter 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 10 librados: watch_flush exit 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 10 client.24997.objecter close_session for osd.5 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7ff404045700 msgr2=0x7ff404047ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6808/1497028198,v1:172.21.15.190:6809/1497028198] conn(0x7ff404045700 0x7ff404047ba0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 client.24997.objecter put_session s=0x7ff404045030 osd=5 1 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 client.24997.objecter shutdown clearing up homeless session... 2024-09-11T03:08:09.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 10 client.24997.objecter successfully canceled tick 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff40403df40 msgr2=0x7ff404040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7ff40403df40 0x7ff404040400 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7ff40c000f90 tx=0x7ff40c007c60 comp rx=0 tx=0).stop 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 10 monclient: shutdown 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff424139b10 msgr2=0x7ff4240a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 --2- 172.21.15.98:0/2792856834 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7ff424139b10 0x7ff4240a51d0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ff41c00ec10 tx=0x7ff41c00c520 comp rx=0 tx=0).stop 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42948e640 1 -- 172.21.15.98:0/2792856834 reap_dead start 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 shutdown_connections 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 >> 172.21.15.98:0/2792856834 conn(0x7ff4240ac770 msgr2=0x7ff42413f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 shutdown_connections 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 -- 172.21.15.98:0/2792856834 wait complete. 2024-09-11T03:08:09.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.351+0000 7ff42aef1640 1 librados: shutdown 2024-09-11T03:08:09.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-11T03:08:09.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient: get_monmap_and_config 2024-09-11T03:08:09.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient: monmap: 2024-09-11T03:08:09.467 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:09.469950+0000 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:09.469950+0000 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.468 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 Processor -- start 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 -- start start 2024-09-11T03:08:09.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient: init 2024-09-11T03:08:09.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient: _add_conns ranks=[7,3,10,5,2,8,1,9,6,4,0] 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9870139b80 0x7f9870139f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): picked mon.noname-i con 0x7f9870139b80 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f987006fe50 0x7f98700a7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): picked mon.noname-g con 0x7f987006fe50 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9870072640 0x7f98700a7f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): picked mon.noname-f con 0x7f9870072640 addr v2:172.21.15.190:3300/0 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 -- --> v2:172.21.15.190:3300/0 -- mon_getmap magic: 0 -- 0x7f98700719c0 con 0x7f9870072640 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f9870071c10 con 0x7f987006fe50 2024-09-11T03:08:09.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7f98700702d0 con 0x7f9870139b80 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f9875d87640 10 monclient(hunting): authenticate will time out at 2043.349853s 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9870072640 0x7f98700a7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f987006fe50 0x7f98700a7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9870139b80 0x7f9870139f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 --2- >> v2:172.21.15.190:3300/0 conn(0x7f9870072640 0x7f98700a7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.98:60246/0 (socket says 172.21.15.98:60246) 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 -- 172.21.15.98:0/2226469981 learned_addr learned my addr 172.21.15.98:0/2226469981 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): get_auth_request con 0x7f9870072640 auth_method 0 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 10 monclient(hunting): get_auth_request con 0x7f987006fe50 auth_method 0 2024-09-11T03:08:09.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 10 monclient(hunting): get_auth_request con 0x7f9870139b80 auth_method 0 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient(hunting): handle_auth_done global_id 26434 payload 274 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 -- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9870139b80 msgr2=0x7f9870139f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 --2- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9870139b80 0x7f9870139f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 -- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f987006fe50 msgr2=0x7f98700a7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 --2- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f987006fe50 0x7f98700a7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 monclient: found mon.noname-f 2024-09-11T03:08:09.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.190:3300/0 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.467+0000 7f986ffff640 1 -- 172.21.15.98:0/2226469981 --> v2:172.21.15.190:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9870071370 con 0x7f9870072640 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986ffff640 1 --2- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 0x7f98700a7f50 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f9868009e80 tx=0x7f986801c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5497e3802d8f1ed server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 1 -- 172.21.15.98:0/2226469981 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98680090d0 con 0x7f9870072640 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: dump: 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.482 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 5 monclient: authenticate success, global_id 26434 2024-09-11T03:08:09.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 1 -- 172.21.15.98:0/2226469981 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98680a5a90 con 0x7f9870072640 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 4 set_mon_vals no callback set 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 1 -- 172.21.15.98:0/2226469981 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98680a5c80 con 0x7f9870072640 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986e7fc640 10 monclient: dump: 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient: shutdown 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 msgr2=0x7f98700a7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 --2- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 0x7f98700a7f50 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f9868009e80 tx=0x7f986801c8d0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- 172.21.15.98:0/2226469981 shutdown_connections 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 --2- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 0x7f98700a7f50 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 --2- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f987006fe50 0x7f98700a7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 --2- 172.21.15.98:0/2226469981 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7f9870139b80 0x7f9870139f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- 172.21.15.98:0/2226469981 >> 172.21.15.98:0/2226469981 conn(0x7f98700ac730 msgr2=0x7f98700a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- 172.21.15.98:0/2226469981 shutdown_connections 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- 172.21.15.98:0/2226469981 wait complete. 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient: monmap: 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:09.486 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 librados: starting msgr at 2024-09-11T03:08:09.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 librados: starting objecter 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 Processor -- start 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- start start 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 librados: setting wanted keys 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 librados: calling monclient init 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient: init 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient: _add_conns ranks=[2,1,5,7,6,8,0,3,4] 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f987006fe50 0x7f9870134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): picked mon.noname-b con 0x7f987006fe50 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:09.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 0x7f9870134fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): picked mon.noname-f con 0x7f9870072640 addr [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9870139b80 0x7f987013a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): picked mon.noname-h con 0x7f9870139b80 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f98701435c0 con 0x7f9870072640 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f9870143440 con 0x7f987006fe50 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f9870143740 con 0x7f9870139b80 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f9875d87640 10 monclient(hunting): authenticate will time out at 2043.355557s 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f987006fe50 0x7f9870134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f987006fe50 0x7f9870134a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:42186/0 (socket says 172.21.15.98:42186) 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 1 -- 172.21.15.98:0/132794501 learned_addr learned my addr 172.21.15.98:0/132794501 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): get_auth_request con 0x7f987006fe50 auth_method 0 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986effd640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 0x7f9870134fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986effd640 10 monclient(hunting): get_auth_request con 0x7f9870072640 auth_method 0 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986effd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986ffff640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9870139b80 0x7f987013a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.471+0000 7f986f7fe640 10 monclient(hunting): handle_auth_done global_id 94475 payload 824 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 1 -- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9870139b80 msgr2=0x7f987013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9870139b80 0x7f987013a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 1 -- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 msgr2=0x7f9870134fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 0x7f9870134fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 1 monclient: found mon.noname-b 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 1 -- 172.21.15.98:0/132794501 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987013e1d0 con 0x7f987006fe50 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986f7fe640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f987006fe50 0x7f9870134a60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f986000e870 tx=0x7f986000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 client.?.objecter ms_handle_connect 0x7f987006fe50 2024-09-11T03:08:09.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 1 -- 172.21.15.98:0/132794501 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9860017070 con 0x7f987006fe50 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: dump: 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.492 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 1 -- 172.21.15.98:0/132794501 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986000cdb0 con 0x7f987006fe50 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 5 monclient: authenticate success, global_id 94475 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 monclient: _renew_subs 2024-09-11T03:08:09.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98700aa310 con 0x7f987006fe50 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 1 -- 172.21.15.98:0/132794501 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98600138e0 con 0x7f987006fe50 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986dffb640 4 set_mon_vals no callback set 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 client.94475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 monclient: dump: 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.494 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 monclient: _renew_subs 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f987013c130 con 0x7f987006fe50 2024-09-11T03:08:09.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 1 librados: init done 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.98:3301/0 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9870139f60 con 0x7f987006fe50 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 1 -- 172.21.15.98:0/132794501 <== mon.2 v2:172.21.15.98:3301/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f9860024060 con 0x7f987006fe50 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985003de70 0x7f9850040330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 1 -- 172.21.15.98:0/132794501 <== mon.2 v2:172.21.15.98:3301/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f986000f040 con 0x7f987006fe50 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 client.94475.objecter ms_dispatch 0x7f9870000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 3 client.94475.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 3 client.94475.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 20 client.94475.objecter dump_active .. 0 homeless 2024-09-11T03:08:09.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986effd640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985003de70 0x7f9850040330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986effd640 10 monclient: get_auth_request con 0x7f985003de70 auth_method 0 2024-09-11T03:08:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f9867fff640 10 client.94475.objecter ms_handle_connect 0x7f985003de70 2024-09-11T03:08:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.475+0000 7f986effd640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985003de70 0x7f9850040330 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 crypto rx=0x7f9870070910 tx=0x7f9858008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.479+0000 7f9867fff640 1 -- 172.21.15.98:0/132794501 <== mon.2 v2:172.21.15.98:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f986000c940 con 0x7f987006fe50 2024-09-11T03:08:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.479+0000 7f9867fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.479+0000 7f9867fff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:09.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.639+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f9870069090 con 0x7f985003de70 2024-09-11T03:08:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.639+0000 7f9867fff640 1 -- 172.21.15.98:0/132794501 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9870069090 con 0x7f985003de70 2024-09-11T03:08:09.642 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 5 to deep-scrub 2024-09-11T03:08:09.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 10 librados: watch_flush enter 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 10 librados: watch_flush exit 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 20 client.94475.objecter shutdown clearing up homeless session... 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 10 client.94475.objecter successfully canceled tick 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985003de70 msgr2=0x7f9850040330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985003de70 0x7f9850040330 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 crypto rx=0x7f9870070910 tx=0x7f9858008040 comp rx=0 tx=0).stop 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 10 monclient: shutdown 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f987006fe50 msgr2=0x7f9870134a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f987006fe50 0x7f9870134a60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f986000e870 tx=0x7f986000ed30 comp rx=0 tx=0).stop 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 shutdown_connections 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f9870139b80 0x7f987013a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9870072640 0x7f9870134fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f985003de70 0x7f9850040330 unknown :-1 s=CLOSED pgs=3725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 --2- 172.21.15.98:0/132794501 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f987006fe50 0x7f9870134a60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 >> 172.21.15.98:0/132794501 conn(0x7f98700ac730 msgr2=0x7f98701385e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 shutdown_connections 2024-09-11T03:08:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 -- 172.21.15.98:0/132794501 wait complete. 2024-09-11T03:08:09.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.643+0000 7f9875d87640 1 librados: shutdown 2024-09-11T03:08:09.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T03:08:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient: get_monmap_and_config 2024-09-11T03:08:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient: monmap: 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:09.767339+0000 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:09.767339+0000 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.765 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:09.766 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:09.766 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 1 Processor -- start 2024-09-11T03:08:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 1 -- start start 2024-09-11T03:08:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient: init 2024-09-11T03:08:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient: _add_conns ranks=[9,8,0,5,1,6,3,4,10,7,2] 2024-09-11T03:08:09.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f66ec1375e0 0x7f66ec1379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient(hunting): picked mon.noname-a con 0x7f66ec1375e0 addr v2:172.21.15.98:3300/0 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec071eb0 0x7f66ec137f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient(hunting): picked mon.noname-e con 0x7f66ec071eb0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66ec0a4d90 addr v1:172.21.15.98:6789/0 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66ec071c70 con 0x7f66ec0a4d90 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f66ec072330 con 0x7f66ec1375e0 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f66ec138440 con 0x7f66ec071eb0 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f31f8640 10 monclient(hunting): authenticate will time out at 2043.647534s 2024-09-11T03:08:09.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec071eb0 0x7f66ec137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f66ec1375e0 0x7f66ec1379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f1795640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f66ec0a4d90 0x7f66ec0a51a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:48260/0 (socket says 172.21.15.98:48260) 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f1795640 1 -- 172.21.15.98:0/3485754979 learned_addr learned my addr 172.21.15.98:0/3485754979 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 10 monclient(hunting): get_auth_request con 0x7f66ec071eb0 auth_method 0 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 10 monclient(hunting): get_auth_request con 0x7f66ec1375e0 auth_method 0 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.763+0000 7f66f0f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 10 monclient(hunting): handle_auth_done global_id 104346 payload 274 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 1 -- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec071eb0 msgr2=0x7f66ec137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 1 --2- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec071eb0 0x7f66ec137f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 1 -- 172.21.15.98:0/3485754979 >> v1:172.21.15.98:6789/0 conn(0x7f66ec0a4d90 legacy=0x7f66ec0a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 1 monclient: found mon.noname-a 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 1 -- 172.21.15.98:0/3485754979 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec0a18d0 con 0x7f66ec1375e0 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 1 -- 172.21.15.98:0/3485754979 <== mon.0 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125864558 0 0) 0x7f66ec071c70 con 0x7f66ec0a4d90 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66ebfff640 1 --2- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec071eb0 0x7f66ec137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f0f94640 1 --2- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ec1375e0 0x7f66ec1379c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0097f0 tx=0x7f66dc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e522ec381afcad5c server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 1 -- 172.21.15.98:0/3485754979 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66dc0b0070 con 0x7f66ec1375e0 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: dump: 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.773 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.774 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 1 -- 172.21.15.98:0/3485754979 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66dc0aa810 con 0x7f66ec1375e0 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 1 -- 172.21.15.98:0/3485754979 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66dc0a9300 con 0x7f66ec1375e0 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 5 monclient: authenticate success, global_id 104346 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 4 set_mon_vals no callback set 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66eb7fe640 10 monclient: dump: 2024-09-11T03:08:09.775 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.776 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 10 monclient: shutdown 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 -- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ec1375e0 msgr2=0x7f66ec1379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 --2- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ec1375e0 0x7f66ec1379c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0097f0 tx=0x7f66dc01c360 comp rx=0 tx=0).stop 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 -- 172.21.15.98:0/3485754979 shutdown_connections 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 --2- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec071eb0 0x7f66ec137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 --2- 172.21.15.98:0/3485754979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66ec1375e0 0x7f66ec1379c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 -- 172.21.15.98:0/3485754979 >> 172.21.15.98:0/3485754979 conn(0x7f66ec0aa3f0 msgr2=0x7f66ec0aa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 -- 172.21.15.98:0/3485754979 shutdown_connections 2024-09-11T03:08:09.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 -- 172.21.15.98:0/3485754979 wait complete. 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 10 monclient: build_initial_monmap 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 10 monclient: monmap: 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:09.782 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 librados: starting msgr at 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 librados: starting objecter 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.767+0000 7f66f31f8640 1 Processor -- start 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 -- start start 2024-09-11T03:08:09.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 librados: setting wanted keys 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 librados: calling monclient init 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: init 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _add_conns ranks=[4,8,3,2,6,7,0,5,1] 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f66ec071eb0 0x7f66ec135f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): picked mon.noname-c con 0x7f66ec071eb0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec0a4d90 0x7f66ec136460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): picked mon.noname-e con 0x7f66ec0a4d90 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66ec1375e0 0x7f66ec13b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): picked mon.noname-g con 0x7f66ec1375e0 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7f66ec142600 con 0x7f66ec1375e0 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f66ec142480 con 0x7f66ec071eb0 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f66ec142780 con 0x7f66ec0a4d90 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient(hunting): authenticate will time out at 2043.652514s 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec0a4d90 0x7f66ec136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f66ec071eb0 0x7f66ec135f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec0a4d90 0x7f66ec136460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35290/0 (socket says 172.21.15.98:35290) 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 -- 172.21.15.98:0/1955393666 learned_addr learned my addr 172.21.15.98:0/1955393666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): get_auth_request con 0x7f66ec0a4d90 auth_method 0 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient(hunting): get_auth_request con 0x7f66ec071eb0 auth_method 0 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient(hunting): handle_auth_done global_id 25577 payload 824 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 -- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f66ec071eb0 msgr2=0x7f66ec135f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f66ec071eb0 0x7f66ec135f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 -- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66ec1375e0 msgr2=0x7f66ec13b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f1795640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66ec1375e0 0x7f66ec13b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66ec1375e0 0x7f66ec13b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 monclient: found mon.noname-e 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 -- 172.21.15.98:0/1955393666 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ec120eb0 con 0x7f66ec0a4d90 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f66ec071eb0 0x7f66ec135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66ebfff640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec0a4d90 0x7f66ec136460 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f66e000e980 tx=0x7f66e000ee40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 client.?.objecter ms_handle_connect 0x7f66ec0a4d90 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 1 -- 172.21.15.98:0/1955393666 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66e0017070 con 0x7f66ec0a4d90 2024-09-11T03:08:09.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: dump: 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.788 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: _finish_auth 0 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: _check_auth_tickets 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 1 -- 172.21.15.98:0/1955393666 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e000ce70 con 0x7f66ec0a4d90 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 5 monclient: authenticate success, global_id 25577 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _renew_subs 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:09.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ec0a3710 con 0x7f66ec0a4d90 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 1 -- 172.21.15.98:0/1955393666 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66e00138e0 con 0x7f66ec0a4d90 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 client.25577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66eaffd640 4 set_mon_vals no callback set 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 monclient: dump: 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:09.790 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _renew_subs 2024-09-11T03:08:09.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ec13d150 con 0x7f66ec0a4d90 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 librados: init done 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ec0a51a0 con 0x7f66ec0a4d90 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 1 -- 172.21.15.98:0/1955393666 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f66e0013470 con 0x7f66ec0a4d90 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66d803df40 0x7f66d8040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 1 -- 172.21.15.98:0/1955393666 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f66e000f040 con 0x7f66ec0a4d90 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 client.25577.objecter ms_dispatch 0x7f66ec000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 3 client.25577.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:09.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 3 client.25577.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 20 client.25577.objecter dump_active .. 0 homeless 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66d803df40 0x7f66d8040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 10 monclient: get_auth_request con 0x7f66d803df40 auth_method 0 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66e97fa640 10 client.25577.objecter ms_handle_connect 0x7f66d803df40 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.771+0000 7f66f0f94640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66d803df40 0x7f66d8040400 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0ad000 tx=0x7f66dc0b5040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.775+0000 7f66e97fa640 1 -- 172.21.15.98:0/1955393666 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f66e0021070 con 0x7f66ec0a4d90 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.775+0000 7f66e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:09.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.775+0000 7f66e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:09.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.931+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f66ec06e7f0 con 0x7f66d803df40 2024-09-11T03:08:09.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.935+0000 7f66e97fa640 1 -- 172.21.15.98:0/1955393666 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114038 (secure 0 0 0) 0x7f66dc0dcd90 con 0x7f66d803df40 2024-09-11T03:08:09.939 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T03:08:09.939 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T03:08:09.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 10 librados: watch_flush enter 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 10 librados: watch_flush exit 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 20 client.25577.objecter shutdown clearing up homeless session... 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 10 client.25577.objecter successfully canceled tick 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66d803df40 msgr2=0x7f66d8040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66d803df40 0x7f66d8040400 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0ad000 tx=0x7f66dc0b5040 comp rx=0 tx=0).stop 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 10 monclient: shutdown 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec0a4d90 msgr2=0x7f66ec136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec0a4d90 0x7f66ec136460 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f66e000e980 tx=0x7f66e000ee40 comp rx=0 tx=0).stop 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 shutdown_connections 2024-09-11T03:08:09.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f66d803df40 0x7f66d8040400 unknown :-1 s=CLOSED pgs=3726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7f66ec1375e0 0x7f66ec13b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f66ec0a4d90 0x7f66ec136460 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 --2- 172.21.15.98:0/1955393666 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f66ec071eb0 0x7f66ec135f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:09.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 >> 172.21.15.98:0/1955393666 conn(0x7f66ec0aa3f0 msgr2=0x7f66ec0a9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:09.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 shutdown_connections 2024-09-11T03:08:09.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 -- 172.21.15.98:0/1955393666 wait complete. 2024-09-11T03:08:09.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:09.939+0000 7f66f31f8640 1 librados: shutdown 2024-09-11T03:08:09.956 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1241,"stamp":"2024-09-11T03:08:08.135589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":451712,"kb_used_data":61920,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565779328,"statfs":{"total":579820584960,"available":579358031872,"internally_reserved":0,"allocated":63406080,"data_stored":35633462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":399039138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003747"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925479+0000","last_change":"2024-09-11T02:51:02.707390+0000","last_active":"2024-09-11T03:07:32.925479+0000","last_peered":"2024-09-11T03:07:32.925479+0000","last_clean":"2024-09-11T03:07:32.925479+0000","last_became_active":"2024-09-11T02:51:02.704466+0000","last_became_peered":"2024-09-11T02:51:02.704466+0000","last_unstale":"2024-09-11T03:07:32.925479+0000","last_undegraded":"2024-09-11T03:07:32.925479+0000","last_fullsized":"2024-09-11T03:07:32.925479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:51:38.363080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939381+0000","last_change":"2024-09-11T02:51:02.751947+0000","last_active":"2024-09-11T03:07:33.939381+0000","last_peered":"2024-09-11T03:07:33.939381+0000","last_clean":"2024-09-11T03:07:33.939381+0000","last_became_active":"2024-09-11T02:51:02.747449+0000","last_became_peered":"2024-09-11T02:51:02.747449+0000","last_unstale":"2024-09-11T03:07:33.939381+0000","last_undegraded":"2024-09-11T03:07:33.939381+0000","last_fullsized":"2024-09-11T03:07:33.939381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:44:34.946819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.923807+0000","last_change":"2024-09-11T02:51:02.757155+0000","last_active":"2024-09-11T03:07:32.923807+0000","last_peered":"2024-09-11T03:07:32.923807+0000","last_clean":"2024-09-11T03:07:32.923807+0000","last_became_active":"2024-09-11T02:51:02.746704+0000","last_became_peered":"2024-09-11T02:51:02.746704+0000","last_unstale":"2024-09-11T03:07:32.923807+0000","last_undegraded":"2024-09-11T03:07:32.923807+0000","last_fullsized":"2024-09-11T03:07:32.923807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:08:27.349903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938159+0000","last_change":"2024-09-11T02:51:02.703570+0000","last_active":"2024-09-11T03:07:33.938159+0000","last_peered":"2024-09-11T03:07:33.938159+0000","last_clean":"2024-09-11T03:07:33.938159+0000","last_became_active":"2024-09-11T02:51:02.695884+0000","last_became_peered":"2024-09-11T02:51:02.695884+0000","last_unstale":"2024-09-11T03:07:33.938159+0000","last_undegraded":"2024-09-11T03:07:33.938159+0000","last_fullsized":"2024-09-11T03:07:33.938159+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:24.055472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930411+0000","last_change":"2024-09-11T02:51:02.604922+0000","last_active":"2024-09-11T03:07:32.930411+0000","last_peered":"2024-09-11T03:07:32.930411+0000","last_clean":"2024-09-11T03:07:32.930411+0000","last_became_active":"2024-09-11T02:51:02.604043+0000","last_became_peered":"2024-09-11T02:51:02.604043+0000","last_unstale":"2024-09-11T03:07:32.930411+0000","last_undegraded":"2024-09-11T03:07:32.930411+0000","last_fullsized":"2024-09-11T03:07:32.930411+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:44:55.319066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932296+0000","last_change":"2024-09-11T02:51:02.574684+0000","last_active":"2024-09-11T03:07:33.932296+0000","last_peered":"2024-09-11T03:07:33.932296+0000","last_clean":"2024-09-11T03:07:33.932296+0000","last_became_active":"2024-09-11T02:51:02.574238+0000","last_became_peered":"2024-09-11T02:51:02.574238+0000","last_unstale":"2024-09-11T03:07:33.932296+0000","last_undegraded":"2024-09-11T03:07:33.932296+0000","last_fullsized":"2024-09-11T03:07:33.932296+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:10:59.496994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938047+0000","last_change":"2024-09-11T02:51:02.752301+0000","last_active":"2024-09-11T03:07:33.938047+0000","last_peered":"2024-09-11T03:07:33.938047+0000","last_clean":"2024-09-11T03:07:33.938047+0000","last_became_active":"2024-09-11T02:51:02.745188+0000","last_became_peered":"2024-09-11T02:51:02.745188+0000","last_unstale":"2024-09-11T03:07:33.938047+0000","last_undegraded":"2024-09-11T03:07:33.938047+0000","last_fullsized":"2024-09-11T03:07:33.938047+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:05:51.425341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924911+0000","last_change":"2024-09-11T02:51:02.583697+0000","last_active":"2024-09-11T03:07:32.924911+0000","last_peered":"2024-09-11T03:07:32.924911+0000","last_clean":"2024-09-11T03:07:32.924911+0000","last_became_active":"2024-09-11T02:51:02.578172+0000","last_became_peered":"2024-09-11T02:51:02.578172+0000","last_unstale":"2024-09-11T03:07:32.924911+0000","last_undegraded":"2024-09-11T03:07:32.924911+0000","last_fullsized":"2024-09-11T03:07:32.924911+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:55:49.570822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939024+0000","last_change":"2024-09-11T02:51:02.671125+0000","last_active":"2024-09-11T03:07:33.939024+0000","last_peered":"2024-09-11T03:07:33.939024+0000","last_clean":"2024-09-11T03:07:33.939024+0000","last_became_active":"2024-09-11T02:51:02.666973+0000","last_became_peered":"2024-09-11T02:51:02.666973+0000","last_unstale":"2024-09-11T03:07:33.939024+0000","last_undegraded":"2024-09-11T03:07:33.939024+0000","last_fullsized":"2024-09-11T03:07:33.939024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:57.810050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925926+0000","last_change":"2024-09-11T02:51:02.746877+0000","last_active":"2024-09-11T03:07:32.925926+0000","last_peered":"2024-09-11T03:07:32.925926+0000","last_clean":"2024-09-11T03:07:32.925926+0000","last_became_active":"2024-09-11T02:51:02.737993+0000","last_became_peered":"2024-09-11T02:51:02.737993+0000","last_unstale":"2024-09-11T03:07:32.925926+0000","last_undegraded":"2024-09-11T03:07:32.925926+0000","last_fullsized":"2024-09-11T03:07:32.925926+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:58:49.770401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932737+0000","last_change":"2024-09-11T02:51:02.692125+0000","last_active":"2024-09-11T03:07:33.932737+0000","last_peered":"2024-09-11T03:07:33.932737+0000","last_clean":"2024-09-11T03:07:33.932737+0000","last_became_active":"2024-09-11T02:51:02.671301+0000","last_became_peered":"2024-09-11T02:51:02.671301+0000","last_unstale":"2024-09-11T03:07:33.932737+0000","last_undegraded":"2024-09-11T03:07:33.932737+0000","last_fullsized":"2024-09-11T03:07:33.932737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:39:47.378191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930877+0000","last_change":"2024-09-11T02:51:02.713974+0000","last_active":"2024-09-11T03:07:32.930877+0000","last_peered":"2024-09-11T03:07:32.930877+0000","last_clean":"2024-09-11T03:07:32.930877+0000","last_became_active":"2024-09-11T02:51:02.712319+0000","last_became_peered":"2024-09-11T02:51:02.712319+0000","last_unstale":"2024-09-11T03:07:32.930877+0000","last_undegraded":"2024-09-11T03:07:32.930877+0000","last_fullsized":"2024-09-11T03:07:32.930877+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:12.353704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1353,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930489+0000","last_change":"2024-09-11T02:51:02.678057+0000","last_active":"2024-09-11T03:07:32.930489+0000","last_peered":"2024-09-11T03:07:32.930489+0000","last_clean":"2024-09-11T03:07:32.930489+0000","last_became_active":"2024-09-11T02:51:02.672559+0000","last_became_peered":"2024-09-11T02:51:02.672559+0000","last_unstale":"2024-09-11T03:07:32.930489+0000","last_undegraded":"2024-09-11T03:07:32.930489+0000","last_fullsized":"2024-09-11T03:07:32.930489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:17:41.716691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":1468,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937864+0000","last_change":"2024-09-11T02:49:56.898304+0000","last_active":"2024-09-11T03:07:33.937864+0000","last_peered":"2024-09-11T03:07:33.937864+0000","last_clean":"2024-09-11T03:07:33.937864+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T03:07:33.937864+0000","last_undegraded":"2024-09-11T03:07:33.937864+0000","last_fullsized":"2024-09-11T03:07:33.937864+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:49:55.869128+0000","last_clean_scrub_stamp":"2024-09-11T02:49:55.869128+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:38:49.137592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1371,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930535+0000","last_change":"2024-09-11T02:50:04.974249+0000","last_active":"2024-09-11T03:07:32.930535+0000","last_peered":"2024-09-11T03:07:32.930535+0000","last_clean":"2024-09-11T03:07:32.930535+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T03:07:32.930535+0000","last_undegraded":"2024-09-11T03:07:32.930535+0000","last_fullsized":"2024-09-11T03:07:32.930535+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:04:43.909555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1352,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932586+0000","last_change":"2024-09-11T02:51:02.712255+0000","last_active":"2024-09-11T03:07:33.932586+0000","last_peered":"2024-09-11T03:07:33.932586+0000","last_clean":"2024-09-11T03:07:33.932586+0000","last_became_active":"2024-09-11T02:51:02.709268+0000","last_became_peered":"2024-09-11T02:51:02.709268+0000","last_unstale":"2024-09-11T03:07:33.932586+0000","last_undegraded":"2024-09-11T03:07:33.932586+0000","last_fullsized":"2024-09-11T03:07:33.932586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:55:41.994663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1377,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937993+0000","last_change":"2024-09-11T02:50:05.027630+0000","last_active":"2024-09-11T03:07:33.937993+0000","last_peered":"2024-09-11T03:07:33.937993+0000","last_clean":"2024-09-11T03:07:33.937993+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T03:07:33.937993+0000","last_undegraded":"2024-09-11T03:07:33.937993+0000","last_fullsized":"2024-09-11T03:07:33.937993+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:45.794848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1355,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.931087+0000","last_change":"2024-09-11T02:51:02.592214+0000","last_active":"2024-09-11T03:07:32.931087+0000","last_peered":"2024-09-11T03:07:32.931087+0000","last_clean":"2024-09-11T03:07:32.931087+0000","last_became_active":"2024-09-11T02:51:02.587630+0000","last_became_peered":"2024-09-11T02:51:02.587630+0000","last_unstale":"2024-09-11T03:07:32.931087+0000","last_undegraded":"2024-09-11T03:07:32.931087+0000","last_fullsized":"2024-09-11T03:07:32.931087+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:26:26.107220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924408+0000","last_change":"2024-09-11T02:50:04.975112+0000","last_active":"2024-09-11T03:07:32.924408+0000","last_peered":"2024-09-11T03:07:32.924408+0000","last_clean":"2024-09-11T03:07:32.924408+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T03:07:32.924408+0000","last_undegraded":"2024-09-11T03:07:32.924408+0000","last_fullsized":"2024-09-11T03:07:32.924408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:29.741593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1351,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932714+0000","last_change":"2024-09-11T02:51:02.686284+0000","last_active":"2024-09-11T03:07:33.932714+0000","last_peered":"2024-09-11T03:07:33.932714+0000","last_clean":"2024-09-11T03:07:33.932714+0000","last_became_active":"2024-09-11T02:51:02.679438+0000","last_became_peered":"2024-09-11T02:51:02.679438+0000","last_unstale":"2024-09-11T03:07:33.932714+0000","last_undegraded":"2024-09-11T03:07:33.932714+0000","last_fullsized":"2024-09-11T03:07:33.932714+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:30:09.310176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1372,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930275+0000","last_change":"2024-09-11T02:50:04.973381+0000","last_active":"2024-09-11T03:07:32.930275+0000","last_peered":"2024-09-11T03:07:32.930275+0000","last_clean":"2024-09-11T03:07:32.930275+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T03:07:32.930275+0000","last_undegraded":"2024-09-11T03:07:32.930275+0000","last_fullsized":"2024-09-11T03:07:32.930275+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.169403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1363,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924938+0000","last_change":"2024-09-11T02:51:02.707327+0000","last_active":"2024-09-11T03:07:32.924938+0000","last_peered":"2024-09-11T03:07:32.924938+0000","last_clean":"2024-09-11T03:07:32.924938+0000","last_became_active":"2024-09-11T02:51:02.655843+0000","last_became_peered":"2024-09-11T02:51:02.655843+0000","last_unstale":"2024-09-11T03:07:32.924938+0000","last_undegraded":"2024-09-11T03:07:32.924938+0000","last_fullsized":"2024-09-11T03:07:32.924938+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:24.067746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932558+0000","last_change":"2024-09-11T02:50:04.976148+0000","last_active":"2024-09-11T03:07:33.932558+0000","last_peered":"2024-09-11T03:07:33.932558+0000","last_clean":"2024-09-11T03:07:33.932558+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T03:07:33.932558+0000","last_undegraded":"2024-09-11T03:07:33.932558+0000","last_fullsized":"2024-09-11T03:07:33.932558+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:19:15.212757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"22'7","reported_seq":1365,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932452+0000","last_change":"2024-09-11T02:51:02.618073+0000","last_active":"2024-09-11T03:07:33.932452+0000","last_peered":"2024-09-11T03:07:33.932452+0000","last_clean":"2024-09-11T03:07:33.932452+0000","last_became_active":"2024-09-11T02:51:02.586208+0000","last_became_peered":"2024-09-11T02:51:02.586208+0000","last_unstale":"2024-09-11T03:07:33.932452+0000","last_undegraded":"2024-09-11T03:07:33.932452+0000","last_fullsized":"2024-09-11T03:07:33.932452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:45:11.456188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938063+0000","last_change":"2024-09-11T02:50:05.027854+0000","last_active":"2024-09-11T03:07:33.938063+0000","last_peered":"2024-09-11T03:07:33.938063+0000","last_clean":"2024-09-11T03:07:33.938063+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T03:07:33.938063+0000","last_undegraded":"2024-09-11T03:07:33.938063+0000","last_fullsized":"2024-09-11T03:07:33.938063+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:21:22.953722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938507+0000","last_change":"2024-09-11T02:51:02.645107+0000","last_active":"2024-09-11T03:07:33.938507+0000","last_peered":"2024-09-11T03:07:33.938507+0000","last_clean":"2024-09-11T03:07:33.938507+0000","last_became_active":"2024-09-11T02:51:02.642031+0000","last_became_peered":"2024-09-11T02:51:02.642031+0000","last_unstale":"2024-09-11T03:07:33.938507+0000","last_undegraded":"2024-09-11T03:07:33.938507+0000","last_fullsized":"2024-09-11T03:07:33.938507+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:02:05.310788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932866+0000","last_change":"2024-09-11T02:50:04.976171+0000","last_active":"2024-09-11T03:07:33.932866+0000","last_peered":"2024-09-11T03:07:33.932866+0000","last_clean":"2024-09-11T03:07:33.932866+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T03:07:33.932866+0000","last_undegraded":"2024-09-11T03:07:33.932866+0000","last_fullsized":"2024-09-11T03:07:33.932866+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:33:21.324243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938183+0000","last_change":"2024-09-11T02:51:02.753940+0000","last_active":"2024-09-11T03:07:33.938183+0000","last_peered":"2024-09-11T03:07:33.938183+0000","last_clean":"2024-09-11T03:07:33.938183+0000","last_became_active":"2024-09-11T02:51:02.748048+0000","last_became_peered":"2024-09-11T02:51:02.748048+0000","last_unstale":"2024-09-11T03:07:33.938183+0000","last_undegraded":"2024-09-11T03:07:33.938183+0000","last_fullsized":"2024-09-11T03:07:33.938183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:44.448053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1376,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939090+0000","last_change":"2024-09-11T02:50:05.028180+0000","last_active":"2024-09-11T03:07:33.939090+0000","last_peered":"2024-09-11T03:07:33.939090+0000","last_clean":"2024-09-11T03:07:33.939090+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T03:07:33.939090+0000","last_undegraded":"2024-09-11T03:07:33.939090+0000","last_fullsized":"2024-09-11T03:07:33.939090+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:50:01.918794+0000","last_clean_scrub_stamp":"2024-09-11T02:50:01.918794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:23.492707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924172+0000","last_change":"2024-09-11T02:51:02.654035+0000","last_active":"2024-09-11T03:07:32.924172+0000","last_peered":"2024-09-11T03:07:32.924172+0000","last_clean":"2024-09-11T03:07:32.924172+0000","last_became_active":"2024-09-11T02:51:02.653755+0000","last_became_peered":"2024-09-11T02:51:02.653755+0000","last_unstale":"2024-09-11T03:07:32.924172+0000","last_undegraded":"2024-09-11T03:07:32.924172+0000","last_fullsized":"2024-09-11T03:07:32.924172+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:30:31.581016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925152+0000","last_change":"2024-09-11T02:51:02.615081+0000","last_active":"2024-09-11T03:07:32.925152+0000","last_peered":"2024-09-11T03:07:32.925152+0000","last_clean":"2024-09-11T03:07:32.925152+0000","last_became_active":"2024-09-11T02:51:02.594230+0000","last_became_peered":"2024-09-11T02:51:02.594230+0000","last_unstale":"2024-09-11T03:07:32.925152+0000","last_undegraded":"2024-09-11T03:07:32.925152+0000","last_fullsized":"2024-09-11T03:07:32.925152+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:18.916193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1356,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925259+0000","last_change":"2024-09-11T02:51:02.736915+0000","last_active":"2024-09-11T03:07:32.925259+0000","last_peered":"2024-09-11T03:07:32.925259+0000","last_clean":"2024-09-11T03:07:32.925259+0000","last_became_active":"2024-09-11T02:51:02.736590+0000","last_became_peered":"2024-09-11T02:51:02.736590+0000","last_unstale":"2024-09-11T03:07:32.925259+0000","last_undegraded":"2024-09-11T03:07:32.925259+0000","last_fullsized":"2024-09-11T03:07:32.925259+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:57:38.031885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930585+0000","last_change":"2024-09-11T02:51:02.678968+0000","last_active":"2024-09-11T03:07:32.930585+0000","last_peered":"2024-09-11T03:07:32.930585+0000","last_clean":"2024-09-11T03:07:32.930585+0000","last_became_active":"2024-09-11T02:51:02.678439+0000","last_became_peered":"2024-09-11T02:51:02.678439+0000","last_unstale":"2024-09-11T03:07:32.930585+0000","last_undegraded":"2024-09-11T03:07:32.930585+0000","last_fullsized":"2024-09-11T03:07:32.930585+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:41:16.872342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1350,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.932552+0000","last_change":"2024-09-11T02:51:02.705476+0000","last_active":"2024-09-11T03:07:33.932552+0000","last_peered":"2024-09-11T03:07:33.932552+0000","last_clean":"2024-09-11T03:07:33.932552+0000","last_became_active":"2024-09-11T02:51:02.700809+0000","last_became_peered":"2024-09-11T02:51:02.700809+0000","last_unstale":"2024-09-11T03:07:33.932552+0000","last_undegraded":"2024-09-11T03:07:33.932552+0000","last_fullsized":"2024-09-11T03:07:33.932552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:31:56.654998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.938812+0000","last_change":"2024-09-11T02:51:02.698839+0000","last_active":"2024-09-11T03:07:33.938812+0000","last_peered":"2024-09-11T03:07:33.938812+0000","last_clean":"2024-09-11T03:07:33.938812+0000","last_became_active":"2024-09-11T02:51:02.695758+0000","last_became_peered":"2024-09-11T02:51:02.695758+0000","last_unstale":"2024-09-11T03:07:33.938812+0000","last_undegraded":"2024-09-11T03:07:33.938812+0000","last_fullsized":"2024-09-11T03:07:33.938812+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:37:00.606584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1357,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.925801+0000","last_change":"2024-09-11T02:51:02.715299+0000","last_active":"2024-09-11T03:07:32.925801+0000","last_peered":"2024-09-11T03:07:32.925801+0000","last_clean":"2024-09-11T03:07:32.925801+0000","last_became_active":"2024-09-11T02:51:02.715041+0000","last_became_peered":"2024-09-11T02:51:02.715041+0000","last_unstale":"2024-09-11T03:07:32.925801+0000","last_undegraded":"2024-09-11T03:07:32.925801+0000","last_fullsized":"2024-09-11T03:07:32.925801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:48:39.561287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.939022+0000","last_change":"2024-09-11T02:51:02.582470+0000","last_active":"2024-09-11T03:07:33.939022+0000","last_peered":"2024-09-11T03:07:33.939022+0000","last_clean":"2024-09-11T03:07:33.939022+0000","last_became_active":"2024-09-11T02:51:02.581916+0000","last_became_peered":"2024-09-11T02:51:02.581916+0000","last_unstale":"2024-09-11T03:07:33.939022+0000","last_undegraded":"2024-09-11T03:07:33.939022+0000","last_fullsized":"2024-09-11T03:07:33.939022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:36:11.711933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1354,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.930286+0000","last_change":"2024-09-11T02:51:02.708269+0000","last_active":"2024-09-11T03:07:32.930286+0000","last_peered":"2024-09-11T03:07:32.930286+0000","last_clean":"2024-09-11T03:07:32.930286+0000","last_became_active":"2024-09-11T02:51:02.708034+0000","last_became_peered":"2024-09-11T02:51:02.708034+0000","last_unstale":"2024-09-11T03:07:32.930286+0000","last_undegraded":"2024-09-11T03:07:32.930286+0000","last_fullsized":"2024-09-11T03:07:32.930286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:44:19.177182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924808+0000","last_change":"2024-09-11T02:51:02.711915+0000","last_active":"2024-09-11T03:07:32.924808+0000","last_peered":"2024-09-11T03:07:32.924808+0000","last_clean":"2024-09-11T03:07:32.924808+0000","last_became_active":"2024-09-11T02:51:02.684472+0000","last_became_peered":"2024-09-11T02:51:02.684472+0000","last_unstale":"2024-09-11T03:07:32.924808+0000","last_undegraded":"2024-09-11T03:07:32.924808+0000","last_fullsized":"2024-09-11T03:07:32.924808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:37:56.385712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1361,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:32.924685+0000","last_change":"2024-09-11T02:51:02.637360+0000","last_active":"2024-09-11T03:07:32.924685+0000","last_peered":"2024-09-11T03:07:32.924685+0000","last_clean":"2024-09-11T03:07:32.924685+0000","last_became_active":"2024-09-11T02:51:02.603633+0000","last_became_peered":"2024-09-11T02:51:02.603633+0000","last_unstale":"2024-09-11T03:07:32.924685+0000","last_undegraded":"2024-09-11T03:07:32.924685+0000","last_fullsized":"2024-09-11T03:07:32.924685+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:25:57.435918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1358,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:07:33.937869+0000","last_change":"2024-09-11T02:51:02.659574+0000","last_active":"2024-09-11T03:07:33.937869+0000","last_peered":"2024-09-11T03:07:33.937869+0000","last_clean":"2024-09-11T03:07:33.937869+0000","last_became_active":"2024-09-11T02:51:02.659063+0000","last_became_peered":"2024-09-11T02:51:02.659063+0000","last_unstale":"2024-09-11T03:07:33.937869+0000","last_undegraded":"2024-09-11T03:07:33.937869+0000","last_fullsized":"2024-09-11T03:07:33.937869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T02:51:01.391757+0000","last_clean_scrub_stamp":"2024-09-11T02:51:01.391757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:28:59.462620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607775,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77408,"kb_used_data":10128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294432,"statfs":{"total":96636764160,"available":96557498368,"internally_reserved":0,"allocated":10371072,"data_stored":5742132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607773,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77404,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294436,"statfs":{"total":96636764160,"available":96557502464,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607776,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77988,"kb_used_data":10708,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293852,"statfs":{"total":96636764160,"available":96556904448,"internally_reserved":0,"allocated":10964992,"data_stored":6332486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607776,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85724,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286116,"statfs":{"total":96636764160,"available":96548982784,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77412,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294428,"statfs":{"total":96636764160,"available":96557494272,"internally_reserved":0,"allocated":10375168,"data_stored":5742137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640478,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55776,"kb_used_data":10704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":10960896,"data_stored":6332481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:08:09.964 INFO:tasks.ceph:pgid 6.16 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.964 INFO:tasks.ceph:pgid 6.17 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.964 INFO:tasks.ceph:pgid 6.14 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.964 INFO:tasks.ceph:pgid 6.15 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.a last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.b last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.8 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.9 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.e last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.f last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.c last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.965 INFO:tasks.ceph:pgid 6.d last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 6.7 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T02:49:55.869128+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=49, tm_sec=55, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 6.4 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 6.5 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.966 INFO:tasks.ceph:pgid 6.6 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 6.0 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 6.1 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 6.2 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 6.3 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T02:50:01.918794+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=50, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.967 INFO:tasks.ceph:pgid 6.11 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.10 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.13 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.12 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.1d last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.1c last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.1f last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.1e last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.968 INFO:tasks.ceph:pgid 6.19 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.969 INFO:tasks.ceph:pgid 6.18 last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.969 INFO:tasks.ceph:pgid 6.1b last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.969 INFO:tasks.ceph:pgid 6.1a last_scrub_stamp 2024-09-11T02:51:01.391757+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=2, tm_min=51, tm_sec=1, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=3, tm_min=8, tm_sec=5, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T03:08:09.969 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T03:08:29.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T03:08:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.099+0000 7f0948219640 10 monclient: get_monmap_and_config 2024-09-11T03:08:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.099+0000 7f0948219640 10 monclient: build_initial_monmap 2024-09-11T03:08:30.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.099+0000 7f0948219640 10 monclient: monmap: 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:30.104351+0000 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:30.104351+0000 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:30.102 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:30.103 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:30.103 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.099+0000 7f0948219640 1 Processor -- start 2024-09-11T03:08:30.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 1 -- start start 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient: init 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient: _add_conns ranks=[9,2,5,0,7,4,1,8,10,6,3] 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f09400a7aa0 0x7f09400a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): picked mon.noname-a con 0x7f09400a7aa0 addr v2:172.21.15.98:3300/0 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400a1db0 0x7f09400a83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): picked mon.noname-b con 0x7f09400a1db0 addr [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09400a15f0 0x7f0940139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): picked mon.noname-h con 0x7f09400a15f0 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7f09400a1a70 con 0x7f09400a7aa0 2024-09-11T03:08:30.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 1 -- --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_getmap magic: 0 -- 0x7f09400a8900 con 0x7f09400a1db0 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f09400a9730 con 0x7f09400a15f0 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient(hunting): authenticate will time out at 2063.984834s 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400a1db0 0x7f09400a83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f09400a7aa0 0x7f09400a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 --2- >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400a1db0 0x7f09400a83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3301/0 says I am v2:172.21.15.98:55798/0 (socket says 172.21.15.98:55798) 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 -- 172.21.15.98:0/4275705568 learned_addr learned my addr 172.21.15.98:0/4275705568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): get_auth_request con 0x7f09400a1db0 auth_method 0 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7f09400a7aa0 0x7f09400a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56412/0 (socket says 172.21.15.98:56412) 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 10 monclient(hunting): get_auth_request con 0x7f09400a7aa0 auth_method 0 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:30.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09467b6640 1 --2- 172.21.15.98:0/4275705568 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09400a15f0 0x7f0940139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient(hunting): handle_auth_done global_id 94484 payload 274 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 -- 172.21.15.98:0/4275705568 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09400a15f0 msgr2=0x7f0940139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 --2- 172.21.15.98:0/4275705568 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09400a15f0 0x7f0940139aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 -- 172.21.15.98:0/4275705568 >> v2:172.21.15.98:3300/0 conn(0x7f09400a7aa0 msgr2=0x7f09400a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 --2- 172.21.15.98:0/4275705568 >> v2:172.21.15.98:3300/0 conn(0x7f09400a7aa0 0x7f09400a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient: _finish_hunting 0 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 monclient: found mon.noname-b 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.98:3301/0 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 -- 172.21.15.98:0/4275705568 --> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09400a3ea0 con 0x7f09400a1db0 2024-09-11T03:08:30.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09467b6640 10 monclient: get_auth_request con 0x7f09400a15f0 auth_method 0 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0945fb5640 1 --2- 172.21.15.98:0/4275705568 >> v2:172.21.15.98:3300/0 conn(0x7f09400a7aa0 0x7f09400a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09467b6640 1 --2- 172.21.15.98:0/4275705568 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09400a15f0 0x7f0940139aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f09457b4640 1 --2- 172.21.15.98:0/4275705568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400a1db0 0x7f09400a83c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0930009870 tx=0x7f093001c1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da4cb28ad98c2d97 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 1 -- 172.21.15.98:0/4275705568 <== mon.2 v2:172.21.15.98:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09300b0070 con 0x7f09400a1db0 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: dump: 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.112 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: _finish_auth 0 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: _check_auth_tickets 2024-09-11T03:08:30.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 1 -- 172.21.15.98:0/4275705568 <== mon.2 v2:172.21.15.98:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09300aa810 con 0x7f09400a1db0 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 1 -- 172.21.15.98:0/4275705568 <== mon.2 v2:172.21.15.98:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09300a9300 con 0x7f09400a1db0 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 5 monclient: authenticate success, global_id 94484 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0948219640 4 set_mon_vals no callback set 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.103+0000 7f0944fb3640 10 monclient: dump: 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.114 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: shutdown 2024-09-11T03:08:30.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/4275705568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400a1db0 msgr2=0x7f09400a83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 --2- 172.21.15.98:0/4275705568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400a1db0 0x7f09400a83c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0930009870 tx=0x7f093001c1f0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/4275705568 shutdown_connections 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 --2- 172.21.15.98:0/4275705568 >> [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] conn(0x7f09400a1db0 0x7f09400a83c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 --2- 172.21.15.98:0/4275705568 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f09400a15f0 0x7f0940139aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 --2- 172.21.15.98:0/4275705568 >> v2:172.21.15.98:3300/0 conn(0x7f09400a7aa0 0x7f09400a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/4275705568 >> 172.21.15.98:0/4275705568 conn(0x7f09400ac850 msgr2=0x7f09400a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/4275705568 shutdown_connections 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/4275705568 wait complete. 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: build_initial_monmap 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: monmap: 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:30.116 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 librados: starting msgr at 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 librados: starting objecter 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 Processor -- start 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- start start 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 librados: setting wanted keys 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 librados: calling monclient init 2024-09-11T03:08:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: init 2024-09-11T03:08:30.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:30.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _add_conns ranks=[8,4,5,1,3,2,7,0,6] 2024-09-11T03:08:30.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09400a15f0 0x7f09401d1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): picked mon.noname-e con 0x7f09400a15f0 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:30.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 --2- >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09400a7aa0 0x7f0940137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): picked mon.noname-c con 0x7f09400a7aa0 addr [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] 2024-09-11T03:08:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 --2- >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0940137d50 0x7f094013cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): picked mon.noname-h con 0x7f0940137d50 addr [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] 2024-09-11T03:08:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- --> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] -- mon_getmap magic: 0 -- 0x7f0940143910 con 0x7f09400a7aa0 2024-09-11T03:08:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- --> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] -- mon_getmap magic: 0 -- 0x7f0940143790 con 0x7f0940137d50 2024-09-11T03:08:30.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7f0940143a90 con 0x7f09400a15f0 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient(hunting): authenticate will time out at 2063.990243s 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09400a15f0 0x7f09401d1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09400a15f0 0x7f09401d1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3304/0 says I am v2:172.21.15.98:35096/0 (socket says 172.21.15.98:35096) 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 -- 172.21.15.98:0/2553469617 learned_addr learned my addr 172.21.15.98:0/2553469617 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): get_auth_request con 0x7f09400a15f0 auth_method 0 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09400a7aa0 0x7f0940137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 10 monclient(hunting): get_auth_request con 0x7f09400a7aa0 auth_method 0 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09457b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient(hunting): handle_auth_done global_id 25586 payload 824 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 -- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0940137d50 msgr2=0x7f094013cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0940137d50 0x7f094013cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 -- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09400a7aa0 msgr2=0x7f0940137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09400a7aa0 0x7f0940137810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient: _finish_hunting 0 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 monclient: found mon.noname-e 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.98:3304/0 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 -- 172.21.15.98:0/2553469617 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09401381b0 con 0x7f09400a15f0 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0945fb5640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09400a15f0 0x7f09401d1a20 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e980 tx=0x7f093c00ee40 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 client.?.objecter ms_handle_connect 0x7f09400a15f0 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 1 -- 172.21.15.98:0/2553469617 <== mon.8 v2:172.21.15.98:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f093c017070 con 0x7f09400a15f0 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: dump: 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.128 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.129 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 1 -- 172.21.15.98:0/2553469617 <== mon.8 v2:172.21.15.98:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c00ce70 con 0x7f09400a15f0 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 5 monclient: authenticate success, global_id 25586 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _renew_subs 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0940122cf0 con 0x7f09400a15f0 2024-09-11T03:08:30.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:30.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 1 -- 172.21.15.98:0/2553469617 <== mon.8 v2:172.21.15.98:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f093c0138e0 con 0x7f09400a15f0 2024-09-11T03:08:30.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-09-11T03:08:30.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0937fff640 4 set_mon_vals no callback set 2024-09-11T03:08:30.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f09367fc640 10 monclient: dump: 2024-09-11T03:08:30.131 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.131 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.132 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 client.25586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _renew_subs 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0940122fd0 con 0x7f09400a15f0 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 librados: init done 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.98:3304/0 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.107+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f094013fe10 con 0x7f09400a15f0 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 1 -- 172.21.15.98:0/2553469617 <== mon.8 v2:172.21.15.98:3304/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7f093c013470 con 0x7f09400a15f0 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f092403df40 0x7f0924040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 1 -- 172.21.15.98:0/2553469617 <== mon.8 v2:172.21.15.98:3304/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7f093c00f040 con 0x7f09400a15f0 2024-09-11T03:08:30.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 10 client.25586.objecter ms_dispatch 0x7f0940000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 3 client.25586.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 3 client.25586.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 20 client.25586.objecter dump_active .. 0 homeless 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09457b4640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f092403df40 0x7f0924040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09457b4640 10 monclient: get_auth_request con 0x7f092403df40 auth_method 0 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09367fc640 10 client.25586.objecter ms_handle_connect 0x7f092403df40 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.111+0000 7f09457b4640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f092403df40 0x7f0924040400 secure :-1 s=READY pgs=3727 cs=0 l=1 rev1=1 crypto rx=0x7f09300ad000 tx=0x7f09300a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.115+0000 7f09367fc640 1 -- 172.21.15.98:0/2553469617 <== mon.8 v2:172.21.15.98:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7f093c021070 con 0x7f09400a15f0 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.115+0000 7f09367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:30.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.115+0000 7f09367fc640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:30.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.283+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 --> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f094013ed40 con 0x7f092403df40 2024-09-11T03:08:30.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f09367fc640 1 -- 172.21.15.98:0/2553469617 <== mgr.4105 v2:172.21.15.190:6824/3356844748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114061 (secure 0 0 0) 0x7f094013ed40 con 0x7f092403df40 2024-09-11T03:08:30.292 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-11T03:08:30.293 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-11T03:08:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 10 librados: watch_flush enter 2024-09-11T03:08:30.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 10 librados: watch_flush exit 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 20 client.25586.objecter shutdown clearing up homeless session... 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 10 client.25586.objecter successfully canceled tick 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f092403df40 msgr2=0x7f0924040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f092403df40 0x7f0924040400 secure :-1 s=READY pgs=3727 cs=0 l=1 rev1=1 crypto rx=0x7f09300ad000 tx=0x7f09300a6000 comp rx=0 tx=0).stop 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 10 monclient: shutdown 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09400a15f0 msgr2=0x7f09401d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09400a15f0 0x7f09401d1a20 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f093c00e980 tx=0x7f093c00ee40 comp rx=0 tx=0).stop 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 shutdown_connections 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7f092403df40 0x7f0924040400 unknown :-1 s=CLOSED pgs=3727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] conn(0x7f0940137d50 0x7f094013cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] conn(0x7f09400a7aa0 0x7f0940137810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.291+0000 7f0948219640 1 --2- 172.21.15.98:0/2553469617 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7f09400a15f0 0x7f09401d1a20 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.295+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 >> 172.21.15.98:0/2553469617 conn(0x7f09400ac850 msgr2=0x7f0940140020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.295+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 shutdown_connections 2024-09-11T03:08:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.295+0000 7f0948219640 1 -- 172.21.15.98:0/2553469617 wait complete. 2024-09-11T03:08:30.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.295+0000 7f0948219640 1 librados: shutdown 2024-09-11T03:08:30.311 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1252,"stamp":"2024-09-11T03:08:30.142780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":451712,"kb_used_data":61920,"kb_used_omap":9,"kb_used_meta":389686,"kb_avail":565779328,"statfs":{"total":579820584960,"available":579358031872,"internally_reserved":0,"allocated":63406080,"data_stored":35633462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9566,"internal_metadata":399039138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003950"},"pg_stats":[{"pgid":"6.16","version":"0'0","reported_seq":1370,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:10.643230+0000","last_change":"2024-09-11T03:08:10.643230+0000","last_active":"2024-09-11T03:08:10.643230+0000","last_peered":"2024-09-11T03:08:10.643230+0000","last_clean":"2024-09-11T03:08:10.643230+0000","last_became_active":"2024-09-11T02:51:02.704466+0000","last_became_peered":"2024-09-11T02:51:02.704466+0000","last_unstale":"2024-09-11T03:08:10.643230+0000","last_undegraded":"2024-09-11T03:08:10.643230+0000","last_fullsized":"2024-09-11T03:08:10.643230+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:10.643065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:10.643065+0000","last_clean_scrub_stamp":"2024-09-11T03:08:10.643065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:48:55.544977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:10.121383+0000","last_change":"2024-09-11T03:08:10.121383+0000","last_active":"2024-09-11T03:08:10.121383+0000","last_peered":"2024-09-11T03:08:10.121383+0000","last_clean":"2024-09-11T03:08:10.121383+0000","last_became_active":"2024-09-11T02:51:02.747449+0000","last_became_peered":"2024-09-11T02:51:02.747449+0000","last_unstale":"2024-09-11T03:08:10.121383+0000","last_undegraded":"2024-09-11T03:08:10.121383+0000","last_fullsized":"2024-09-11T03:08:10.121383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:10.121225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:10.121225+0000","last_clean_scrub_stamp":"2024-09-11T03:08:10.121225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:14:07.238944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1364,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:12.123475+0000","last_change":"2024-09-11T03:08:12.123475+0000","last_active":"2024-09-11T03:08:12.123475+0000","last_peered":"2024-09-11T03:08:12.123475+0000","last_clean":"2024-09-11T03:08:12.123475+0000","last_became_active":"2024-09-11T02:51:02.746704+0000","last_became_peered":"2024-09-11T02:51:02.746704+0000","last_unstale":"2024-09-11T03:08:12.123475+0000","last_undegraded":"2024-09-11T03:08:12.123475+0000","last_fullsized":"2024-09-11T03:08:12.123475+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:12.123223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:12.123223+0000","last_clean_scrub_stamp":"2024-09-11T03:08:12.123223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:13:35.822677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1368,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:15.845312+0000","last_change":"2024-09-11T03:08:15.845312+0000","last_active":"2024-09-11T03:08:15.845312+0000","last_peered":"2024-09-11T03:08:15.845312+0000","last_clean":"2024-09-11T03:08:15.845312+0000","last_became_active":"2024-09-11T02:51:02.695884+0000","last_became_peered":"2024-09-11T02:51:02.695884+0000","last_unstale":"2024-09-11T03:08:15.845312+0000","last_undegraded":"2024-09-11T03:08:15.845312+0000","last_fullsized":"2024-09-11T03:08:15.845312+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:15.845147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:15.845147+0000","last_clean_scrub_stamp":"2024-09-11T03:08:15.845147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:53.612349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1363,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:15.282448+0000","last_change":"2024-09-11T03:08:15.282323+0000","last_active":"2024-09-11T03:08:15.282448+0000","last_peered":"2024-09-11T03:08:15.282448+0000","last_clean":"2024-09-11T03:08:15.282448+0000","last_became_active":"2024-09-11T02:51:02.604043+0000","last_became_peered":"2024-09-11T02:51:02.604043+0000","last_unstale":"2024-09-11T03:08:15.282448+0000","last_undegraded":"2024-09-11T03:08:15.282448+0000","last_fullsized":"2024-09-11T03:08:15.282448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:15.282164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:15.282164+0000","last_clean_scrub_stamp":"2024-09-11T03:08:15.282164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:58:13.274485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:11.060719+0000","last_change":"2024-09-11T03:08:11.060719+0000","last_active":"2024-09-11T03:08:11.060719+0000","last_peered":"2024-09-11T03:08:11.060719+0000","last_clean":"2024-09-11T03:08:11.060719+0000","last_became_active":"2024-09-11T02:51:02.574238+0000","last_became_peered":"2024-09-11T02:51:02.574238+0000","last_unstale":"2024-09-11T03:08:11.060719+0000","last_undegraded":"2024-09-11T03:08:11.060719+0000","last_fullsized":"2024-09-11T03:08:11.060719+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:11.060559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:11.060559+0000","last_clean_scrub_stamp":"2024-09-11T03:08:11.060559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:45:32.159868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1369,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:14.813615+0000","last_change":"2024-09-11T03:08:14.813378+0000","last_active":"2024-09-11T03:08:14.813615+0000","last_peered":"2024-09-11T03:08:14.813615+0000","last_clean":"2024-09-11T03:08:14.813615+0000","last_became_active":"2024-09-11T02:51:02.745188+0000","last_became_peered":"2024-09-11T02:51:02.745188+0000","last_unstale":"2024-09-11T03:08:14.813615+0000","last_undegraded":"2024-09-11T03:08:14.813615+0000","last_fullsized":"2024-09-11T03:08:14.813615+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:14.813132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:14.813132+0000","last_clean_scrub_stamp":"2024-09-11T03:08:14.813132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:10:12.873474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1371,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:07.606476+0000","last_change":"2024-09-11T03:08:07.606388+0000","last_active":"2024-09-11T03:08:07.606476+0000","last_peered":"2024-09-11T03:08:07.606476+0000","last_clean":"2024-09-11T03:08:07.606476+0000","last_became_active":"2024-09-11T02:51:02.578172+0000","last_became_peered":"2024-09-11T02:51:02.578172+0000","last_unstale":"2024-09-11T03:08:07.606476+0000","last_undegraded":"2024-09-11T03:08:07.606476+0000","last_fullsized":"2024-09-11T03:08:07.606476+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:07.606274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:07.606274+0000","last_clean_scrub_stamp":"2024-09-11T03:08:07.606274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:57:36.958142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:11.081655+0000","last_change":"2024-09-11T03:08:11.081655+0000","last_active":"2024-09-11T03:08:11.081655+0000","last_peered":"2024-09-11T03:08:11.081655+0000","last_clean":"2024-09-11T03:08:11.081655+0000","last_became_active":"2024-09-11T02:51:02.666973+0000","last_became_peered":"2024-09-11T02:51:02.666973+0000","last_unstale":"2024-09-11T03:08:11.081655+0000","last_undegraded":"2024-09-11T03:08:11.081655+0000","last_fullsized":"2024-09-11T03:08:11.081655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:11.081496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:11.081496+0000","last_clean_scrub_stamp":"2024-09-11T03:08:11.081496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:31:44.343184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1364,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:11.129303+0000","last_change":"2024-09-11T03:08:11.129303+0000","last_active":"2024-09-11T03:08:11.129303+0000","last_peered":"2024-09-11T03:08:11.129303+0000","last_clean":"2024-09-11T03:08:11.129303+0000","last_became_active":"2024-09-11T02:51:02.737993+0000","last_became_peered":"2024-09-11T02:51:02.737993+0000","last_unstale":"2024-09-11T03:08:11.129303+0000","last_undegraded":"2024-09-11T03:08:11.129303+0000","last_fullsized":"2024-09-11T03:08:11.129303+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:11.129043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:11.129043+0000","last_clean_scrub_stamp":"2024-09-11T03:08:11.129043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:39:33.437112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1359,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:15.119644+0000","last_change":"2024-09-11T03:08:15.119644+0000","last_active":"2024-09-11T03:08:15.119644+0000","last_peered":"2024-09-11T03:08:15.119644+0000","last_clean":"2024-09-11T03:08:15.119644+0000","last_became_active":"2024-09-11T02:51:02.671301+0000","last_became_peered":"2024-09-11T02:51:02.671301+0000","last_unstale":"2024-09-11T03:08:15.119644+0000","last_undegraded":"2024-09-11T03:08:15.119644+0000","last_fullsized":"2024-09-11T03:08:15.119644+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:15.119458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:15.119458+0000","last_clean_scrub_stamp":"2024-09-11T03:08:15.119458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:41:27.648655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:13.206748+0000","last_change":"2024-09-11T03:08:13.206748+0000","last_active":"2024-09-11T03:08:13.206748+0000","last_peered":"2024-09-11T03:08:13.206748+0000","last_clean":"2024-09-11T03:08:13.206748+0000","last_became_active":"2024-09-11T02:51:02.712319+0000","last_became_peered":"2024-09-11T02:51:02.712319+0000","last_unstale":"2024-09-11T03:08:13.206748+0000","last_undegraded":"2024-09-11T03:08:13.206748+0000","last_fullsized":"2024-09-11T03:08:13.206748+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:13.206559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:13.206559+0000","last_clean_scrub_stamp":"2024-09-11T03:08:13.206559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:06.219128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:14.237380+0000","last_change":"2024-09-11T03:08:14.237140+0000","last_active":"2024-09-11T03:08:14.237380+0000","last_peered":"2024-09-11T03:08:14.237380+0000","last_clean":"2024-09-11T03:08:14.237380+0000","last_became_active":"2024-09-11T02:51:02.672559+0000","last_became_peered":"2024-09-11T02:51:02.672559+0000","last_unstale":"2024-09-11T03:08:14.237380+0000","last_undegraded":"2024-09-11T03:08:14.237380+0000","last_fullsized":"2024-09-11T03:08:14.237380+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:14.236956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:14.236956+0000","last_clean_scrub_stamp":"2024-09-11T03:08:14.236956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:29:29.208413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"699'74","reported_seq":1478,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:10.786652+0000","last_change":"2024-09-11T03:08:10.786652+0000","last_active":"2024-09-11T03:08:10.786652+0000","last_peered":"2024-09-11T03:08:10.786652+0000","last_clean":"2024-09-11T03:08:10.786652+0000","last_became_active":"2024-09-11T02:49:56.897388+0000","last_became_peered":"2024-09-11T02:49:56.897388+0000","last_unstale":"2024-09-11T03:08:10.786652+0000","last_undegraded":"2024-09-11T03:08:10.786652+0000","last_fullsized":"2024-09-11T03:08:10.786652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'74","last_scrub_stamp":"2024-09-11T03:08:10.786486+0000","last_deep_scrub":"699'74","last_deep_scrub_stamp":"2024-09-11T03:08:10.786486+0000","last_clean_scrub_stamp":"2024-09-11T03:08:10.786486+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:26.828396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"699'2","reported_seq":1380,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:11.134692+0000","last_change":"2024-09-11T03:08:11.134692+0000","last_active":"2024-09-11T03:08:11.134692+0000","last_peered":"2024-09-11T03:08:11.134692+0000","last_clean":"2024-09-11T03:08:11.134692+0000","last_became_active":"2024-09-11T02:50:02.964067+0000","last_became_peered":"2024-09-11T02:50:02.964067+0000","last_unstale":"2024-09-11T03:08:11.134692+0000","last_undegraded":"2024-09-11T03:08:11.134692+0000","last_fullsized":"2024-09-11T03:08:11.134692+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'2","last_scrub_stamp":"2024-09-11T03:08:11.134481+0000","last_deep_scrub":"699'2","last_deep_scrub_stamp":"2024-09-11T03:08:11.134481+0000","last_clean_scrub_stamp":"2024-09-11T03:08:11.134481+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:29:15.256828+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001057915,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:14.099814+0000","last_change":"2024-09-11T03:08:14.099814+0000","last_active":"2024-09-11T03:08:14.099814+0000","last_peered":"2024-09-11T03:08:14.099814+0000","last_clean":"2024-09-11T03:08:14.099814+0000","last_became_active":"2024-09-11T02:51:02.709268+0000","last_became_peered":"2024-09-11T02:51:02.709268+0000","last_unstale":"2024-09-11T03:08:14.099814+0000","last_undegraded":"2024-09-11T03:08:14.099814+0000","last_fullsized":"2024-09-11T03:08:14.099814+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:14.099655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:14.099655+0000","last_clean_scrub_stamp":"2024-09-11T03:08:14.099655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:05:54.967114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1385,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:13.813910+0000","last_change":"2024-09-11T03:08:13.813910+0000","last_active":"2024-09-11T03:08:13.813910+0000","last_peered":"2024-09-11T03:08:13.813910+0000","last_clean":"2024-09-11T03:08:13.813910+0000","last_became_active":"2024-09-11T02:50:02.957499+0000","last_became_peered":"2024-09-11T02:50:02.957499+0000","last_unstale":"2024-09-11T03:08:13.813910+0000","last_undegraded":"2024-09-11T03:08:13.813910+0000","last_fullsized":"2024-09-11T03:08:13.813910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:13.813709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:13.813709+0000","last_clean_scrub_stamp":"2024-09-11T03:08:13.813709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:51.958950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095282199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1363,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:12.158869+0000","last_change":"2024-09-11T03:08:12.158869+0000","last_active":"2024-09-11T03:08:12.158869+0000","last_peered":"2024-09-11T03:08:12.158869+0000","last_clean":"2024-09-11T03:08:12.158869+0000","last_became_active":"2024-09-11T02:51:02.587630+0000","last_became_peered":"2024-09-11T02:51:02.587630+0000","last_unstale":"2024-09-11T03:08:12.158869+0000","last_undegraded":"2024-09-11T03:08:12.158869+0000","last_fullsized":"2024-09-11T03:08:12.158869+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:12.158679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:12.158679+0000","last_clean_scrub_stamp":"2024-09-11T03:08:12.158679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:22:15.544989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1380,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:09.123096+0000","last_change":"2024-09-11T03:08:09.123096+0000","last_active":"2024-09-11T03:08:09.123096+0000","last_peered":"2024-09-11T03:08:09.123096+0000","last_clean":"2024-09-11T03:08:09.123096+0000","last_became_active":"2024-09-11T02:50:02.959140+0000","last_became_peered":"2024-09-11T02:50:02.959140+0000","last_unstale":"2024-09-11T03:08:09.123096+0000","last_undegraded":"2024-09-11T03:08:09.123096+0000","last_fullsized":"2024-09-11T03:08:09.123096+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:09.122941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:09.122941+0000","last_clean_scrub_stamp":"2024-09-11T03:08:09.122941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:18:17.820661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001326516,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1360,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:13.051118+0000","last_change":"2024-09-11T03:08:13.050985+0000","last_active":"2024-09-11T03:08:13.051118+0000","last_peered":"2024-09-11T03:08:13.051118+0000","last_clean":"2024-09-11T03:08:13.051118+0000","last_became_active":"2024-09-11T02:51:02.679438+0000","last_became_peered":"2024-09-11T02:51:02.679438+0000","last_unstale":"2024-09-11T03:08:13.051118+0000","last_undegraded":"2024-09-11T03:08:13.051118+0000","last_fullsized":"2024-09-11T03:08:13.051118+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:13.050824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:13.050824+0000","last_clean_scrub_stamp":"2024-09-11T03:08:13.050824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:43.850001+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"699'3","reported_seq":1381,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:10.150045+0000","last_change":"2024-09-11T03:08:10.150045+0000","last_active":"2024-09-11T03:08:10.150045+0000","last_peered":"2024-09-11T03:08:10.150045+0000","last_clean":"2024-09-11T03:08:10.150045+0000","last_became_active":"2024-09-11T02:50:02.964151+0000","last_became_peered":"2024-09-11T02:50:02.964151+0000","last_unstale":"2024-09-11T03:08:10.150045+0000","last_undegraded":"2024-09-11T03:08:10.150045+0000","last_fullsized":"2024-09-11T03:08:10.150045+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'3","last_scrub_stamp":"2024-09-11T03:08:10.149862+0000","last_deep_scrub":"699'3","last_deep_scrub_stamp":"2024-09-11T03:08:10.149862+0000","last_clean_scrub_stamp":"2024-09-11T03:08:10.149862+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:10:37.713933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00130167,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1371,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:08.609497+0000","last_change":"2024-09-11T03:08:08.609497+0000","last_active":"2024-09-11T03:08:08.609497+0000","last_peered":"2024-09-11T03:08:08.609497+0000","last_clean":"2024-09-11T03:08:08.609497+0000","last_became_active":"2024-09-11T02:51:02.655843+0000","last_became_peered":"2024-09-11T02:51:02.655843+0000","last_unstale":"2024-09-11T03:08:08.609497+0000","last_undegraded":"2024-09-11T03:08:08.609497+0000","last_fullsized":"2024-09-11T03:08:08.609497+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:08.609399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:08.609399+0000","last_clean_scrub_stamp":"2024-09-11T03:08:08.609399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:32:39.451851+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1375,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:10.091112+0000","last_change":"2024-09-11T03:08:10.091112+0000","last_active":"2024-09-11T03:08:10.091112+0000","last_peered":"2024-09-11T03:08:10.091112+0000","last_clean":"2024-09-11T03:08:10.091112+0000","last_became_active":"2024-09-11T02:50:02.955909+0000","last_became_peered":"2024-09-11T02:50:02.955909+0000","last_unstale":"2024-09-11T03:08:10.091112+0000","last_undegraded":"2024-09-11T03:08:10.091112+0000","last_fullsized":"2024-09-11T03:08:10.091112+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:10.090968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:10.090968+0000","last_clean_scrub_stamp":"2024-09-11T03:08:10.090968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:23:39.152151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0042237200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"699'8","reported_seq":1374,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:08.201443+0000","last_change":"2024-09-11T03:08:08.201443+0000","last_active":"2024-09-11T03:08:08.201443+0000","last_peered":"2024-09-11T03:08:08.201443+0000","last_clean":"2024-09-11T03:08:08.201443+0000","last_became_active":"2024-09-11T02:51:02.586208+0000","last_became_peered":"2024-09-11T02:51:02.586208+0000","last_unstale":"2024-09-11T03:08:08.201443+0000","last_undegraded":"2024-09-11T03:08:08.201443+0000","last_fullsized":"2024-09-11T03:08:08.201443+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"699'8","last_scrub_stamp":"2024-09-11T03:08:08.201320+0000","last_deep_scrub":"699'8","last_deep_scrub_stamp":"2024-09-11T03:08:08.201320+0000","last_clean_scrub_stamp":"2024-09-11T03:08:08.201320+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:30:01.194466+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1384,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:11.795318+0000","last_change":"2024-09-11T03:08:11.795318+0000","last_active":"2024-09-11T03:08:11.795318+0000","last_peered":"2024-09-11T03:08:11.795318+0000","last_clean":"2024-09-11T03:08:11.795318+0000","last_became_active":"2024-09-11T02:50:02.956646+0000","last_became_peered":"2024-09-11T02:50:02.956646+0000","last_unstale":"2024-09-11T03:08:11.795318+0000","last_undegraded":"2024-09-11T03:08:11.795318+0000","last_fullsized":"2024-09-11T03:08:11.795318+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:11.795159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:11.795159+0000","last_clean_scrub_stamp":"2024-09-11T03:08:11.795159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:05:27.003679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001158601,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1369,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:12.776785+0000","last_change":"2024-09-11T03:08:12.776785+0000","last_active":"2024-09-11T03:08:12.776785+0000","last_peered":"2024-09-11T03:08:12.776785+0000","last_clean":"2024-09-11T03:08:12.776785+0000","last_became_active":"2024-09-11T02:51:02.642031+0000","last_became_peered":"2024-09-11T02:51:02.642031+0000","last_unstale":"2024-09-11T03:08:12.776785+0000","last_undegraded":"2024-09-11T03:08:12.776785+0000","last_fullsized":"2024-09-11T03:08:12.776785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:12.776630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:12.776630+0000","last_clean_scrub_stamp":"2024-09-11T03:08:12.776630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:28:22.562377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1375,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:09.138099+0000","last_change":"2024-09-11T03:08:09.138099+0000","last_active":"2024-09-11T03:08:09.138099+0000","last_peered":"2024-09-11T03:08:09.138099+0000","last_clean":"2024-09-11T03:08:09.138099+0000","last_became_active":"2024-09-11T02:50:02.956290+0000","last_became_peered":"2024-09-11T02:50:02.956290+0000","last_unstale":"2024-09-11T03:08:09.138099+0000","last_undegraded":"2024-09-11T03:08:09.138099+0000","last_fullsized":"2024-09-11T03:08:09.138099+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:09.137936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:09.137936+0000","last_clean_scrub_stamp":"2024-09-11T03:08:09.137936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:25:14.372146+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0036788979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1368,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:13.075116+0000","last_change":"2024-09-11T03:08:13.075116+0000","last_active":"2024-09-11T03:08:13.075116+0000","last_peered":"2024-09-11T03:08:13.075116+0000","last_clean":"2024-09-11T03:08:13.075116+0000","last_became_active":"2024-09-11T02:51:02.748048+0000","last_became_peered":"2024-09-11T02:51:02.748048+0000","last_unstale":"2024-09-11T03:08:13.075116+0000","last_undegraded":"2024-09-11T03:08:13.075116+0000","last_fullsized":"2024-09-11T03:08:13.075116+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:13.075010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:13.075010+0000","last_clean_scrub_stamp":"2024-09-11T03:08:13.075010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:26.410213+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1384,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:09.758076+0000","last_change":"2024-09-11T03:08:09.758076+0000","last_active":"2024-09-11T03:08:09.758076+0000","last_peered":"2024-09-11T03:08:09.758076+0000","last_clean":"2024-09-11T03:08:09.758076+0000","last_became_active":"2024-09-11T02:50:02.959395+0000","last_became_peered":"2024-09-11T02:50:02.959395+0000","last_unstale":"2024-09-11T03:08:09.758076+0000","last_undegraded":"2024-09-11T03:08:09.758076+0000","last_fullsized":"2024-09-11T03:08:09.758076+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:09.757918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:09.757918+0000","last_clean_scrub_stamp":"2024-09-11T03:08:09.757918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:26:41.892314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014774479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1364,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:10.085884+0000","last_change":"2024-09-11T03:08:10.085884+0000","last_active":"2024-09-11T03:08:10.085884+0000","last_peered":"2024-09-11T03:08:10.085884+0000","last_clean":"2024-09-11T03:08:10.085884+0000","last_became_active":"2024-09-11T02:51:02.653755+0000","last_became_peered":"2024-09-11T02:51:02.653755+0000","last_unstale":"2024-09-11T03:08:10.085884+0000","last_undegraded":"2024-09-11T03:08:10.085884+0000","last_fullsized":"2024-09-11T03:08:10.085884+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:10.085730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:10.085730+0000","last_clean_scrub_stamp":"2024-09-11T03:08:10.085730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:07:27.760043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1370,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:09.625151+0000","last_change":"2024-09-11T03:08:09.625151+0000","last_active":"2024-09-11T03:08:09.625151+0000","last_peered":"2024-09-11T03:08:09.625151+0000","last_clean":"2024-09-11T03:08:09.625151+0000","last_became_active":"2024-09-11T02:51:02.594230+0000","last_became_peered":"2024-09-11T02:51:02.594230+0000","last_unstale":"2024-09-11T03:08:09.625151+0000","last_undegraded":"2024-09-11T03:08:09.625151+0000","last_fullsized":"2024-09-11T03:08:09.625151+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:09.625062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:09.625062+0000","last_clean_scrub_stamp":"2024-09-11T03:08:09.625062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:16:31.449844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1364,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:13.141689+0000","last_change":"2024-09-11T03:08:13.141689+0000","last_active":"2024-09-11T03:08:13.141689+0000","last_peered":"2024-09-11T03:08:13.141689+0000","last_clean":"2024-09-11T03:08:13.141689+0000","last_became_active":"2024-09-11T02:51:02.736590+0000","last_became_peered":"2024-09-11T02:51:02.736590+0000","last_unstale":"2024-09-11T03:08:13.141689+0000","last_undegraded":"2024-09-11T03:08:13.141689+0000","last_fullsized":"2024-09-11T03:08:13.141689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:13.141529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:13.141529+0000","last_clean_scrub_stamp":"2024-09-11T03:08:13.141529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:49:52.760890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:16.310095+0000","last_change":"2024-09-11T03:08:16.310095+0000","last_active":"2024-09-11T03:08:16.310095+0000","last_peered":"2024-09-11T03:08:16.310095+0000","last_clean":"2024-09-11T03:08:16.310095+0000","last_became_active":"2024-09-11T02:51:02.678439+0000","last_became_peered":"2024-09-11T02:51:02.678439+0000","last_unstale":"2024-09-11T03:08:16.310095+0000","last_undegraded":"2024-09-11T03:08:16.310095+0000","last_fullsized":"2024-09-11T03:08:16.310095+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:16.309926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:16.309926+0000","last_clean_scrub_stamp":"2024-09-11T03:08:16.309926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:38:32.245875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1358,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:12.050702+0000","last_change":"2024-09-11T03:08:12.050702+0000","last_active":"2024-09-11T03:08:12.050702+0000","last_peered":"2024-09-11T03:08:12.050702+0000","last_clean":"2024-09-11T03:08:12.050702+0000","last_became_active":"2024-09-11T02:51:02.700809+0000","last_became_peered":"2024-09-11T02:51:02.700809+0000","last_unstale":"2024-09-11T03:08:12.050702+0000","last_undegraded":"2024-09-11T03:08:12.050702+0000","last_fullsized":"2024-09-11T03:08:12.050702+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:12.050543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:12.050543+0000","last_clean_scrub_stamp":"2024-09-11T03:08:12.050543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:07:59.833902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1367,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:16.827408+0000","last_change":"2024-09-11T03:08:16.827408+0000","last_active":"2024-09-11T03:08:16.827408+0000","last_peered":"2024-09-11T03:08:16.827408+0000","last_clean":"2024-09-11T03:08:16.827408+0000","last_became_active":"2024-09-11T02:51:02.695758+0000","last_became_peered":"2024-09-11T02:51:02.695758+0000","last_unstale":"2024-09-11T03:08:16.827408+0000","last_undegraded":"2024-09-11T03:08:16.827408+0000","last_fullsized":"2024-09-11T03:08:16.827408+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:16.827230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:16.827230+0000","last_clean_scrub_stamp":"2024-09-11T03:08:16.827230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:39.679805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1365,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:14.123597+0000","last_change":"2024-09-11T03:08:14.123597+0000","last_active":"2024-09-11T03:08:14.123597+0000","last_peered":"2024-09-11T03:08:14.123597+0000","last_clean":"2024-09-11T03:08:14.123597+0000","last_became_active":"2024-09-11T02:51:02.715041+0000","last_became_peered":"2024-09-11T02:51:02.715041+0000","last_unstale":"2024-09-11T03:08:14.123597+0000","last_undegraded":"2024-09-11T03:08:14.123597+0000","last_fullsized":"2024-09-11T03:08:14.123597+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:14.123411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:14.123411+0000","last_clean_scrub_stamp":"2024-09-11T03:08:14.123411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:58:33.576464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1369,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:17.815834+0000","last_change":"2024-09-11T03:08:17.815834+0000","last_active":"2024-09-11T03:08:17.815834+0000","last_peered":"2024-09-11T03:08:17.815834+0000","last_clean":"2024-09-11T03:08:17.815834+0000","last_became_active":"2024-09-11T02:51:02.581916+0000","last_became_peered":"2024-09-11T02:51:02.581916+0000","last_unstale":"2024-09-11T03:08:17.815834+0000","last_undegraded":"2024-09-11T03:08:17.815834+0000","last_fullsized":"2024-09-11T03:08:17.815834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:17.815673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:17.815673+0000","last_clean_scrub_stamp":"2024-09-11T03:08:17.815673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:49:32.486735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1362,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:17.346145+0000","last_change":"2024-09-11T03:08:17.346145+0000","last_active":"2024-09-11T03:08:17.346145+0000","last_peered":"2024-09-11T03:08:17.346145+0000","last_clean":"2024-09-11T03:08:17.346145+0000","last_became_active":"2024-09-11T02:51:02.708034+0000","last_became_peered":"2024-09-11T02:51:02.708034+0000","last_unstale":"2024-09-11T03:08:17.346145+0000","last_undegraded":"2024-09-11T03:08:17.346145+0000","last_fullsized":"2024-09-11T03:08:17.346145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:17.345954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:17.345954+0000","last_clean_scrub_stamp":"2024-09-11T03:08:17.345954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:05:04.015721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1370,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:11.672006+0000","last_change":"2024-09-11T03:08:11.672006+0000","last_active":"2024-09-11T03:08:11.672006+0000","last_peered":"2024-09-11T03:08:11.672006+0000","last_clean":"2024-09-11T03:08:11.672006+0000","last_became_active":"2024-09-11T02:51:02.684472+0000","last_became_peered":"2024-09-11T02:51:02.684472+0000","last_unstale":"2024-09-11T03:08:11.672006+0000","last_undegraded":"2024-09-11T03:08:11.672006+0000","last_fullsized":"2024-09-11T03:08:11.672006+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:11.671800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:11.671800+0000","last_clean_scrub_stamp":"2024-09-11T03:08:11.671800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:15:45.245146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1369,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:12.685378+0000","last_change":"2024-09-11T03:08:12.685378+0000","last_active":"2024-09-11T03:08:12.685378+0000","last_peered":"2024-09-11T03:08:12.685378+0000","last_clean":"2024-09-11T03:08:12.685378+0000","last_became_active":"2024-09-11T02:51:02.603633+0000","last_became_peered":"2024-09-11T02:51:02.603633+0000","last_unstale":"2024-09-11T03:08:12.685378+0000","last_undegraded":"2024-09-11T03:08:12.685378+0000","last_fullsized":"2024-09-11T03:08:12.685378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:12.685196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:12.685196+0000","last_clean_scrub_stamp":"2024-09-11T03:08:12.685196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:11:37.878093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1366,"reported_epoch":699,"state":"active+clean","last_fresh":"2024-09-11T03:08:12.065873+0000","last_change":"2024-09-11T03:08:12.065873+0000","last_active":"2024-09-11T03:08:12.065873+0000","last_peered":"2024-09-11T03:08:12.065873+0000","last_clean":"2024-09-11T03:08:12.065873+0000","last_became_active":"2024-09-11T02:51:02.659063+0000","last_became_peered":"2024-09-11T02:51:02.659063+0000","last_unstale":"2024-09-11T03:08:12.065873+0000","last_undegraded":"2024-09-11T03:08:12.065873+0000","last_fullsized":"2024-09-11T03:08:12.065873+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T03:08:12.065714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T03:08:12.065714+0000","last_clean_scrub_stamp":"2024-09-11T03:08:12.065714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:39.929162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607779,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77408,"kb_used_data":10128,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294432,"statfs":{"total":96636764160,"available":96557498368,"internally_reserved":0,"allocated":10371072,"data_stored":5742132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607777,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77404,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294436,"statfs":{"total":96636764160,"available":96557502464,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607780,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77988,"kb_used_data":10708,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94293852,"statfs":{"total":96636764160,"available":96556904448,"internally_reserved":0,"allocated":10964992,"data_stored":6332486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607780,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85724,"kb_used_data":10124,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94286116,"statfs":{"total":96636764160,"available":96548982784,"internally_reserved":0,"allocated":10366976,"data_stored":5742113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640482,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77412,"kb_used_data":10132,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94294428,"statfs":{"total":96636764160,"available":96557494272,"internally_reserved":0,"allocated":10375168,"data_stored":5742137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640483,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55776,"kb_used_data":10704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":10960896,"data_stored":6332481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T03:08:30.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T03:08:30.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.435+0000 7fae7dabb640 10 monclient: get_monmap_and_config 2024-09-11T03:08:30.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.435+0000 7fae7dabb640 10 monclient: build_initial_monmap 2024-09-11T03:08:30.437 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.435+0000 7fae7dabb640 10 monclient: monmap: 2024-09-11T03:08:30.437 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:30.437 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.437 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T03:08:30.439875+0000 2024-09-11T03:08:30.438 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T03:08:30.439875+0000 2024-09-11T03:08:30.438 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:30.438 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.438 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.98:6789/0 mon.noname-a-legacy 2024-09-11T03:08:30.438 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.190:6789/0 mon.noname-f-legacy 2024-09-11T03:08:30.438 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:9: v2:172.21.15.98:3300/0 mon.noname-a 2024-09-11T03:08:30.440 INFO:teuthology.orchestra.run.smithi098.stderr:10: v2:172.21.15.190:3300/0 mon.noname-f 2024-09-11T03:08:30.441 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.435+0000 7fae7dabb640 1 Processor -- start 2024-09-11T03:08:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.435+0000 7fae7dabb640 1 -- start start 2024-09-11T03:08:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.435+0000 7fae7dabb640 10 monclient: init 2024-09-11T03:08:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient: _add_conns ranks=[7,3,9,10,1,5,4,6,2,0,8] 2024-09-11T03:08:30.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 --2- >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fae78139b10 0x7fae780725d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): picked mon.noname-i con 0x7fae78139b10 addr [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] 2024-09-11T03:08:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae7806fe30 0x7fae78139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): picked mon.noname-g con 0x7fae7806fe30 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae780a7a10 0x7fae780a7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): picked mon.noname-a con 0x7fae780a7a10 addr v2:172.21.15.98:3300/0 2024-09-11T03:08:30.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 -- --> v2:172.21.15.98:3300/0 -- mon_getmap magic: 0 -- 0x7fae780a9650 con 0x7fae780a7a10 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fae78071350 con 0x7fae7806fe30 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 -- --> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] -- mon_getmap magic: 0 -- 0x7fae78071630 con 0x7fae78139b10 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient(hunting): authenticate will time out at 2064.320142s 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae780a7a10 0x7fae780a7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 --2- >> v2:172.21.15.98:3300/0 conn(0x7fae780a7a10 0x7fae780a7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56424/0 (socket says 172.21.15.98:56424) 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 -- 172.21.15.98:0/831397561 learned_addr learned my addr 172.21.15.98:0/831397561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): get_auth_request con 0x7fae780a7a10 auth_method 0 2024-09-11T03:08:30.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae76ffd640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae7806fe30 0x7fae78139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae777fe640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fae78139b10 0x7fae780725d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae76ffd640 10 monclient(hunting): get_auth_request con 0x7fae7806fe30 auth_method 0 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae777fe640 10 monclient(hunting): get_auth_request con 0x7fae78139b10 auth_method 0 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient(hunting): handle_auth_done global_id 104373 payload 274 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 -- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fae78139b10 msgr2=0x7fae780725d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fae78139b10 0x7fae780725d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 -- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae7806fe30 msgr2=0x7fae78139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-11T03:08:30.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae7806fe30 0x7fae78139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient: _finish_hunting 0 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 monclient: found mon.noname-a 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 -- 172.21.15.98:0/831397561 --> v2:172.21.15.98:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae780702b0 con 0x7fae780a7a10 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae76ffd640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae7806fe30 0x7fae78139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae77fff640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a7ea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae70009e50 tx=0x7fae7001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d30c280c3a25703 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:30.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 1 -- 172.21.15.98:0/831397561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae700090d0 con 0x7fae780a7a10 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: dump: 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.458 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: _finish_auth 0 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: _check_auth_tickets 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:30.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 1 -- 172.21.15.98:0/831397561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae700a59a0 con 0x7fae780a7a10 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 5 monclient: authenticate success, global_id 104373 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient: get_monmap_and_config success 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 1 -- 172.21.15.98:0/831397561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae700ad3f0 con 0x7fae780a7a10 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 4 set_mon_vals no callback set 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae767fc640 10 monclient: dump: 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.461 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 10 monclient: shutdown 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 -- 172.21.15.98:0/831397561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 msgr2=0x7fae780a7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a7ea0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae70009e50 tx=0x7fae7001c8f0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 -- 172.21.15.98:0/831397561 shutdown_connections 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a7ea0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae7806fe30 0x7fae78139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 --2- 172.21.15.98:0/831397561 >> [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] conn(0x7fae78139b10 0x7fae780725d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.439+0000 7fae7dabb640 1 -- 172.21.15.98:0/831397561 >> 172.21.15.98:0/831397561 conn(0x7fae780ac770 msgr2=0x7fae780a8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- 172.21.15.98:0/831397561 shutdown_connections 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- 172.21.15.98:0/831397561 wait complete. 2024-09-11T03:08:30.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: build_initial_monmap 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: monmap: 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.noname-a 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.noname-f 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.noname-b 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.noname-g 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.noname-c 2024-09-11T03:08:30.463 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.noname-h 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.noname-d 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.noname-i 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.noname-e 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 librados: starting msgr at 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 librados: starting objecter 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 Processor -- start 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- start start 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 librados: setting wanted keys 2024-09-11T03:08:30.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 librados: calling monclient init 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: init 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _reopen_session rank -1 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _add_conns ranks=[0,3,8,4,5,1,6,7,2] 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): picked mon.noname-a con 0x7fae780a7a10 addr [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 --2- >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae78139b10 0x7fae7813fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): picked mon.noname-g con 0x7fae78139b10 addr [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 --2- >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae78140100 0x7fae781d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): picked mon.noname-e con 0x7fae78140100 addr [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fae781435c0 con 0x7fae780a7a10 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- --> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] -- mon_getmap magic: 0 -- 0x7fae78143440 con 0x7fae78139b10 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): start opening mon connection 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- --> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] -- mon_getmap magic: 0 -- 0x7fae78143740 con 0x7fae78140100 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): _renew_subs 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient(hunting): authenticate will time out at 2064.325700s 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:56436/0 (socket says 172.21.15.98:56436) 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 -- 172.21.15.98:0/276382442 learned_addr learned my addr 172.21.15.98:0/276382442 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): get_auth_request con 0x7fae780a7a10 auth_method 0 2024-09-11T03:08:30.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae78140100 0x7fae781d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 10 monclient(hunting): get_auth_request con 0x7fae78140100 auth_method 0 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-11T03:08:30.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae76ffd640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae78139b10 0x7fae7813fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae76ffd640 10 monclient(hunting): get_auth_request con 0x7fae78139b10 auth_method 0 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient(hunting): handle_auth_done global_id 104382 payload 824 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 -- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae78140100 msgr2=0x7fae781d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae78140100 0x7fae781d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 -- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae78139b10 msgr2=0x7fae7813fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae78139b10 0x7fae7813fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient: _finish_hunting 0 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 monclient: found mon.noname-a 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.98:3300/0 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 -- 172.21.15.98:0/276382442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7813cde0 con 0x7fae780a7a10 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae777fe640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a5140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00d620 tx=0x7fae6c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 client.?.objecter ms_handle_connect 0x7fae780a7a10 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 client.?.objecter resend_mon_ops 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae6c007380 con 0x7fae780a7a10 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: dump: 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.469 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.470 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: _finish_auth 0 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: _check_auth_tickets 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae6c019070 con 0x7fae780a7a10 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 5 monclient: authenticate success, global_id 104382 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _renew_subs 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae78073260 con 0x7fae780a7a10 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: handle_config config(0 keys) 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fae6c014430 con 0x7fae780a7a10 2024-09-11T03:08:30.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 monclient: dump: 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 797ac5bf-8bfb-4032-bff5-2e861cee7479 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-09-11T02:49:17.084567+0000 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.f 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:2: [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon.b 2024-09-11T03:08:30.472 INFO:teuthology.orchestra.run.smithi098.stderr:3: [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] mon.g 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:4: [v2:172.21.15.98:3302/0,v1:172.21.15.98:6791/0] mon.c 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:5: [v2:172.21.15.190:3302/0,v1:172.21.15.190:6791/0] mon.h 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:6: [v2:172.21.15.98:3303/0,v1:172.21.15.98:6792/0] mon.d 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:7: [v2:172.21.15.190:3303/0,v1:172.21.15.190:6792/0] mon.i 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:8: [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] mon.e 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae75ffb640 4 set_mon_vals no callback set 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 client.104382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _renew_subs 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:30.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae7813d680 con 0x7fae780a7a10 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 librados: init done 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 48526+0+0 (secure 0 0 0) 0x7fae6c013070 con 0x7fae780a7a10 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae781404e0 con 0x7fae780a7a10 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae6403df40 0x7fae64040400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(699..699 src has 1..699) ==== 5529+0+0 (secure 0 0 0) 0x7fae6c00f040 con 0x7fae780a7a10 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 10 client.104382.objecter ms_dispatch 0x7fae78000ba0 osd_map(699..699 src has 1..699) 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 3 client.104382.objecter handle_osd_map got epochs [699,699] > 0 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 3 client.104382.objecter handle_osd_map decoding full epoch 699 2024-09-11T03:08:30.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.443+0000 7fae5ffff640 20 client.104382.objecter dump_active .. 0 homeless 2024-09-11T03:08:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.447+0000 7fae76ffd640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae6403df40 0x7fae64040400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-11T03:08:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.447+0000 7fae76ffd640 10 monclient: get_auth_request con 0x7fae6403df40 auth_method 0 2024-09-11T03:08:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.447+0000 7fae5ffff640 10 client.104382.objecter ms_handle_connect 0x7fae6403df40 2024-09-11T03:08:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.447+0000 7fae76ffd640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae6403df40 0x7fae64040400 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7fae780a57e0 tx=0x7fae68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-11T03:08:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.451+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187845 (secure 0 0 0) 0x7fae781404e0 con 0x7fae780a7a10 2024-09-11T03:08:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.451+0000 7fae5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-11T03:08:30.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.451+0000 7fae5ffff640 10 monclient: _finish_command 1 = system:0 2024-09-11T03:08:30.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.611+0000 7fae7dabb640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T03:08:30.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.611+0000 7fae7dabb640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T03:08:30.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.611+0000 7fae7dabb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.98:3300/0 2024-09-11T03:08:30.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.611+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fae780a7ea0 con 0x7fae780a7a10 2024-09-11T03:08:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.619+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fae6c04de50 con 0x7fae780a7a10 2024-09-11T03:08:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.619+0000 7fae5ffff640 10 monclient: handle_config config(1 keys) 2024-09-11T03:08:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.619+0000 7fae75ffb640 4 set_mon_vals no callback set 2024-09-11T03:08:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.619+0000 7fae75ffb640 10 set_mon_vals mon_health_to_clog = false 2024-09-11T03:08:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.619+0000 7fae5ffff640 1 -- 172.21.15.98:0/276382442 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7fae7813cde0 con 0x7fae780a7a10 2024-09-11T03:08:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.619+0000 7fae5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-11T03:08:30.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.619+0000 7fae5ffff640 10 monclient: _finish_command 2 = system:0 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 10 librados: watch_flush enter 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 10 librados: watch_flush exit 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 20 client.104382.objecter shutdown clearing up homeless session... 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 10 client.104382.objecter successfully canceled tick 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae6403df40 msgr2=0x7fae64040400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae6403df40 0x7fae64040400 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7fae780a57e0 tx=0x7fae68008040 comp rx=0 tx=0).stop 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 10 monclient: shutdown 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-11T03:08:30.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 msgr2=0x7fae780a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a5140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00d620 tx=0x7fae6c00dae0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 shutdown_connections 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:6824/3356844748,v1:172.21.15.190:6825/3356844748] conn(0x7fae6403df40 0x7fae64040400 unknown :-1 s=CLOSED pgs=3728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3304/0,v1:172.21.15.98:6793/0] conn(0x7fae78140100 0x7fae781d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.190:3301/0,v1:172.21.15.190:6790/0] conn(0x7fae78139b10 0x7fae7813fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 --2- 172.21.15.98:0/276382442 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fae780a7a10 0x7fae780a5140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 >> 172.21.15.98:0/276382442 conn(0x7fae780ac770 msgr2=0x7fae78137cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 shutdown_connections 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 -- 172.21.15.98:0/276382442 wait complete. 2024-09-11T03:08:30.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-11T03:08:30.627+0000 7fae7dabb640 1 librados: shutdown 2024-09-11T03:08:30.642 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T03:08:30.642 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T03:08:30.643 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T03:08:30.643 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:30.722 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T03:08:30.722 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T03:08:30.722 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:30.819 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T03:08:30.819 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T03:08:30.819 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:30.894 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T03:08:30.895 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T03:08:30.895 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:30.958 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T03:08:30.958 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T03:08:30.958 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.035 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T03:08:31.036 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-11T03:08:31.036 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.119 INFO:tasks.ceph.osd.5:Stopped 2024-09-11T03:08:31.119 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T03:08:31.120 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T03:08:31.120 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.149 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T03:08:31.149 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T03:08:31.150 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T03:08:31.150 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.173 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T03:08:31.173 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T03:08:31.173 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.232 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T03:08:31.232 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T03:08:31.232 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.309 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T03:08:31.310 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-09-11T03:08:31.310 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.382 INFO:tasks.ceph.mon.d:Stopped 2024-09-11T03:08:31.383 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-09-11T03:08:31.383 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.464 INFO:tasks.ceph.mon.e:Stopped 2024-09-11T03:08:31.464 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-09-11T03:08:31.464 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.510 INFO:tasks.ceph.mon.f:Stopped 2024-09-11T03:08:31.510 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-09-11T03:08:31.511 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.528 INFO:tasks.ceph.mon.g:Stopped 2024-09-11T03:08:31.528 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-09-11T03:08:31.528 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.596 INFO:tasks.ceph.mon.h:Stopped 2024-09-11T03:08:31.597 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-09-11T03:08:31.597 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T03:08:31.660 INFO:tasks.ceph.mon.i:Stopped 2024-09-11T03:08:31.660 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T03:08:31.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-11T03:08:31.703 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T03:08:31.703 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T03:08:32.198 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T03:08:32.198 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T03:08:32.385 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T03:08:32.385 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T03:08:32.511 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T03:08:32.512 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T03:08:32.807 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T03:08:32.807 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-11T03:08:33.023 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T03:08:33.023 DEBUG:teuthology.orchestra.run.smithi190:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-11T03:08:33.207 INFO:tasks.ceph:Archiving mon data... 2024-09-11T03:08:33.208 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.a.tgz 2024-09-11T03:08:33.209 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-11T03:08:33.215 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.IZfs3lY7hl 2024-09-11T03:08:33.215 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IZfs3lY7hl 2024-09-11T03:08:33.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.IZfs3lY7hl 2024-09-11T03:08:33.590 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.IZfs3lY7hl is 1MB 2024-09-11T03:08:33.665 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.IZfs3lY7hl 2024-09-11T03:08:33.670 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.b.tgz 2024-09-11T03:08:33.670 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-11T03:08:33.718 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.95X9B2Zq2u 2024-09-11T03:08:33.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.95X9B2Zq2u 2024-09-11T03:08:34.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.95X9B2Zq2u 2024-09-11T03:08:34.098 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.95X9B2Zq2u is 1MB 2024-09-11T03:08:34.173 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.95X9B2Zq2u 2024-09-11T03:08:34.179 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.c.tgz 2024-09-11T03:08:34.179 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-11T03:08:34.226 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.soQHOm8GE0 2024-09-11T03:08:34.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.soQHOm8GE0 2024-09-11T03:08:34.522 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.soQHOm8GE0 2024-09-11T03:08:34.586 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.soQHOm8GE0 is 1MB 2024-09-11T03:08:34.657 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.soQHOm8GE0 2024-09-11T03:08:34.662 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.d.tgz 2024-09-11T03:08:34.662 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-11T03:08:34.710 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.bkgIiSqBV7 2024-09-11T03:08:34.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.bkgIiSqBV7 2024-09-11T03:08:35.018 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.bkgIiSqBV7 2024-09-11T03:08:35.082 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.bkgIiSqBV7 is 1MB 2024-09-11T03:08:35.193 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.bkgIiSqBV7 2024-09-11T03:08:35.199 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.e.tgz 2024-09-11T03:08:35.199 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-11T03:08:35.251 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.CJAjd1kW23 2024-09-11T03:08:35.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.CJAjd1kW23 2024-09-11T03:08:35.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.CJAjd1kW23 2024-09-11T03:08:35.606 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.CJAjd1kW23 is 1MB 2024-09-11T03:08:35.677 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.CJAjd1kW23 2024-09-11T03:08:35.682 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.f.tgz 2024-09-11T03:08:35.682 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-11T03:08:35.688 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.7xISpdojXP 2024-09-11T03:08:35.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.7xISpdojXP 2024-09-11T03:08:35.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.7xISpdojXP 2024-09-11T03:08:36.061 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.7xISpdojXP is 1MB 2024-09-11T03:08:36.133 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.7xISpdojXP 2024-09-11T03:08:36.139 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.g.tgz 2024-09-11T03:08:36.139 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-11T03:08:36.186 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.XaWOxKf1dD 2024-09-11T03:08:36.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.XaWOxKf1dD 2024-09-11T03:08:36.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.XaWOxKf1dD 2024-09-11T03:08:36.562 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.XaWOxKf1dD is 1MB 2024-09-11T03:08:36.636 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.XaWOxKf1dD 2024-09-11T03:08:36.642 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.h.tgz 2024-09-11T03:08:36.642 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-11T03:08:36.690 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.liOaNjQQYd 2024-09-11T03:08:36.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.liOaNjQQYd 2024-09-11T03:08:36.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.liOaNjQQYd 2024-09-11T03:08:37.058 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.liOaNjQQYd is 1MB 2024-09-11T03:08:37.128 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.liOaNjQQYd 2024-09-11T03:08:37.134 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/data/mon.i.tgz 2024-09-11T03:08:37.134 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-09-11T03:08:37.182 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.eu9N9kEwzp 2024-09-11T03:08:37.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.eu9N9kEwzp 2024-09-11T03:08:37.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.eu9N9kEwzp 2024-09-11T03:08:37.529 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.eu9N9kEwzp is 1MB 2024-09-11T03:08:37.605 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.eu9N9kEwzp 2024-09-11T03:08:37.611 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T03:08:37.611 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T03:08:37.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T03:08:37.672 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T03:08:37.714 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T03:08:37.714 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T03:08:37.716 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/remote/smithi098/crash 2024-09-11T03:08:37.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T03:08:37.730 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/crash to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/remote/smithi190/crash 2024-09-11T03:08:37.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T03:08:37.770 INFO:tasks.ceph:Compressing logs... 2024-09-11T03:08:37.770 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T03:08:37.775 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T03:08:37.794 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36331.log 2024-09-11T03:08:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31790.log 2024-09-11T03:08:37.795 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29453.log 2024-09-11T03:08:37.797 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35144.log 2024-09-11T03:08:37.797 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36331.log.gz 2024-09-11T03:08:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34962.log 2024-09-11T03:08:37.798 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31790.log.gz 2024-09-11T03:08:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29453.log: /var/log/ceph/ceph-client.admin.35144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29753.log 2024-09-11T03:08:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37655.log 2024-09-11T03:08:37.798 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35144.log.gz 2024-09-11T03:08:37.798 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31346.log 2024-09-11T03:08:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37655.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29753.log.gz 2024-09-11T03:08:37.806 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34962.log.gz 2024-09-11T03:08:37.806 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38097.log 2024-09-11T03:08:37.806 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29453.log.gz 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29612.log 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37655.log.gz 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36826.log 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.29612.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30321.log 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38097.log.gz 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36826.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.31346.log.gz 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34910.log 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30321.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29612.log.gz 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36826.log.gz 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37681.log 2024-09-11T03:08:37.807 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30321.log.gz 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35606.log 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36304.log 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37681.log: /var/log/ceph/ceph-client.admin.35606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34026.log 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34910.log.gz 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36304.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37681.log.gz 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37941.log 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35606.log.gz 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35944.log 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36304.log.gz 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34182.log 2024-09-11T03:08:37.808 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.34026.log.gz 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30035.log 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35944.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37941.log.gz 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30347.log 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35944.log.gz 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37603.log 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30347.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.34182.log.gz 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35788.log 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30035.log.gz 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36592.log 2024-09-11T03:08:37.809 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30347.log.gz 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31920.log 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37603.log.gz 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30191.log 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35788.log.gz 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37525.log 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30191.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36592.log.gz 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37369.log 2024-09-11T03:08:37.810 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31920.log.gz 2024-09-11T03:08:37.875 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28746.log 2024-09-11T03:08:37.875 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29448.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29753.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28557.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28746.log.gz 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28741.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29026.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25412.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28193.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:gzip/var/log/ceph/ceph-client.admin.25412.log: -5 --verbose 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29753.log.gz 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr: -- /var/log/ceph/ceph-client.admin.28465.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29498.log 2024-09-11T03:08:37.876 INFO:teuthology.orchestra.run.smithi190.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25412.log.gz/var/log/ceph/ceph-client.admin.28465.log: 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29448.log.gzgzip 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29752.log 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29498.log: 91.4% 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28741.log.gz 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.f.log: -- replaced with /var/log/ceph/ceph-client.admin.28193.log.gz 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28755.log 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29498.log.gz 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29838.log 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29395.log 2024-09-11T03:08:37.877 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28755.log: /var/log/ceph/ceph-client.admin.29838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28617.log 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29395.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.28557.log.gz 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28169.log 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28617.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29838.log.gz 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28744.log 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28169.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29752.log.gz 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28755.log.gz 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28938.log 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29395.log.gz 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28350.log 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28617.log.gz 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28465.log.gz/var/log/ceph/ceph-client.admin.28938.log: 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28383.log 2024-09-11T03:08:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28434.log 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28350.log: /var/log/ceph/ceph-client.admin.28383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25316.log 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28434.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28744.log.gz 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28676.log 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25316.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28434.log.gz 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.29026.log.gz 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28177.log 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28680.log 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25316.log.gz 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28676.log: /var/log/ceph/ceph-client.admin.28177.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T03:08:37.879 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28680.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28383.log.gz 2024-09-11T03:08:37.880 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-11T03:08:37.880 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.3.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.28177.log.gz 2024-09-11T03:08:37.880 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29499.log 2024-09-11T03:08:37.880 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.5.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.28680.log.gz 2024-09-11T03:08:37.880 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28198.log 2024-09-11T03:08:37.880 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29499.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.29499.log.gz 2024-09-11T03:08:37.880 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28610.log 2024-09-11T03:08:37.890 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28497.log 2024-09-11T03:08:37.890 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28610.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.28938.log.gz 2024-09-11T03:08:37.903 INFO:teuthology.orchestra.run.smithi190.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28610.log.gz 2024-09-11T03:08:37.903 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28705.log 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31634.log 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30191.log.gz 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37525.log.gz 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37369.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32934.log 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36462.log 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31634.log.gz 2024-09-11T03:08:37.906 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35040.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32934.log.gz 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35196.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38019.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35040.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36462.log.gz 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33168.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38019.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35196.log.gz 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35040.log.gz 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32130.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38019.log.gz 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33974.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33168.log: /var/log/ceph/ceph-client.admin.32130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37421.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30956.log 2024-09-11T03:08:37.907 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32130.log.gz 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33168.log.gz 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.37421.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33974.log.gz 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30581.log 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36774.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37421.log.gz 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30956.log.gz 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33792.log 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30581.log: 89.7%/var/log/ceph/ceph-client.admin.33740.log: -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29804.log 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30581.log.gz 2024-09-11T03:08:37.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31060.log 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:gzip 90.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz -- 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/ceph-client.admin.33766.log 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29804.log: /var/log/ceph/ceph-client.admin.31060.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33792.log.gz 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36100.log 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29804.log.gz 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31060.log.gz 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31008.log 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33766.log: /var/log/ceph/ceph-client.admin.36100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37192.log 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31294.log 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37192.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33766.log.gz 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36100.log.gz 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32026.log 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31008.log.gz 2024-09-11T03:08:37.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35866.log 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32026.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37192.log.gz 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29983.log 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31294.log.gz 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32026.log.gz 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31712.log 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30399.log 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31712.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29983.log.gz 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35866.log.gz 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33922.log 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31712.log.gz 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31816.log 2024-09-11T03:08:37.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33610.log 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30399.log.gz 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31816.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33922.log.gz 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35248.log 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.33610.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30930.log 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0%/var/log/ceph/ceph-client.admin.35248.log: -- replaced with /var/log/ceph/ceph-client.admin.31816.log.gz 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30165.log 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33610.log.gz 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37034.log 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35248.log.gz 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30982.log 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30930.log.gz 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37034.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T03:08:37.911 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30165.log.gz 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37034.log.gz 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35840.log 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37008.log 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35840.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30982.log.gz 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36410.log 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37008.log: gzip 90.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.35840.log.gz --verbose 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/ceph-client.admin.33532.log 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/ceph-client.admin.36410.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36488.log 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.37008.log.gz 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33532.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21475.log 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36410.log.gz 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29385.log 2024-09-11T03:08:37.912 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33532.log.gz 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.21475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37166.log 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21475.log.gz 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36488.log.gz 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34314.log 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31320.log 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29385.log.gz 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34314.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37166.log.gz 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36514.log 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37577.log 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31320.log: /var/log/ceph/ceph-client.admin.36514.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34314.log.gz 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31216.log 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33844.log 2024-09-11T03:08:37.913 INFO:teuthology.orchestra.run.smithi098.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36514.log.gz 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31216.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31320.log.gz 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36722.log 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33844.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36722.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31216.log.gz 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30113.log 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33844.log.gz 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32960.log 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36722.log.gz 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32986.log 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30113.log: /var/log/ceph/ceph-client.admin.32960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29749.log 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32986.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31138.log 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30113.log.gz 89.6% 2024-09-11T03:08:37.914 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32960.log.gz 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32619.log 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29749.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32986.log.gz 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37114.log 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31528.log 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29749.log.gz 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31138.log.gz 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34936.log 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35580.log 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37114.log.gz 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34936.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.32619.log.gz 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31450.log 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr:gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31528.log.gz -5 2024-09-11T03:08:37.915 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-mon.e.log 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35580.log: /var/log/ceph/ceph-client.admin.31450.log: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32104.log 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34936.log.gz 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29606.log 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35580.log.gz 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31450.log.gz 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36074.log 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32104.log: /var/log/ceph/ceph-client.admin.29606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31112.log 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32338.log 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32104.log.gz 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29606.log.gz 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31112.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36074.log.gz 2024-09-11T03:08:37.916 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31112.log.gz 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34648.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30477.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32286.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31972.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32338.log.gz 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32286.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34648.log.gz 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.30477.log.gz 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32286.log.gz 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34856.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34726.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30826.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33405.log 2024-09-11T03:08:37.917 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31972.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34856.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30826.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34726.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30826.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37863.log 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35118.log 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30610.log 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32673.log 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33405.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30610.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37863.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35118.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30610.log.gz 2024-09-11T03:08:37.918 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34884.log 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35892.log 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32078.log 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30555.log 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32673.log.gz 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34884.log.gz 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32078.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35892.log.gz 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30425.log 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30555.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32078.log.gz 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31398.log 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29905.log 2024-09-11T03:08:37.919 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30555.log.gz/var/log/ceph/ceph-client.admin.31398.log: 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29593.log 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30425.log.gz 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30009.log 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29905.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31398.log.gz 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30694.log 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6%/var/log/ceph/ceph-client.admin.30009.log: -- replaced with /var/log/ceph/ceph-client.admin.29905.log.gz 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34622.log 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29593.log.gz 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30009.log.gz 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38045.log 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34414.log 2024-09-11T03:08:37.920 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30694.log.gz 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34622.log.gz 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37140.log: /var/log/ceph/ceph-client.admin.38045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34518.log 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34414.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38045.log.gz -5 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34414.log.gz 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33714.log 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31476.log 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34518.log.gz 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35398.log 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz/var/log/ceph/ceph-client.admin.31476.log: 2024-09-11T03:08:37.921 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31502.log 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33714.log.gz 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35398.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31476.log.gz 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31502.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31502.log.gz 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35398.log.gz 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35450.log 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35632.log 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32882.log 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0%/var/log/ceph/ceph-client.admin.35632.log: -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36278.log 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35450.log.gz 2024-09-11T03:08:37.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32882.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35996.log 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35632.log.gz 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36278.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36278.log.gz 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32882.log.gz 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33501.log 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35274.log 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31868.log 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2%/var/log/ceph/ceph-client.admin.35274.log: -- replaced with /var/log/ceph/ceph-client.admin.35996.log.gz 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31868.log: gzip -5 --verbose -- 90.6% /var/log/ceph/ceph-client.admin.36357.log 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33501.log.gz 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35274.log.gz 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34988.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2024-09-11T03:08:37.923 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31868.log.gz 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29779.log 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37499.log 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33272.log 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36357.log.gz/var/log/ceph/ceph-client.admin.37499.log: 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32778.log 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29779.log.gz 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31190.log 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37499.log.gz 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33688.log 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34700.log 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1%/var/log/ceph/ceph-client.admin.33688.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33272.log.gz 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32778.log.gz 2024-09-11T03:08:37.924 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36022.log 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31190.log.gz 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33688.log.gz 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36152.log 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34104.log 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8%/var/log/ceph/ceph-client.admin.36152.log: -- replaced with /var/log/ceph/ceph-client.admin.34700.log.gz 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29608.log 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36022.log.gz 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37218.log 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34104.log: /var/log/ceph/ceph-client.admin.29608.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36152.log.gz 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34104.log.gz 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29608.log.gz 2024-09-11T03:08:37.925 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31554.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35814.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36436.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9%/var/log/ceph/ceph-client.admin.35814.log: -- replaced with /var/log/ceph/ceph-client.admin.37218.log.gz 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29313.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31554.log.gz 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36436.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35814.log.gz 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36436.log.gz 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32726.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32182.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32156.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30269.log 2024-09-11T03:08:37.926 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29313.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32726.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32156.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32182.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32156.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32208.log 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33558.log 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33379.log 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37447.log 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30269.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32208.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33379.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33558.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33379.log.gz 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-09-11T03:08:37.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31268.log 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37473.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37447.log.gz 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34804.log 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30217.log 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31268.log: /var/log/ceph/ceph-client.admin.34804.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31268.log.gz 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34804.log.gz 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33636.log 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37395.log 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31660.log 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30217.log.gz 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37395.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33636.log.gz 2024-09-11T03:08:37.928 INFO:teuthology.orchestra.run.smithi098.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37395.log.gz 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36696.log 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31660.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29457.log 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31660.log.gz 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.d.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36696.log.gz 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37785.log 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29456.log 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29457.log: /var/log/ceph/ceph-client.admin.37785.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37785.log.gz 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29457.log.gz 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33142.log 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31738.log 2024-09-11T03:08:37.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29456.log: /var/log/ceph/ceph-client.admin.33142.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29456.log.gz 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33142.log.gz 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31738.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31738.log.gz 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25479.log 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30636.log 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34752.log: /var/log/ceph/ceph-client.admin.25479.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.25479.log.gz 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34262.log 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30636.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30636.log.gz 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29411.log 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37837.log 2024-09-11T03:08:37.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34262.log: /var/log/ceph/ceph-client.admin.29411.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34262.log.gz 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29411.log.gz 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31164.log 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37837.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37837.log.gz 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30061.log 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35970.log 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31164.log: /var/log/ceph/ceph-client.admin.30061.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31164.log.gz 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30061.log.gz 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36618.log 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35970.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.35970.log.gz 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34544.log 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34236.log 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36618.log: /var/log/ceph/ceph-client.admin.34544.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36618.log.gz 2024-09-11T03:08:37.931 INFO:teuthology.orchestra.run.smithi098.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34544.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29613.log 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34236.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34236.log.gz 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29148.log 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34156.log 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29613.log: /var/log/ceph/ceph-client.admin.29148.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29613.log.gz 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34466.log 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29148.log.gz 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34156.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34156.log.gz 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34078.log 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34466.log: /var/log/ceph/ceph-client.admin.34078.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34466.log.gz 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34078.log.gz 2024-09-11T03:08:37.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30529.log 2024-09-11T03:08:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36126.log 2024-09-11T03:08:37.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.30529.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30529.log.gz 2024-09-11T03:08:37.945 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28497.log: 92.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29446.log 2024-09-11T03:08:37.945 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28705.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28705.log.gz 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28497.log.gz 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28756.log 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29446.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29446.log.gz 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29397.log 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28169.log.gz 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29500.log 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29397.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.28676.log.gz 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28756.log.gz 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28228.log 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25436.log 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28228.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29397.log.gz 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.i.log 2024-09-11T03:08:37.946 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25436.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29500.log.gz 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25436.log.gz 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.g.log 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.i.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29756.log 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr: 91.5%/var/log/ceph/ceph-mon.g.log: -- replaced with /var/log/ceph/ceph-client.admin.28228.log.gz 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28420.log 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29756.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.29756.log.gz 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29148.log 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28420.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28420.log.gz 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28403.log 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29148.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28403.log: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.28198.log.gz 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29148.log.gz 2024-09-11T03:08:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28234.log 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33246.log 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36126.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36126.log.gz 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37915.log 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33246.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33246.log.gz 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35502.log 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37915.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37915.log.gz 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25169.log 2024-09-11T03:08:37.949 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35502.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35502.log.gz 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34340.log 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.25169.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.25169.log.gz 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37316.log 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34340.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34340.log.gz 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31034.log 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37316.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37316.log.gz 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29780.log 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31034.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31034.log.gz 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32856.log 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29780.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29780.log.gz 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29931.log 2024-09-11T03:08:37.950 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32856.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32856.log.gz 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29605.log 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29931.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29931.log.gz 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25402.log 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29605.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29605.log.gz 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37811.log 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.25402.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.25402.log.gz 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33818.log 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37811.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37811.log.gz 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35066.log 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33818.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33818.log.gz 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30662.log 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35066.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.35066.log.gz 2024-09-11T03:08:37.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37343.log 2024-09-11T03:08:37.953 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mgr.x.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28403.log.gz 2024-09-11T03:08:37.953 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-11T03:08:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30662.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30662.log.gz 2024-09-11T03:08:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31686.log 2024-09-11T03:08:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37343.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37343.log.gz 2024-09-11T03:08:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37629.log 2024-09-11T03:08:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31686.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31686.log.gz 2024-09-11T03:08:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32473.log 2024-09-11T03:08:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37629.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37629.log.gz 2024-09-11T03:08:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36048.log 2024-09-11T03:08:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32473.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32473.log.gz 2024-09-11T03:08:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-11T03:08:37.958 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29164.log 2024-09-11T03:08:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36048.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36048.log.gz 2024-09-11T03:08:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31998.log 2024-09-11T03:08:37.974 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28475.log 2024-09-11T03:08:37.976 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29164.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29164.log.gz 2024-09-11T03:08:37.978 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37062.log 2024-09-11T03:08:37.980 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31998.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31998.log.gz 2024-09-11T03:08:37.990 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29497.log 2024-09-11T03:08:37.994 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32645.log 2024-09-11T03:08:37.995 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28475.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.28475.log.gz 2024-09-11T03:08:37.996 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37062.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.37062.log.gz 2024-09-11T03:08:38.002 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38071.log 2024-09-11T03:08:38.004 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32645.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.32645.log.gz 2024-09-11T03:08:38.010 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28872.log 2024-09-11T03:08:38.010 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33870.log 2024-09-11T03:08:38.011 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38071.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38071.log.gz 2024-09-11T03:08:38.013 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29497.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29497.log.gz 2024-09-11T03:08:38.018 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31242.log 2024-09-11T03:08:38.019 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33870.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33870.log.gz 2024-09-11T03:08:38.026 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-09-11T03:08:38.026 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37993.log 2024-09-11T03:08:38.027 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28872.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28872.log.gz 2024-09-11T03:08:38.027 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31242.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31242.log.gz 2024-09-11T03:08:38.034 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29837.log 2024-09-11T03:08:38.034 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33012.log 2024-09-11T03:08:38.035 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37993.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37993.log.gz 2024-09-11T03:08:38.042 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-mon.h.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29194.log 2024-09-11T03:08:38.042 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34778.log 2024-09-11T03:08:38.043 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33012.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33012.log.gz 2024-09-11T03:08:38.045 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29837.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29837.log.gz 2024-09-11T03:08:38.050 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33324.log 2024-09-11T03:08:38.051 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34778.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34778.log.gz 2024-09-11T03:08:38.055 INFO:teuthology.orchestra.run.smithi098.stderr: 90.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T03:08:38.058 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34130.log 2024-09-11T03:08:38.059 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33324.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33324.log.gz 2024-09-11T03:08:38.060 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37290.log 2024-09-11T03:08:38.060 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34130.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34130.log.gz 2024-09-11T03:08:38.061 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29450.log 2024-09-11T03:08:38.062 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25340.log 2024-09-11T03:08:38.062 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37290.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.37290.log.gz 2024-09-11T03:08:38.063 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32699.log 2024-09-11T03:08:38.063 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29450.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29450.log.gz 2024-09-11T03:08:38.064 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36982.log 2024-09-11T03:08:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32699.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32699.log.gz 2024-09-11T03:08:38.065 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36852.log 2024-09-11T03:08:38.066 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36982.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36982.log.gz 2024-09-11T03:08:38.067 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34596.log 2024-09-11T03:08:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36852.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36852.log.gz 2024-09-11T03:08:38.068 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31894.log 2024-09-11T03:08:38.069 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34596.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34596.log.gz 2024-09-11T03:08:38.070 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34000.log 2024-09-11T03:08:38.070 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31894.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31894.log.gz 2024-09-11T03:08:38.071 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31086.log 2024-09-11T03:08:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34000.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34000.log.gz 2024-09-11T03:08:38.072 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30800.log 2024-09-11T03:08:38.073 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31086.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31086.log.gz 2024-09-11T03:08:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32830.log 2024-09-11T03:08:38.074 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30800.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30800.log.gz 2024-09-11T03:08:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30852.log 2024-09-11T03:08:38.076 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32830.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32830.log.gz 2024-09-11T03:08:38.077 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32234.log 2024-09-11T03:08:38.078 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30852.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30852.log.gz 2024-09-11T03:08:38.078 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32260.log 2024-09-11T03:08:38.079 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29501.log 2024-09-11T03:08:38.079 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25340.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25340.log.gz 2024-09-11T03:08:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32234.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32234.log.gz 2024-09-11T03:08:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30503.log 2024-09-11T03:08:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32260.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.32260.log.gz 2024-09-11T03:08:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35918.log 2024-09-11T03:08:38.081 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30503.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30503.log.gz 2024-09-11T03:08:38.082 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34210.log 2024-09-11T03:08:38.083 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35918.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35918.log.gz 2024-09-11T03:08:38.083 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T03:08:38.084 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34210.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34210.log.gz 2024-09-11T03:08:38.085 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30243.log 2024-09-11T03:08:38.094 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28185.log 2024-09-11T03:08:38.094 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29501.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29501.log.gz 2024-09-11T03:08:38.106 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35762.log 2024-09-11T03:08:38.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30243.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30243.log.gz 2024-09-11T03:08:38.110 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29125.log 2024-09-11T03:08:38.122 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28185.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T03:08:38.122 INFO:teuthology.orchestra.run.smithi098.stderr: 96.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33353.log 2024-09-11T03:08:38.123 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35762.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35762.log.gz 2024-09-11T03:08:38.123 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T03:08:38.124 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34570.log 2024-09-11T03:08:38.125 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33353.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33353.log.gz 2024-09-11T03:08:38.125 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29263.log 2024-09-11T03:08:38.126 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34570.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34570.log.gz 2024-09-11T03:08:38.127 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34830.log 2024-09-11T03:08:38.128 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29263.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29263.log.gz 2024-09-11T03:08:38.128 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33090.log 2024-09-11T03:08:38.129 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34830.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34830.log.gz 2024-09-11T03:08:38.130 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32390.log 2024-09-11T03:08:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33090.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33090.log.gz 2024-09-11T03:08:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37967.log 2024-09-11T03:08:38.132 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32390.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32390.log.gz 2024-09-11T03:08:38.133 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30904.log 2024-09-11T03:08:38.134 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28844.log 2024-09-11T03:08:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37967.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37967.log.gz 2024-09-11T03:08:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30451.log 2024-09-11T03:08:38.135 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph.audit.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29125.log.gz 2024-09-11T03:08:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30904.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30904.log.gz 2024-09-11T03:08:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31372.log 2024-09-11T03:08:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30451.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30451.log.gz 2024-09-11T03:08:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37759.log 2024-09-11T03:08:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31372.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31372.log.gz 2024-09-11T03:08:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34366.log 2024-09-11T03:08:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37759.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37759.log.gz 2024-09-11T03:08:38.140 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35684.log 2024-09-11T03:08:38.141 INFO:teuthology.orchestra.run.smithi190.stderr: 92.9% 93.6% -- replaced with /var/log/ceph/ceph-client.admin.28185.log.gz 2024-09-11T03:08:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34366.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.34366.log.gz 2024-09-11T03:08:38.142 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33298.log 2024-09-11T03:08:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35684.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35684.log.gz 2024-09-11T03:08:38.143 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32908.log 2024-09-11T03:08:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33298.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33298.log.gz 2024-09-11T03:08:38.145 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30139.log 2024-09-11T03:08:38.146 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32908.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32908.log.gz 2024-09-11T03:08:38.146 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35476.log 2024-09-11T03:08:38.147 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30139.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30139.log.gz 2024-09-11T03:08:38.148 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29288.log 2024-09-11T03:08:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35476.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35476.log.gz 2024-09-11T03:08:38.149 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34440.log 2024-09-11T03:08:38.150 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29288.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29288.log.gz 2024-09-11T03:08:38.151 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36930.log 2024-09-11T03:08:38.152 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34440.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.34440.log.gz 2024-09-11T03:08:38.153 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36384.log 2024-09-11T03:08:38.153 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29194.log.gz 2024-09-11T03:08:38.154 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36930.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36930.log.gz 2024-09-11T03:08:38.154 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34052.log 2024-09-11T03:08:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36384.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36384.log.gz 2024-09-11T03:08:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38121.log 2024-09-11T03:08:38.157 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34052.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34052.log.gz 2024-09-11T03:08:38.157 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32364.log 2024-09-11T03:08:38.158 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29104.log 2024-09-11T03:08:38.159 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.38121.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38121.log.gz 2024-09-11T03:08:38.159 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30373.log 2024-09-11T03:08:38.160 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32364.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32364.log.gz 2024-09-11T03:08:38.160 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35344.log 2024-09-11T03:08:38.161 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30373.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30373.log.gz 2024-09-11T03:08:38.161 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29454.log 2024-09-11T03:08:38.162 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35344.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35344.log.gz 2024-09-11T03:08:38.163 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33896.log 2024-09-11T03:08:38.164 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28844.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.28844.log.gz 2024-09-11T03:08:38.164 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29454.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29454.log.gz 2024-09-11T03:08:38.164 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33038.log 2024-09-11T03:08:38.166 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33896.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33896.log.gz 2024-09-11T03:08:38.166 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34288.log 2024-09-11T03:08:38.167 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33038.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33038.log.gz 2024-09-11T03:08:38.168 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36800.log 2024-09-11T03:08:38.169 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34288.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34288.log.gz 2024-09-11T03:08:38.169 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30878.log 2024-09-11T03:08:38.170 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29754.log 2024-09-11T03:08:38.171 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36800.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36800.log.gz 2024-09-11T03:08:38.171 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36878.log 2024-09-11T03:08:38.172 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30878.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30878.log.gz 2024-09-11T03:08:38.172 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36748.log 2024-09-11T03:08:38.173 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36878.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36878.log.gz 2024-09-11T03:08:38.174 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29359.log 2024-09-11T03:08:38.175 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36748.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36748.log.gz 2024-09-11T03:08:38.175 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31764.log 2024-09-11T03:08:38.176 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29359.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29359.log.gz 2024-09-11T03:08:38.177 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32052.log 2024-09-11T03:08:38.178 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31764.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31764.log.gz 2024-09-11T03:08:38.178 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37707.log 2024-09-11T03:08:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32052.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32052.log.gz 2024-09-11T03:08:38.180 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37088.log 2024-09-11T03:08:38.181 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37707.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37707.log.gz 2024-09-11T03:08:38.181 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36904.log 2024-09-11T03:08:38.183 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37088.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37088.log.gz 2024-09-11T03:08:38.183 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35222.log 2024-09-11T03:08:38.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36904.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36904.log.gz 2024-09-11T03:08:38.184 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-11T03:08:38.186 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35222.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35222.log.gz 2024-09-11T03:08:38.186 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35372.log 2024-09-11T03:08:38.190 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29104.log: 91.6% 96.7% 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29104.log.gz 2024-09-11T03:08:38.190 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28901.log 2024-09-11T03:08:38.191 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29754.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T03:08:38.191 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T03:08:38.198 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29442.log 2024-09-11T03:08:38.199 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35372.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35372.log.gz 2024-09-11T03:08:38.204 INFO:teuthology.orchestra.run.smithi190.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29754.log.gz 2024-09-11T03:08:38.206 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29396.log 2024-09-11T03:08:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33194.log 2024-09-11T03:08:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29442.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29442.log.gz 2024-09-11T03:08:38.215 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32804.log 2024-09-11T03:08:38.217 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33194.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33194.log.gz 2024-09-11T03:08:38.226 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28901.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28363.log 2024-09-11T03:08:38.228 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29396.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.29396.log.gz 2024-09-11T03:08:38.229 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28901.log.gz 2024-09-11T03:08:38.238 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29957.log 2024-09-11T03:08:38.239 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32804.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32804.log.gz 2024-09-11T03:08:38.242 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28161.log 2024-09-11T03:08:38.246 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30295.log 2024-09-11T03:08:38.247 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29957.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29957.log.gz 2024-09-11T03:08:38.249 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28363.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28363.log.gz 2024-09-11T03:08:38.250 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25364.log 2024-09-11T03:08:38.266 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36644.log 2024-09-11T03:08:38.267 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30295.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30295.log.gz 2024-09-11T03:08:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28740.log 2024-09-11T03:08:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25364.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25364.log.gz 2024-09-11T03:08:38.274 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33116.log 2024-09-11T03:08:38.275 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36644.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36644.log.gz 2024-09-11T03:08:38.278 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-11T03:08:38.281 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28740.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28740.log.gz 2024-09-11T03:08:38.282 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29333.log 2024-09-11T03:08:38.283 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33116.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33116.log.gz 2024-09-11T03:08:38.290 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37551.log 2024-09-11T03:08:38.291 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29333.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29333.log.gz 2024-09-11T03:08:38.294 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29755.log 2024-09-11T03:08:38.298 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33064.log 2024-09-11T03:08:38.299 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37551.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37551.log.gz 2024-09-11T03:08:38.306 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29447.log 2024-09-11T03:08:38.306 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32752.log 2024-09-11T03:08:38.307 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33064.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33064.log.gz 2024-09-11T03:08:38.309 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29755.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29755.log.gz 2024-09-11T03:08:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31842.log 2024-09-11T03:08:38.314 INFO:teuthology.orchestra.run.smithi190.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28161.log.gz 2024-09-11T03:08:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32752.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32752.log.gz 2024-09-11T03:08:38.330 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28757.log 2024-09-11T03:08:38.330 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35554.log 2024-09-11T03:08:38.331 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31842.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31842.log.gz 2024-09-11T03:08:38.332 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29447.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.29447.log.gz 2024-09-11T03:08:38.338 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36178.log 2024-09-11T03:08:38.339 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35554.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35554.log.gz 2024-09-11T03:08:38.346 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28279.log 2024-09-11T03:08:38.346 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36213.log 2024-09-11T03:08:38.347 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28757.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28757.log.gz 2024-09-11T03:08:38.347 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36178.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36178.log.gz 2024-09-11T03:08:38.354 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35658.log 2024-09-11T03:08:38.355 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36213.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36213.log.gz 2024-09-11T03:08:38.358 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29839.log 2024-09-11T03:08:38.362 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33584.log 2024-09-11T03:08:38.363 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35658.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35658.log.gz 2024-09-11T03:08:38.370 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37733.log 2024-09-11T03:08:38.371 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33584.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33584.log.gz 2024-09-11T03:08:38.378 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29012.log 2024-09-11T03:08:38.378 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35736.log 2024-09-11T03:08:38.379 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37733.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37733.log.gz 2024-09-11T03:08:38.381 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29839.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29839.log.gz 2024-09-11T03:08:38.386 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33662.log 2024-09-11T03:08:38.387 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35736.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35736.log.gz 2024-09-11T03:08:38.394 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35710.log 2024-09-11T03:08:38.395 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33662.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33662.log.gz 2024-09-11T03:08:38.397 INFO:teuthology.orchestra.run.smithi190.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T03:08:38.398 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28209.log 2024-09-11T03:08:38.400 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.29012.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29012.log.gz 2024-09-11T03:08:38.402 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36566.log 2024-09-11T03:08:38.403 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35710.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35710.log.gz 2024-09-11T03:08:38.410 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36540.log 2024-09-11T03:08:38.411 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36566.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36566.log.gz 2024-09-11T03:08:38.416 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25388.log 2024-09-11T03:08:38.418 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28209.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.28209.log.gz 2024-09-11T03:08:38.419 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34674.log 2024-09-11T03:08:38.419 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36540.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36540.log.gz 2024-09-11T03:08:38.422 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28960.log 2024-09-11T03:08:38.423 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.25388.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.25388.log.gz 2024-09-11T03:08:38.430 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35528.log 2024-09-11T03:08:38.431 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34674.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34674.log.gz 2024-09-11T03:08:38.438 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32312.log 2024-09-11T03:08:38.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35528.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35528.log.gz 2024-09-11T03:08:38.446 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35170.log 2024-09-11T03:08:38.447 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.32312.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32312.log.gz 2024-09-11T03:08:38.454 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/ceph-client.admin.28960.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.28279.log.gz 2024-09-11T03:08:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-09-11T03:08:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35170.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35170.log.gz 2024-09-11T03:08:38.462 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31946.log 2024-09-11T03:08:38.463 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34492.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-09-11T03:08:38.466 INFO:teuthology.orchestra.run.smithi190.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.28350.log.gz 2024-09-11T03:08:38.467 INFO:teuthology.orchestra.run.smithi190.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.28960.log.gz 2024-09-11T03:08:38.468 INFO:teuthology.orchestra.run.smithi098.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T03:08:38.471 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37889.log 2024-09-11T03:08:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31946.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31946.log.gz 2024-09-11T03:08:38.472 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29809.log 2024-09-11T03:08:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.37889.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37889.log.gz 2024-09-11T03:08:38.473 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36670.log 2024-09-11T03:08:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.29809.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29809.log.gz 2024-09-11T03:08:38.474 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30087.log 2024-09-11T03:08:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.36670.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36670.log.gz 2024-09-11T03:08:38.475 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35424.log 2024-09-11T03:08:38.476 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.30087.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30087.log.gz 2024-09-11T03:08:38.477 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33948.log 2024-09-11T03:08:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35424.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35424.log.gz 2024-09-11T03:08:38.478 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31424.log 2024-09-11T03:08:38.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33948.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33948.log.gz 2024-09-11T03:08:38.480 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35092.log 2024-09-11T03:08:38.481 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.31424.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31424.log.gz 2024-09-11T03:08:38.481 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33220.log 2024-09-11T03:08:38.482 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35092.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35092.log.gz 2024-09-11T03:08:38.484 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.33220.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.33220.log.gz 2024-09-11T03:08:38.486 INFO:teuthology.orchestra.run.smithi098.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-09-11T03:08:38.860 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-09-11T03:08:39.113 INFO:teuthology.orchestra.run.smithi098.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-11T03:08:39.136 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-11T03:08:39.277 INFO:teuthology.orchestra.run.smithi190.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-09-11T03:08:39.377 INFO:teuthology.orchestra.run.smithi098.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-09-11T03:08:39.719 INFO:teuthology.orchestra.run.smithi190.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-09-11T03:08:39.768 INFO:teuthology.orchestra.run.smithi190.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.28234.log.gz 2024-09-11T03:08:40.038 INFO:teuthology.orchestra.run.smithi190.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-09-11T03:08:44.256 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T03:08:45.103 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-11T03:08:45.155 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T03:08:45.310 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-11T03:08:45.321 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T03:08:45.323 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-09-11T03:08:45.323 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m7.502s 2024-09-11T03:08:45.323 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m29.654s 2024-09-11T03:08:45.323 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m1.003s 2024-09-11T03:08:45.818 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T03:08:45.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-11T03:08:45.820 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m8.042s 2024-09-11T03:08:45.820 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m26.313s 2024-09-11T03:08:45.820 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.940s 2024-09-11T03:08:45.821 INFO:tasks.ceph:Archiving logs... 2024-09-11T03:08:45.821 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/remote/smithi098/log 2024-09-11T03:08:45.822 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T03:08:49.489 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/remote/smithi190/log 2024-09-11T03:08:49.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T03:08:53.238 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T03:08:53.248 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T03:08:53.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T03:08:53.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T03:08:53.338 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T03:08:53.347 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T03:08:53.347 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:08:53.350 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T03:08:53.361 INFO:teuthology.orchestra.run.smithi098.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T03:08:53.361 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================== 2024-09-11T03:08:53.361 INFO:teuthology.orchestra.run.smithi098.stdout:*hv01.front.sepi 67.205.162.81 3 u 35 64 377 0.053 -2.585 0.299 2024-09-11T03:08:53.361 INFO:teuthology.orchestra.run.smithi098.stdout:+hv02.front.sepi 96.245.170.99 3 u 39 64 377 0.101 -2.136 0.337 2024-09-11T03:08:53.361 INFO:teuthology.orchestra.run.smithi098.stdout:+hv03.front.sepi 74.6.168.72 3 u 35 64 377 0.078 +0.945 0.486 2024-09-11T03:08:53.361 INFO:teuthology.orchestra.run.smithi098.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-11T03:08:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T03:08:53.363 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2024-09-11T03:08:53.363 INFO:teuthology.orchestra.run.smithi190.stdout:+hv01.front.sepi 67.205.162.81 3 u 39 64 377 0.073 -8.917 1.245 2024-09-11T03:08:53.363 INFO:teuthology.orchestra.run.smithi190.stdout:*hv02.front.sepi 96.245.170.99 3 u 40 64 377 0.103 -6.864 0.838 2024-09-11T03:08:53.363 INFO:teuthology.orchestra.run.smithi190.stdout:+hv03.front.sepi 74.6.168.72 3 u 37 64 377 0.093 -5.662 1.430 2024-09-11T03:08:53.363 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-11T03:08:53.365 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T03:08:53.393 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T03:08:53.394 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T03:08:53.469 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T03:08:53.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T03:08:53.615 INFO:teuthology.task.internal:Duration was 1746.797025 seconds 2024-09-11T03:08:53.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T03:08:53.662 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T03:08:53.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T03:08:53.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T03:08:53.702 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T03:08:53.702 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-09-11T03:08:53.702 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T03:08:53.752 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-09-11T03:08:53.752 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T03:08:53.763 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T03:08:53.763 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T03:08:53.795 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T03:08:53.837 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T03:08:53.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T03:08:53.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T03:08:53.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T03:08:53.944 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T03:08:53.944 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T03:08:53.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T03:08:53.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T03:08:54.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:08:54.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:08:54.012 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-09-11T03:08:54.018 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-09-11T03:08:54.039 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:08:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:08:54.086 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T03:08:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T03:08:54.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T03:08:54.101 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T03:08:54.102 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/remote/smithi098 2024-09-11T03:08:54.102 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T03:08:54.151 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/hyelloji-2024-09-10_17:16:14-rados-wip-hemanth-testing-2024-09-10-0723-distro-default-smithi/7900006/remote/smithi190 2024-09-11T03:08:54.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T03:08:54.192 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T03:08:54.192 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T03:08:54.194 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T03:08:54.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T03:08:54.230 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T03:08:54.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T03:08:54.262 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T03:08:54.262 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T03:08:54.265 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T03:08:54.269 INFO:teuthology.orchestra.run.smithi098.stdout: 658623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 03:08 /home/ubuntu/cephtest 2024-09-11T03:08:54.272 INFO:teuthology.orchestra.run.smithi190.stdout: 658653 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 03:08 /home/ubuntu/cephtest 2024-09-11T03:08:54.274 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T03:08:54.337 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T03:08:54.474 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1746.7970254421234 flavor: default owner: scheduled_hyelloji@teuthology success: true 2024-09-11T03:08:54.474 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T03:08:54.575 INFO:teuthology.run:pass